From patchwork Thu Jul 20 17:39:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 123423 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3298343vqt; Thu, 20 Jul 2023 11:14:26 -0700 (PDT) X-Google-Smtp-Source: APBJJlF/vFA8QEofnY0V0s0XG0ltrsKwr7TNnu/ryFr8a4uasgb7mpc2plnQLspZVGvSR6Yflcwt X-Received: by 2002:a17:903:1249:b0:1b9:cb27:7f43 with SMTP id u9-20020a170903124900b001b9cb277f43mr212600plh.43.1689876866164; Thu, 20 Jul 2023 11:14:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689876866; cv=none; d=google.com; s=arc-20160816; b=Nq/wPQuxBH6evVbPghBjJfLCVzVHy0lUbLqIB4s7UV+Uh1TGVd4MEkshfVrEIJaM/D sHk4E5nClR3RuhK0xzMGg/wg0ystgSw7FV1gHN+YN/AU5rUayIUzMKBdzbwxsq2JqaPA hla8pI8Yyce13k/C9yhGdHGtY3RnLZL7K1WPOXZogb9XXSO8CSjffI5RClsuOr+DF7Nl uxEQoa9X5cIZV4+Qbc4odlw88AN0GcEm4HL7AwZbg7Dvcs/q/05GfZ4S/42kS/3GIlOH O+LBZOaEmPD9eGUvdM0zr623g+bFjhb8ZsOHpDD3A2/RgftB/h3fcLn1RxkptJhMUdyN CvkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=sBTD77nZvv18xeO6UXUxdZ0TBIDcYAZAkWaNswyfelY=; fh=JgIgyeVfHwvOXq2SUW9xixt7IvWppQhIeupjE3HP1BQ=; b=BK7YvxlIt3yiCaI0hu0CY83yLHoxUk5F3m2shcsfllmc3X+argVKdLpfwts2GBvEOi IVLm618phcJxaekt4K20Z2DQ34rsdI1AUjYvVcZnZ1iWfsXVN5lqLVj+vWrOLW3H7LPb BxRhkLEWag04xdpgJKmleu13ZFozjx8w+So4q2nkfD6rzCJBlyHCGW+1lqQl5hqGRYHV Oj4jq/erPwrbvVNVaTneRO9xCZWJJ0cCCoHxna6GhWipJvNior2jOfmRZGgge/u6LOwI 9tPyIVSmLVpCEw6aKA9AXiavOvhUTsJqdUXUQZIUjth6x8UM4/6/oiJFzBNZW9VmdvPU lFZg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=1yT9HCrQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ka7-20020a170903334700b001b53a3fbcc3si1269177plb.328.2023.07.20.11.14.12; Thu, 20 Jul 2023 11:14:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=1yT9HCrQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230220AbjGTRkK (ORCPT + 99 others); Thu, 20 Jul 2023 13:40:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229681AbjGTRkG (ORCPT ); Thu, 20 Jul 2023 13:40:06 -0400 Received: from mail-ej1-x649.google.com (mail-ej1-x649.google.com [IPv6:2a00:1450:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDDB9E52 for ; Thu, 20 Jul 2023 10:40:04 -0700 (PDT) Received: by mail-ej1-x649.google.com with SMTP id a640c23a62f3a-994320959f4so73994466b.2 for ; Thu, 20 Jul 2023 10:40:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689874803; x=1690479603; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sBTD77nZvv18xeO6UXUxdZ0TBIDcYAZAkWaNswyfelY=; b=1yT9HCrQQBAuoPLZ7UGNCLsWKvdxdYuhPi2NOHvNyCTSGgIEes/2FWg7ihwLHAhBfR 1PeLv9ipNyiodNyHDDK6BSNY8lzNRKw8Oix+MqAbxNmnzi9yiavk+N7YHSIgr0bFHzPs 8jV8LZHfabUsgSYAepexYCKMLMQ+dZQBsP2w2FU868WrtZgu64b0q/VPbS+dyJeruvFn BleAB1+spyaED66wcjfeaBbfoPl03mU6B1mt1b7bDlMZWQrCquq9K2CxeCXPiKIQWf4k IKxjpvdbWrbdBgYYm3DhUM1180bxFCEzrpPPJVvTv1C44fNSmbNxAKCG5CH6A9gChutu jrtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689874803; x=1690479603; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sBTD77nZvv18xeO6UXUxdZ0TBIDcYAZAkWaNswyfelY=; b=K7N7YvAA6BfU4Hyi8bh9JTFU3uqxrTGz3l2oLDhV7tDY+TGjrs7YRtKfwjVCkV2QbZ 5jgul+TDWyQTkrqDjQ/quhsmbQBjMe4wuDHqrXgfLyYJ4bDIhycdbgDYqig9gR7eaeVN /u9UDDmjikIVRxlE+EUAiquVTB/EgIHss0wxP0DcdlNjwpqrI9bGKUT7rh8DvyKyarSC uOoP5G0syDGzt3uyfN9nJbPkpMJwg8VtKhnpRYDtUv46XmdTclfZR2BR6Ay9dElZjFiw d9NQKX3bYmVxcsI640N5PzmWPjXsBzGCKJ1mxDUuY0xC8eJ/HHhZkSsObfDEUffdfDSY QquQ== X-Gm-Message-State: ABy/qLbIZQznpQoV/Kb7Bw9bCVgsf3ptwCPNsa/SzdD5urTKNkEAFQUl gmrGDHlSzXEdAU1jryZgExtize/qkWo= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:c495:c29a:e275:1dfb]) (user=glider job=sendgmr) by 2002:a50:cc9c:0:b0:51d:e3e6:cc6b with SMTP id q28-20020a50cc9c000000b0051de3e6cc6bmr25395edi.6.1689874803086; Thu, 20 Jul 2023 10:40:03 -0700 (PDT) Date: Thu, 20 Jul 2023 19:39:52 +0200 In-Reply-To: <20230720173956.3674987-1-glider@google.com> Mime-Version: 1.0 References: <20230720173956.3674987-1-glider@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230720173956.3674987-2-glider@google.com> Subject: [PATCH v4 1/5] lib/bitmap: add bitmap_{set,get}_value() From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org, Arnd Bergmann X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771964324765915414 X-GMAIL-MSGID: 1771964324765915414 From: Syed Nayyar Waris The two new functions allow setting/getting values of length up to BITS_PER_LONG bits at arbitrary position in the bitmap. The code was taken from "bitops: Introduce the for_each_set_clump macro" by Syed Nayyar Waris with a couple of minor changes: - instead of using roundup(), which adds an unnecessary dependency on , we calculate space as BITS_PER_LONG-offset; - indentation is reduced by not using else-clauses (suggested by checkpatch for bitmap_get_value()) Cc: Arnd Bergmann Signed-off-by: Syed Nayyar Waris Signed-off-by: William Breathitt Gray Link: https://lore.kernel.org/lkml/fe12eedf3666f4af5138de0e70b67a07c7f40338.1592224129.git.syednwaris@gmail.com/ Suggested-by: Yury Norov Co-developed-by: Alexander Potapenko Signed-off-by: Alexander Potapenko --- v4: - Address comments by Andy Shevchenko and Yury Norov: - prevent passing values >= 64 to GENMASK() - fix commit authorship - change comments - check for unlikely(nbits==0) - drop unnecessary const declarations - fix kernel-doc comments - rename bitmap_{get,set}_value() to bitmap_{read,write}() --- include/linux/bitmap.h | 60 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/include/linux/bitmap.h b/include/linux/bitmap.h index 03644237e1efb..bc21c09a2e038 100644 --- a/include/linux/bitmap.h +++ b/include/linux/bitmap.h @@ -76,7 +76,11 @@ struct device; * bitmap_to_arr32(buf, src, nbits) Copy nbits from buf to u32[] dst * bitmap_to_arr64(buf, src, nbits) Copy nbits from buf to u64[] dst * bitmap_get_value8(map, start) Get 8bit value from map at start + * bitmap_read(map, start, nbits) Read an nbits-sized value from + * map at start * bitmap_set_value8(map, value, start) Set 8bit value to map at start + * bitmap_write(map, value, start, nbits) Write an nbits-sized value to + * map at start * * Note, bitmap_zero() and bitmap_fill() operate over the region of * unsigned longs, that is, bits behind bitmap till the unsigned long @@ -583,6 +587,33 @@ static inline unsigned long bitmap_get_value8(const unsigned long *map, return (map[index] >> offset) & 0xFF; } +/** + * bitmap_read - read a value of n-bits from the memory region + * @map: address to the bitmap memory region + * @start: bit offset of the n-bit value + * @nbits: size of value in bits, up to BITS_PER_LONG + * + * Returns: value of nbits located at the @start bit offset within the @map + * memory region. + */ +static inline unsigned long bitmap_read(const unsigned long *map, + unsigned long start, + unsigned long nbits) +{ + size_t index = BIT_WORD(start); + unsigned long offset = start % BITS_PER_LONG; + unsigned long space = BITS_PER_LONG - offset; + unsigned long value_low, value_high; + + if (unlikely(!nbits)) + return 0; + if (space >= nbits) + return (map[index] >> offset) & GENMASK(nbits - 1, 0); + value_low = map[index] & BITMAP_FIRST_WORD_MASK(start); + value_high = map[index + 1] & BITMAP_LAST_WORD_MASK(start + nbits); + return (value_low >> offset) | (value_high << space); +} + /** * bitmap_set_value8 - set an 8-bit value within a memory region * @map: address to the bitmap memory region @@ -599,6 +630,35 @@ static inline void bitmap_set_value8(unsigned long *map, unsigned long value, map[index] |= value << offset; } +/** + * bitmap_write - write n-bit value within a memory region + * @map: address to the bitmap memory region + * @value: value of nbits + * @start: bit offset of the n-bit value + * @nbits: size of value in bits, up to BITS_PER_LONG + */ +static inline void bitmap_write(unsigned long *map, + unsigned long value, + unsigned long start, unsigned long nbits) +{ + size_t index = BIT_WORD(start); + unsigned long offset = start % BITS_PER_LONG; + unsigned long space = BITS_PER_LONG - offset; + + if (unlikely(!nbits)) + return; + value &= GENMASK(nbits - 1, 0); + if (space >= nbits) { + map[index] &= ~(GENMASK(nbits - 1, 0) << offset); + map[index] |= value << offset; + return; + } + map[index] &= ~BITMAP_FIRST_WORD_MASK(start); + map[index] |= value << offset; + map[index + 1] &= ~BITMAP_LAST_WORD_MASK(start + nbits); + map[index + 1] |= (value >> space); +} + #endif /* __ASSEMBLY__ */ #endif /* __LINUX_BITMAP_H */ From patchwork Thu Jul 20 17:39:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 123415 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3284805vqt; Thu, 20 Jul 2023 10:50:13 -0700 (PDT) X-Google-Smtp-Source: APBJJlFOY7u7qyx6ybtsLOl8M7PxIHTaC2CalZc7jYeqS50KsU0F8p6r9qO1LqfbS6xwVFLLl8ZY X-Received: by 2002:a17:902:c3c6:b0:1b8:944a:a932 with SMTP id j6-20020a170902c3c600b001b8944aa932mr215098plj.2.1689875413268; Thu, 20 Jul 2023 10:50:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689875413; cv=none; d=google.com; s=arc-20160816; b=buOziLqTThyeN2caoj/CYrQcYk3sAnjE+bj3LsdQHith4rqha9KCxPhhIELPOOqkjk 8C8fFlWy3DIdXCbdRcfqIvGaG3zGb7v+IK9+BaugTcrYnyJ17YVTwc9vKjUdrjzMY7g3 /QIj7hlmofY6+ijICdyn/zpqpABV62TdpOSVCem308RgtMwkAIWG9nSdoXeflXUxdDLk Vki6X3JMsrK4zSfLCAjNnOsxS7EsGeU1RWVNRGLhjUHuRjWZTg6mr/jTPwmRXhi8cWmq h8MPVZ55Q2LwY/JQ3o+Jd6Bw3xwSuw4QLaIpHpKrJIMymsnPxY8TFx5z/jf5jMqr+6Ht Y99Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Vk3m7k+eyF/HEh+8Emc9v/tBu8X0A5nY3rtePFTQyzI=; fh=O56S7FQoQXDYw0ENjMcjZy7yKrdXtC51zI+f09bWPts=; b=Zs6nZOhg/lJvulC1JtkBSLnDEuYZfKfdQSbAuu8Qu5X5uy1sdy6UvZIa+H+U/s2kVb 06L5TDpvq3KsFWcjY5fQAeSTuakvnQPMSYZ5jyAJzeT+gm3zAHycyAYG65bLEiAeXHg6 FUh6NA+hgagX0WYJEbrU0IeCB0TrtKeXRJNjjobEeZP94dzUTMsMWGig/UrX3keGMhz+ 1mI6Wdg2YSghppqshLWm2VscYMPKEKi3fUA/jyVO1QcqkrVo/BeN/mQit7B1575YeZRm 62zUIwYCT4dYqsoXXUrBh1CXF9uw4TiUN9s2AKZ2Jgplog6FLO0NVnkqPn9/P3aMpmlP MxXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=yuen82uP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u9-20020a1709026e0900b001bb21d502f0si1237200plk.76.2023.07.20.10.49.54; Thu, 20 Jul 2023 10:50:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=yuen82uP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231825AbjGTRkO (ORCPT + 99 others); Thu, 20 Jul 2023 13:40:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230144AbjGTRkI (ORCPT ); Thu, 20 Jul 2023 13:40:08 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08E1F1733 for ; Thu, 20 Jul 2023 10:40:07 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-56fffdea2d0so10150047b3.1 for ; Thu, 20 Jul 2023 10:40:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689874806; x=1690479606; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Vk3m7k+eyF/HEh+8Emc9v/tBu8X0A5nY3rtePFTQyzI=; b=yuen82uPoZ3tJbBBsgQocm7UlNUmQmKXoIc5PfKNJRK9D/8X+5MbFbx3Pocqqcbcfk sNkqbHJudzXLgp+t4oIBhs0mlZNEwiG7Z9MbyLn8U0aothHG2GpCZ5XX89a4i4LbNSTf gwObxpo3nrhm7ypSclv71wZ8tIxe/yWVeIQJtqEJyA8UxHTvSb1xAvfPSoEM5Aqyhdhc 5PEJUQQI0gSpBr2nS0lKnbv9VkCnVcWAb4UwwoMF/HT5PjyYSgL6RlVIj9EanViWBdcZ ki9cfNtxau3FCisOBQ8zKbF26CCYBygGeoR+cfUOaZC728vmoerlRdnOpQbgGD0ZF9oo QecQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689874806; x=1690479606; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Vk3m7k+eyF/HEh+8Emc9v/tBu8X0A5nY3rtePFTQyzI=; b=NHnwbguX5Q/7mbDVsPWyg8dgpgyHFM+sYUI6DanBWLlAd3YrDI7SHBPOxqzyPWre8w 7bmvqMkA0FHOs8ePHM4fkMymZjDZAxWyxpvNqibA+9+IE4R8MGPCZ4Yr6+W/YZCkAEUI +l24yUEpky2Lkrn8k5XCxqKdCFWdjfPd6NzuYfITu/F5yDogRd9kxLiR0EBlhJMfEm6e Iq5Cv90gsTMy9/N9QFOmabQT3233fuUPk3w/8in4M/GA30GlDTDJhYr6tKcJ1s1MIIKz IOlBZxhx6dmGEAMv3R9xWnmNP+Eun5Q/Uo60GLNETCMkWd/8zK8jCHXNoP/6NJi+9+SS H5lQ== X-Gm-Message-State: ABy/qLarKQ56fZ3WTyuXVTDRPzipMIxjAIFVzWD9TqaRvjTwoEFgYorV mc+JGc1BNFPbZhkjYR+BV0yTSRB7OY8= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:c495:c29a:e275:1dfb]) (user=glider job=sendgmr) by 2002:a05:6902:1005:b0:cb3:c343:19e5 with SMTP id w5-20020a056902100500b00cb3c34319e5mr56992ybt.2.1689874806295; Thu, 20 Jul 2023 10:40:06 -0700 (PDT) Date: Thu, 20 Jul 2023 19:39:53 +0200 In-Reply-To: <20230720173956.3674987-1-glider@google.com> Mime-Version: 1.0 References: <20230720173956.3674987-1-glider@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230720173956.3674987-3-glider@google.com> Subject: [PATCH v4 2/5] lib/test_bitmap: add tests for bitmap_{set,get}_value() From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771962801499885577 X-GMAIL-MSGID: 1771962801499885577 Add basic tests ensuring that values can be added at arbitrary positions of the bitmap, including those spanning into the adjacent unsigned longs. Signed-off-by: Alexander Potapenko Reviewed-by: Andy Shevchenko --- This patch was previously called "lib/test_bitmap: add tests for bitmap_{set,get}_value_unaligned" (https://lore.kernel.org/lkml/20230713125706.2884502-3-glider@google.com/) v4: - Address comments by Andy Shevchenko: added Reviewed-by: and a link to the previous discussion - Address comments by Yury Norov: - expand the bitmap to catch more corner cases - add code testing that bitmap_set_value() does not touch adjacent bits - add code testing the nbits==0 case - rename bitmap_{get,set}_value() to bitmap_{read,write}() v3: - switch to using bitmap_{set,get}_value() - change the expected bit pattern in test_set_get_value(), as the test was incorrectly assuming 0 is the LSB. --- lib/test_bitmap.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/lib/test_bitmap.c b/lib/test_bitmap.c index 187f5b2db4cf1..601000c7799df 100644 --- a/lib/test_bitmap.c +++ b/lib/test_bitmap.c @@ -71,6 +71,17 @@ __check_eq_uint(const char *srcfile, unsigned int line, return true; } +static bool __init +__check_eq_ulong(const char *srcfile, unsigned int line, + const unsigned long exp_ulong, unsigned long x) +{ + if (exp_ulong != x) { + pr_err("[%s:%u] expected %lu, got %lu\n", + srcfile, line, exp_ulong, x); + return false; + } + return true; +} static bool __init __check_eq_bitmap(const char *srcfile, unsigned int line, @@ -186,6 +197,7 @@ __check_eq_str(const char *srcfile, unsigned int line, }) #define expect_eq_uint(...) __expect_eq(uint, ##__VA_ARGS__) +#define expect_eq_ulong(...) __expect_eq(ulong, ##__VA_ARGS__) #define expect_eq_bitmap(...) __expect_eq(bitmap, ##__VA_ARGS__) #define expect_eq_pbl(...) __expect_eq(pbl, ##__VA_ARGS__) #define expect_eq_u32_array(...) __expect_eq(u32_array, ##__VA_ARGS__) @@ -1222,6 +1234,73 @@ static void __init test_bitmap_const_eval(void) BUILD_BUG_ON(~var != ~BIT(25)); } +/* + * Test bitmap should be big enough to include the cases when start is not in + * the first word, and start+nbits lands in the following word. + */ +#define TEST_BIT_LEN (BITS_PER_LONG * 3) +#define TEST_BYTE_LEN (BITS_TO_LONGS(TEST_BIT_LEN) * sizeof(unsigned long)) +static void __init test_set_get_value(void) +{ + DECLARE_BITMAP(bitmap, TEST_BIT_LEN); + DECLARE_BITMAP(exp_bitmap, TEST_BIT_LEN); + /* Prevent constant folding. */ + volatile unsigned long zero_bits = 0; + unsigned long val, bit; + int i; + + /* Setting/getting zero bytes should not crash the kernel. */ + bitmap_write(NULL, 0, 0, zero_bits); + val = bitmap_read(NULL, 0, zero_bits); + expect_eq_ulong(0, val); + + /* + * Ensure that bitmap_read() reads the same value that was previously + * written, and two consequent values are correctly merged. + * The resulting bit pattern is asymmetric to rule out possible issues + * with bit numeration order. + */ + for (i = 0; i < TEST_BIT_LEN - 7; i++) { + bitmap_zero(bitmap, TEST_BIT_LEN); + bitmap_write(bitmap, 0b10101UL, i, 5); + val = bitmap_read(bitmap, i, 5); + expect_eq_ulong(0b10101UL, val); + bitmap_write(bitmap, 0b101UL, i + 5, 3); + val = bitmap_read(bitmap, i + 5, 3); + expect_eq_ulong(0b101UL, val); + val = bitmap_read(bitmap, i, 8); + expect_eq_ulong(0b10110101UL, val); + } + + /* + * Check that setting a single bit does not accidentally touch the + * adjacent bits. + */ + for (i = 0; i < TEST_BIT_LEN; i++) { + /* + * A 0b10101010 pattern to catch both 0s replaced to 1s and vice + * versa. + */ + memset(bitmap, 0xaa, TEST_BYTE_LEN); + memset(exp_bitmap, 0xaa, TEST_BYTE_LEN); + for (bit = 0; bit <= 1; bit++) { + bitmap_write(bitmap, bit, i, 1); + __assign_bit(i, exp_bitmap, bit); + expect_eq_bitmap(exp_bitmap, bitmap, TEST_BIT_LEN); + } + } + + /* Ensure setting 0 bits does not change anything. */ + memset(bitmap, 0xaa, TEST_BYTE_LEN); + memset(exp_bitmap, 0xaa, TEST_BYTE_LEN); + for (i = 0; i < TEST_BIT_LEN; i++) { + bitmap_write(bitmap, ~0UL, i, 0); + expect_eq_bitmap(exp_bitmap, bitmap, TEST_BIT_LEN); + } +} +#undef TEST_BYTE_LEN +#undef TEST_BIT_LEN + static void __init selftest(void) { test_zero_clear(); @@ -1249,6 +1328,8 @@ static void __init selftest(void) test_for_each_clear_bitrange_from(); test_for_each_set_clump8(); test_for_each_set_bit_wrap(); + + test_set_get_value(); } KSTM_MODULE_LOADERS(test_bitmap); From patchwork Thu Jul 20 17:39:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 123416 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3287976vqt; Thu, 20 Jul 2023 10:57:18 -0700 (PDT) X-Google-Smtp-Source: APBJJlHxc9vIJBAu0ufQnIhVbuBb0A8QjxBYKTqTOr2/yvXJPfOb68d0PYbepmCHczLh9Cti6lYI X-Received: by 2002:a05:6512:704:b0:4fc:855f:2d17 with SMTP id b4-20020a056512070400b004fc855f2d17mr1764753lfs.54.1689875838103; Thu, 20 Jul 2023 10:57:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689875838; cv=none; d=google.com; s=arc-20160816; b=q843gtH3KCU5s9/foJ+AF9M9B+BrVpJOkMwIk4yRYT3ltO6EOUZpbwoKRb6IPOuoD/ ptnGQXdlfzxcOO/3mcHqBwwrTjyOhjQLmbMUS/smhhxsOcok47tnPXGuwxZGv84WgO5H kZakD4H2lPj7M7j3Pgvutu6DIjc/DqBBgA9UlE5amsxzc2oB6bUAjLQXeJI48ftRUsDw lJO1ANRrahI5ES/CF7zlTKyXvsaZMMEs+MXjgAnF+yiOCvIP5eS3E2mT9Qyqiq6kOdwW pAIn0GE7NZPQAPU4zT7Pt9dXC2RA/xb6ybK0xwXTsFBgTzz041uU7R5Et/aZiqEce+Xm HFBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=QyLcAjsRGHg3YdTbeFrMs0jC9ZNfsu8l2EPbq5hpOTw=; fh=O56S7FQoQXDYw0ENjMcjZy7yKrdXtC51zI+f09bWPts=; b=KQP3O5EwMDftIYHTB0utAKvZV9ARVRx8NpkVGdlBlrnMB747832L+6kNGxtzkjNXGs bC7tzkEd88kU+GimQkOwkTM801LT3kUqTt4OUzwFnSSCY2UoWGY/dRxoGROYqX6PN+qx R3Uugz+Ww3PzSSu6cOESKdoz4ZBNN81bSEI0PEKaV6+iRdM610Z3a/TgM4T6cMdfqGdN 0shXMZucFXmzDQHJb6ehhhUbC2qE82hpLoMHVaATfVBRbtz7SOA8TIrK9oi/JWuLrwv2 fdaeFQHjElVNebrGOH/LwQGBICP/Q13UFaH09HB+PED5sMAGrNNhS5WZtbpV7JjjNel2 v90g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=TKJeI6mT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w9-20020aa7cb49000000b00521d83e8d9esi1165477edt.150.2023.07.20.10.56.53; Thu, 20 Jul 2023 10:57:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=TKJeI6mT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231857AbjGTRkZ (ORCPT + 99 others); Thu, 20 Jul 2023 13:40:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230144AbjGTRkV (ORCPT ); Thu, 20 Jul 2023 13:40:21 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E90E31999 for ; Thu, 20 Jul 2023 10:40:10 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id a640c23a62f3a-993eeb3a950so77441466b.2 for ; Thu, 20 Jul 2023 10:40:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689874809; x=1690479609; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QyLcAjsRGHg3YdTbeFrMs0jC9ZNfsu8l2EPbq5hpOTw=; b=TKJeI6mT5cm/0eupLOKH/nUeOwO6BuvhLBiKaeZcjMCJ5Hc6RrSY+6Lq2g5Gtukby2 xMKphmMtRJV/BaWT0UgCI266KA/QNAof74zcrj/zO+/cAhMyBnGUr1JG9YkP0vD6jyHC YWJ/4hZmlBijJlXKnBK73sqSYrrvtzhT3MhJ94qZlvv23PVy2UtHWyJ84czbxXy6r709 UCb/LGzJbGWu3lceFDvomHS4pEw4IkCerhJtzAl777yM//4bHmpJ2xaQOe8ntPkxFUA2 VYLxxHh8yEAHbvM7PGVzKM70m8JaYeMeA3Rv4UUGmGedhPBBq+sX1ZPmezJbIxY0sCmz TA0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689874809; x=1690479609; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QyLcAjsRGHg3YdTbeFrMs0jC9ZNfsu8l2EPbq5hpOTw=; b=j5imSJqfgGzxVOAzGKffoGKL4Nga3woa5/XJ9VUL5NbGVOxFUewJaRDRuP/iZVDIC0 vxLiCDCffIVX9jap4WULK6EAbAwBijZqry/ydSJ4ELXx2W6fjJKLIQqj8MMhFukFg/AG fjMs1hrDD0hlh3/xqVESwiCtQ+NTsHWXV5TO9i1dKGDa3X/X9og7hmhaWh3sCUJZtCUp XUyBcYYVNKiFzMwV24Fin54sCwtuMUCzv7Zf2Mtzp8a5KAJeY4E4W/RrVQy49lUtoPbs Tmy5NUM/cTV0zzYVw2Mk8biKwLe82cdKyF1DLvjySWuzGuFxOWPQeroUpIzwt/Z8zMuZ 2FFQ== X-Gm-Message-State: ABy/qLYle+MtZXkeaUxBXHlndrM+oGysPLIqV/B905cJFvUQ8Hu7JVxa at1eajSa1QzjmDXRwM++WAUd8o4vDRM= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:c495:c29a:e275:1dfb]) (user=glider job=sendgmr) by 2002:a17:907:1de7:b0:99b:4866:e952 with SMTP id og39-20020a1709071de700b0099b4866e952mr10099ejc.7.1689874809321; Thu, 20 Jul 2023 10:40:09 -0700 (PDT) Date: Thu, 20 Jul 2023 19:39:54 +0200 In-Reply-To: <20230720173956.3674987-1-glider@google.com> Mime-Version: 1.0 References: <20230720173956.3674987-1-glider@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230720173956.3674987-4-glider@google.com> Subject: [PATCH v4 3/5] arm64: mte: implement CONFIG_ARM64_MTE_COMP From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771963246659283365 X-GMAIL-MSGID: 1771963246659283365 The config implements the algorithm compressing memory tags for ARM MTE during swapping. The algorithm is based on RLE and specifically targets 128-byte buffers of tags corresponding to a single page. In the common case a buffer can be compressed into 63 bits, making it possible to store it without additional memory allocation. Suggested-by: Evgenii Stepanov Signed-off-by: Alexander Potapenko --- v4: - Addressed comments by Andy Shevchenko: - expanded "MTE" to "Memory Tagging Extension" in Kconfig - fixed kernel-doc comments, moved them to C source - changed variables to unsigned where applicable - some code simplifications, fewer unnecessary assignments - added the mte_largest_idx_bits() helper - added namespace prefixes to all functions - added missing headers (but removed bits.h) - Addressed comments by Yury Norov: - removed test-only functions from mtecomp.h - dropped the algoritm name (all functions are now prefixed with "mte") - added more comments - got rid of MTE_RANGES_INLINE - renamed bitmap_{get,set}_value() to bitmap_{read,write}() - moved the big comment explaining the algorithm to Documentation/arch/arm64/mte-tag-compression.rst, expanded it, add a link to it from Documentation/arch/arm64/index.rst - removed hardcoded ranges from mte_alloc_size()/mte_size_to_ranges() v3: - Addressed comments by Andy Shevchenko: - use bitmap_{set,get}_value() writte by Syed Nayyar Waris - switched to unsigned long everywhere (fewer casts) - simplified the code, removed redundant checks - dropped ea0_compress_inline() - added bit size constants and helpers to access the bitmap - explicitly initialize all compressed sizes in ea0_compress_to_buf() - initialize all handle bits v2: - as suggested by Yury Norov, switched from struct bitq (which is not needed anymore) to - add missing symbol exports --- Documentation/arch/arm64/index.rst | 1 + .../arch/arm64/mte-tag-compression.rst | 212 ++++++++++ arch/arm64/Kconfig | 9 + arch/arm64/include/asm/mtecomp.h | 13 + arch/arm64/mm/Makefile | 1 + arch/arm64/mm/mtecomp.c | 382 ++++++++++++++++++ 6 files changed, 618 insertions(+) create mode 100644 Documentation/arch/arm64/mte-tag-compression.rst create mode 100644 arch/arm64/include/asm/mtecomp.h create mode 100644 arch/arm64/mm/mtecomp.c diff --git a/Documentation/arch/arm64/index.rst b/Documentation/arch/arm64/index.rst index d08e924204bf1..bf6c1583233a9 100644 --- a/Documentation/arch/arm64/index.rst +++ b/Documentation/arch/arm64/index.rst @@ -19,6 +19,7 @@ ARM64 Architecture legacy_instructions memory memory-tagging-extension + mte-tag-compression perf pointer-authentication ptdump diff --git a/Documentation/arch/arm64/mte-tag-compression.rst b/Documentation/arch/arm64/mte-tag-compression.rst new file mode 100644 index 0000000000000..af6716d53c1a8 --- /dev/null +++ b/Documentation/arch/arm64/mte-tag-compression.rst @@ -0,0 +1,212 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================================== +Tag Compression for Memory Tagging Extension (MTE) +================================================== + +This document describes the algorithm used to compress memory tags used by the +ARM Memory Tagging Extension (MTE) + +Introduction +============ + +MTE assigns tags to memory pages: for 4K pages those tags occupy 128 bytes +(256 4-bit tags each corresponding to a 16-byte MTE granule). By default, MTE +carves out 3.125% (1/16) of the available physical memory to store the tags. + +When MTE pages are saved to swap, their tags need to be stored in the kernel +memory. If the system swap is used heavily, these tags may take a substantial +portion of the physical memory, which in the case of a zram-backed swap may +even exceed the memory used to store the swapped pages themselves. To reduce +memory waste, ``CONFIG_ARM64_MTE_COMP`` allows the kernel to store the tags in +compressed form. + +Implementation details +====================== + +The algorithm attempts to compress a 128-byte (``MTE_GRANULES_PER_PAGE / 2``) +array of tags into a smaller byte sequence that can be stored in a +16-, 32-, or 64-byte buffer. A special case is storing the tags inline in +an 8-byte pointer. + +Tag manipulation and storage +---------------------------- + +Tags for swapped pages are stored in an XArray that maps swap entries to 63-bit +values (see ``arch/arm64/mm/mteswap.c``). In the case when +``CONFIG_ARM64_MTE_COMP=n``, these values contain pointers to 128-byte buffers +allocated with kmalloc(). Otherwise, they are 63-bit handles used by the +functions declared in ``arch/arm64/include/asm/mtecomp.h``: + +- mte_compress() compresses the given 128-byte ``tags`` buffer, allocates + storage for it, and returns an opaque handle addressing that storage; +- mte_decompress() decompresses the tags addressed by ``handle`` + and fills the 128-byte ``tags`` buffer; +- mte_release_handle() releases the storage handle returned by + mte_compress() (so that this handle cannot be used anymore); +- mte_storage_size() calculates the size occupied by the tags addressed + by ``handle``. + +Depending on the size of compressed data, ``mte_compress()`` stores it in one of +the size classes backed by kmem caches: ``mte-tags-16``, ``mte-tags-32``, +``mte-tags-64``, or ``mte-tags-128`` (for the data that cannot be compressed +into 64 bytes and is stored uncompressed). +A practical common case allows the tags to be compressed into 8 bytes - then +they are stored in the handle itself. + +Handle format +------------- + +The handle returned by ``mte_compress()`` is an ``unsigned long`` that has its +bit 63 set to 0 (XArray entries must not exceed ``LONG_MAX``):: + + 63 62 60 ... 2 0 + +---+--------+-----+------------+ + | 0 | INLINE | ... | SIZE_CLASS | + +---+--------+-----+------------+ + +Bits ``62..60`` is the inline/out-of-line marker: if they all are set to 1, the +data is stored out-of-line in the buffer pointed to by +``(handle | BIT(63)) & ~7UL``. Otherwise, the data is stored inline in the +handle itself. + +Bits ``2..0`` denote the size class for out-of-line allocations: + +- ``0b001`` for ``mte-tags-16``; +- ``0b010`` for ``mte-tags-32``; +- ``0b100`` for ``mte-tags-64``; +- ``0b000`` for ``mte-tags-128``. + + +Tag compression +--------------- + +The compression algorithm is a variation of RLE (run-length encoding) and works +as follows. + +1. The input array of 128 bytes is transformed into tag ranges (two arrays: + ``r_tags[]`` containing tag values and ``r_sizes[]`` containing range + lengths) by ``mte_tags_to_ranges()``. Note that ``r_sizes[]`` sums up to 256. + +2. The number of the largest element of ``r_sizes[]`` is stored in + ``largest_idx``. The element itself is thrown away from ``r_sizes[]``, + because it can be reconstructed from the sum of the remaining elements. Note + that now none of the remaining ``r_sizes[]`` elements exceeds 127. + +3. Depending on the number ``N`` of ranges, a storage class is picked:: + + N <= 6: 8 bytes (inline case, no allocation required); + 6 < N <= 11: 16 bytes + 11 < N <= 23: 32 bytes + 23 < N <= 46: 64 bytes + 46 < N: 128 bytes (no compression will be performed) + +(See `Why these numbers?`_ below). + +4. For the inline case, the following values are stored packed in the 8-byte + handle (``i`` means a ````-bit unsigned integer):: + + largest_idx : i4 + r_tags[0..5] : i4 x 6 + r_sizes[0..4] : i7 x 5 + + (if N is less than 6, ``r_tags`` and ``r_sizes`` are padded up with zero + values) + + Because ``largest_idx`` is <= 5, bit 63 of the handle is always 0 (so it + can be stored in the XArray), and bits 62..60 cannot all be 1, so it can be + distinguished from a kernel pointer. + +5. For the out-of-line case, the storage is allocated from one of the + ``mte-tags-{16,32,64,128}`` kmem caches. The resulting pointer is aligned + on 8 bytes, so its bits 2..0 can be used to store the size class (see above). + + Bit 63 of the pointer is zeroed out, so that it can be stored in XArray. + +6. The data layout in the allocated storage is as follows:: + + largest_idx : i6 + r_tags[0..N] : i4 x N + r_sizes[0..N-1] : i7 x (N-1) + +Tag decompression +----------------- + +The decompression algorithm performs the steps below. + +1. Decide if data is stored inline (bits ``62..60`` of the handle ``!= 0b111``) + or out-of line. + +2. For the inline case, treat the handle itself as the input buffer. + +3. For the out-of-line case, look at bits ``2..0`` of the handle to understand + the input buffer length. To obtain the pointer to the input buffer, unset + bits ``2..0`` of the handle and set bit ``63``. + +4. If the input buffer is 128 byte long, copy its contents to the output + buffer. + +5. Otherwise, read ``largest_idx``, ``r_tags[]`` and ``r_sizes[]`` from the + input buffer. Calculate the removed largest element of ``r_sizes[]`` as + ``largest = 256 - sum(r_sizes)`` and insert it into ``r_sizes`` at + position ``largest_idx``. + +6. For each ``r_sizes[i] > 0``, add a 4-bit value ``r_tags[i]`` to the output + buffer ``r_sizes[i]`` times. + + +Why these numbers? +------------------ + +To be able to reconstruct N tag ranges from the compressed data, we need to +store ``largest_idx``, ``r_tags[N]``, and ``r_sizes[N-1]``. Knowing that the +sizes do not exceed 127, those can be packed into 7 bits, whereas a single tag +occupies 4 bits, and ``largest_idx`` cannot take more than 8 bits. + +Now, for each ``S``-byte size class it is possible to find the maximal number +``M`` such as ``8 + 4 * M + 7 * (M - 1) <= 8 * S``, +i.e. ``M = (8 * S - 1) / 11``:: + + +-------------+----+--------------+ + | Buffer size | M | Storage bits | + +-------------+----+--------------+ + | 8 | 5 | 56 | + | 16 | 11 | 122 | + | 32 | 23 | 254 | + | 64 | 46 | 507 | + +-------------+----+--------------+ + +We can notice that ``M`` (and therefore ``largest_idx``) actually always fits +into 6 bits. For the inline case it is even guaranteed to fit into 3 bits, which +lets us squeeze an extra range into a 8-byte buffer. Because the inline case +requires bit 63 of the handle to be zero, we add that bit to ``largest_idx``, +knowing it will not be used. + +For the revised ``largest_idx`` sizes, we now pick the maximal number ``N`` +such as ``(L + 4 * N + 7 * (N - 1) <= 8 * S``, where ``L = 4`` in the inline +case and ``L = 6`` otherwise. +In other words, ``N = (8 * S + 7 - L) / 11``, therefore:: + + +-------------+----+--------------+ + | Buffer size | N | Storage bits | + +-------------+----+--------------+ + | 8 | 6 | 63 | + | 16 | 11 | 120 | + | 32 | 23 | 252 | + | 64 | 46 | 505 | + +-------------+----+--------------+ + + +Note +---- + +Tag compression and decompression implicitly rely on the fixed MTE tag size +(4 bits) and number of tags per page. Should these values change, the algorithm +may need to be revised. + + +Programming Interface +===================== + + .. kernel-doc:: arch/arm64/mm/mtecomp.c + diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2511b30d0f67..3ac6e302b1509 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2093,6 +2093,15 @@ config ARM64_EPAN if the cpu does not implement the feature. endmenu # "ARMv8.7 architectural features" +config ARM64_MTE_COMP + bool "Tag compression for ARM64 Memory Tagging Extension" + default y + depends on ARM64_MTE + help + Enable tag compression support for ARM64 Memory Tagging Extension. + + 128-byte tag buffers corresponding to 4K pages can be compressed to save heap memory. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/include/asm/mtecomp.h b/arch/arm64/include/asm/mtecomp.h new file mode 100644 index 0000000000000..71552bc429882 --- /dev/null +++ b/arch/arm64/include/asm/mtecomp.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_MTECOMP_H +#define __ASM_MTECOMP_H + +#include + +unsigned long mte_compress(u8 *tags); +bool mte_decompress(unsigned long handle, u8 *tags); +void mte_release_handle(unsigned long handle); +size_t mte_storage_size(unsigned long handle); + +#endif // __ASM_MTECOMP_H diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d0..46778f6dd83c2 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd.o obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o +obj-$(CONFIG_ARM64_MTE_COMP) += mtecomp.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/mtecomp.c b/arch/arm64/mm/mtecomp.c new file mode 100644 index 0000000000000..6d27f69f9521f --- /dev/null +++ b/arch/arm64/mm/mtecomp.c @@ -0,0 +1,382 @@ +// SPDX-License-Identifier: GPL-2.0-only + +/* + * MTE tag compression algorithm. + * Proposed by Evgenii Stepanov + * See Documentation/arch/arm64/mte-tag-compression.rst for more details. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +/* The handle must fit into an Xarray value. */ +#define MTE_HANDLE_MASK GENMASK_ULL(62, 0) + +/* Out-of-line handles have 0b111 in bits 62..60. */ +#define MTE_NOINLINE_MASK GENMASK_ULL(62, 60) + +/* Cache index is stored in the lowest pointer bits. */ +#define MTE_CACHE_ID_MASK GENMASK_ULL(2, 0) + +/* + * Four separate caches to store out-of-line data: + * 0: mte-tags-128 + * 1: mte-tags-16 + * 2: mte-tags-32 + * 3: mte-tags-64 + */ +#define MTECOMP_NUM_CACHES 4 +static struct kmem_cache *mtecomp_caches[MTECOMP_NUM_CACHES]; + +/* + * Sizes of compressed values. These depend on MTE_TAG_SIZE and + * MTE_GRANULES_PER_PAGE. + */ +#define MTE_BITS_PER_TAG 4 +#define MTE_BITS_PER_SIZE 7 +#define MTE_BITS_PER_LARGEST_IDX_INLINE 4 +#define MTE_BITS_PER_LARGEST_IDX 6 + +/* Translate allocation size into mtecomp_caches[] index. */ +static unsigned int mte_size_to_cache_id(size_t len) +{ + if (len < 128) + return fls(len >> 4); + return 0; +} + +/* Translate mtecomp_caches[] index into allocation size. */ +static size_t mte_cache_id_to_size(unsigned int id) +{ + if (id == 0) + return 128; + return 8 << id; +} + +/** + * mte_tags_to_ranges() - break @tags into arrays of tag ranges. + * @tags: 128-byte array containing 256 MTE tags. + * @out_tags: u8 array to store the tag of every range. + * @out_sizes: unsigned short array to store the size of every range. + * @out_len: length of @out_tags and @out_sizes (output parameter, initially + * equal to lengths of out_tags[] and out_sizes[]). + */ +void mte_tags_to_ranges(u8 *tags, u8 *out_tags, unsigned short *out_sizes, + size_t *out_len) +{ + u8 prev_tag = tags[0] / 16; /* First tag in the array. */ + unsigned int cur_idx = 0, i, j; + u8 cur_tag; + + memset(out_tags, 0, array_size(*out_len, sizeof(*out_tags))); + memset(out_sizes, 0, array_size(*out_len, sizeof(*out_sizes))); + + out_tags[0] = prev_tag; + for (i = 0; i < MTE_PAGE_TAG_STORAGE; i++) { + for (j = 0; j < 2; j++) { + cur_tag = j ? (tags[i] % 16) : (tags[i] / 16); + if (cur_tag == prev_tag) { + out_sizes[cur_idx]++; + } else { + cur_idx++; + prev_tag = cur_tag; + out_tags[cur_idx] = prev_tag; + out_sizes[cur_idx] = 1; + } + } + } + *out_len = cur_idx + 1; +} +EXPORT_SYMBOL_NS(mte_tags_to_ranges, MTECOMP); + +/** + * mte_ranges_to_tags() - fill @tags using given tag ranges. + * @r_tags: u8[] containing the tag of every range. + * @r_sizes: unsigned short[] containing the size of every range. + * @r_len: length of @r_tags and @r_sizes. + * @tags: 128-byte array to write the tags to. + */ +void mte_ranges_to_tags(u8 *r_tags, unsigned short *r_sizes, size_t r_len, + u8 *tags) +{ + unsigned int i, j, pos = 0; + u8 prev; + + for (i = 0; i < r_len; i++) { + for (j = 0; j < r_sizes[i]; j++) { + if (pos % 2) + tags[pos / 2] = (prev << 4) | r_tags[i]; + else + prev = r_tags[i]; + pos++; + } + } +} +EXPORT_SYMBOL_NS(mte_ranges_to_tags, MTECOMP); + +/* + * Translate allocation size into maximum number of ranges that it can hold. + * + * It is the biggest number N such as: + * 4 + 4*N + 7*(N-1) <= 63 bits, for the inline case, + * or + * 6 + 4*N + 7*(N-1) <= array size in bits (128, 256, or 512), + * for the out-of line case. + */ +static size_t mte_size_to_ranges(size_t size) +{ + size_t largest_bits; + size_t ret = 0; + + largest_bits = (size == 8) ? MTE_BITS_PER_LARGEST_IDX_INLINE : + MTE_BITS_PER_LARGEST_IDX; + ret = (size * 8 + MTE_BITS_PER_SIZE - largest_bits) / + (MTE_BITS_PER_TAG + MTE_BITS_PER_SIZE); + return ret; +} + +/* Translate @num_ranges into the allocation size needed to hold them. */ +static size_t mte_alloc_size(unsigned int num_ranges) +{ + size_t sizes[4] = { 8, 16, 32, 64 }; + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(sizes); i++) { + if (num_ranges <= mte_size_to_ranges(sizes[i])) + return sizes[i]; + } + return 128; +} + +/* Is the data stored inline in the handle itself? */ +static bool mte_is_inline(unsigned long handle) +{ + return (handle & MTE_NOINLINE_MASK) != MTE_NOINLINE_MASK; +} + +/** + * mte_storage_size() - calculate the memory occupied by compressed tags. + * @handle: storage handle returned by mte_compress. + * Returns: size of the storage used for @handle. + */ +size_t mte_storage_size(unsigned long handle) +{ + if (mte_is_inline(handle)) + return 8; + return mte_cache_id_to_size(handle & MTE_CACHE_ID_MASK); +} +EXPORT_SYMBOL_NS(mte_storage_size, MTECOMP); + +static void mte_bitmap_write(unsigned long *bitmap, unsigned long value, + unsigned long *pos, unsigned long bits) +{ + bitmap_write(bitmap, value, *pos, bits); + *pos += bits; +} + +static inline unsigned long mte_largest_idx_bits(size_t size) +{ + if (size == 8) + return MTE_BITS_PER_LARGEST_IDX_INLINE; + return MTE_BITS_PER_LARGEST_IDX; +} + +/* Compress ranges into the buffer that can accommodate up to max_ranges. */ +static void mte_compress_to_buf(size_t len, u8 *tags, unsigned short *sizes, + unsigned long *bitmap, size_t size) +{ + unsigned long bit_pos = 0, l_bits; + unsigned int largest_idx, i; + unsigned short largest = 0; + size_t max_ranges; + + for (i = 0; i < len; i++) { + if (sizes[i] > largest) { + largest = sizes[i]; + largest_idx = i; + } + } + l_bits = mte_largest_idx_bits(size); + max_ranges = mte_size_to_ranges(size); + mte_bitmap_write(bitmap, largest_idx, &bit_pos, l_bits); + for (i = 0; i < len; i++) + mte_bitmap_write(bitmap, tags[i], &bit_pos, MTE_BITS_PER_TAG); + for (i = len; i < max_ranges; i++) + mte_bitmap_write(bitmap, 0, &bit_pos, MTE_BITS_PER_TAG); + for (i = 0; i < len; i++) { + if (i != largest_idx) + mte_bitmap_write(bitmap, sizes[i], &bit_pos, + MTE_BITS_PER_SIZE); + } + for (i = len; i < max_ranges; i++) + mte_bitmap_write(bitmap, 0, &bit_pos, MTE_BITS_PER_SIZE); +} + +/** + * mte_compress() - compress the given tag array. + * @tags: 128-byte array to read the tags from. + * + * Compresses the tags and returns a 64-bit opaque handle pointing to the + * tag storage. May allocate memory, which is freed by @mte_release_handle(). + * Returns: 64-bit tag storage handle. + */ +unsigned long mte_compress(u8 *tags) +{ + unsigned short r_sizes[256]; + struct kmem_cache *cache; + unsigned long *storage; + unsigned int cache_id; + size_t alloc_size; + u8 r_tags[256]; + size_t r_len; + /* + * mte_compress_to_buf() only initializes the bits that mte_decompress() + * will read. But when the tags are stored in the handle itself, it must + * have all its bits initialized. + */ + unsigned long result = 0; + + r_len = sizeof(r_tags); + mte_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + alloc_size = mte_alloc_size(r_len); + if (alloc_size == 8) { + mte_compress_to_buf(r_len, r_tags, r_sizes, &result, + alloc_size); + return result; + } + cache_id = mte_size_to_cache_id(alloc_size); + cache = mtecomp_caches[cache_id]; + storage = kmem_cache_alloc(cache, GFP_KERNEL); + if (alloc_size < 128) { + /* alloc_size is always a multiple of sizeof(unsigned long). */ + mte_compress_to_buf(r_len, r_tags, r_sizes, storage, + alloc_size); + return ((unsigned long)storage | cache_id) & MTE_HANDLE_MASK; + } + memcpy(storage, tags, alloc_size); + return (unsigned long)storage & MTE_HANDLE_MASK; +} +EXPORT_SYMBOL_NS(mte_compress, MTECOMP); + +static unsigned long mte_bitmap_read(const unsigned long *bitmap, + unsigned long *pos, unsigned long bits) +{ + unsigned long result; + + result = bitmap_read(bitmap, *pos, bits); + *pos += bits; + return result; +} + +/* Decompress the contents of the given buffer into @tags. */ +static bool mte_decompress_from_buf(const unsigned long *bitmap, size_t size, + u8 *tags) +{ + unsigned short r_sizes[46], sum = 0; + unsigned long bit_pos = 0, l_bits; + unsigned int largest_idx, i; + size_t max_ranges; + u8 r_tags[46]; + + max_ranges = mte_size_to_ranges(size); + l_bits = mte_largest_idx_bits(size); + largest_idx = mte_bitmap_read(bitmap, &bit_pos, l_bits); + for (i = 0; i < max_ranges; i++) + r_tags[i] = mte_bitmap_read(bitmap, &bit_pos, MTE_BITS_PER_TAG); + for (i = 0; i < max_ranges; i++) { + if (i == largest_idx) + continue; + r_sizes[i] = + mte_bitmap_read(bitmap, &bit_pos, MTE_BITS_PER_SIZE); + if (!r_sizes[i]) { + max_ranges = i; + break; + } + sum += r_sizes[i]; + } + if (sum >= 256) + return false; + r_sizes[largest_idx] = 256 - sum; + mte_ranges_to_tags(r_tags, r_sizes, max_ranges, tags); + return true; +} + +/* Get pointer to the out-of-line storage from a handle. */ +static void *mte_storage(unsigned long handle) +{ + if (mte_is_inline(handle)) + return NULL; + return (void *)((handle & (~MTE_CACHE_ID_MASK)) | BIT_ULL(63)); +} + +/** + * mte_decompress() - decompress the tag array addressed by the handle. + * @handle: handle returned by @mte_decompress() + * @tags: 128-byte array to write the tags to. + * + * Reads the compressed data and writes it into the user-supplied tag array. + * Returns: true on success, false on error. + */ +bool mte_decompress(unsigned long handle, u8 *tags) +{ + unsigned long *storage = mte_storage(handle); + size_t size = mte_storage_size(handle); + + switch (size) { + case 8: + return mte_decompress_from_buf(&handle, size, tags); + case 128: + memcpy(tags, storage, size); + return true; + default: + return mte_decompress_from_buf(storage, size, tags); + } +} +EXPORT_SYMBOL_NS(mte_decompress, MTECOMP); + +/** + * mte_release_handle() - release the handle returned by mte_compress(). + * @handle: handle returned by mte_compress(). + */ +void mte_release_handle(unsigned long handle) +{ + struct kmem_cache *c; + void *storage; + size_t size; + + storage = mte_storage(handle); + if (!storage) + return; + + size = mte_storage_size(handle); + c = mtecomp_caches[mte_size_to_cache_id(size)]; + kmem_cache_free(c, storage); +} +EXPORT_SYMBOL_NS(mte_release_handle, MTECOMP); + +/* Set up mtecomp_caches[]. */ +static int mtecomp_init(void) +{ + unsigned int i; + char name[16]; + size_t size; + + static_assert(MTE_PAGE_TAG_STORAGE == 128); + static_assert(MTE_TAG_SIZE == MTE_BITS_PER_TAG); + for (i = 0; i < MTECOMP_NUM_CACHES; i++) { + size = mte_cache_id_to_size(i); + snprintf(name, sizeof(name), "mte-tags-%ld", size); + mtecomp_caches[i] = + kmem_cache_create(name, size, size, 0, NULL); + } + return 0; +} +module_init(mtecomp_init); From patchwork Thu Jul 20 17:39:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 123421 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3294432vqt; Thu, 20 Jul 2023 11:07:22 -0700 (PDT) X-Google-Smtp-Source: APBJJlF4Pc3OCQpCjyrtr6tu8jZm7Z23mrqKLJnoKMKLA3+vbSUN1lATdePJcnKEjq/TsL2KoclV X-Received: by 2002:a17:90a:b781:b0:263:5333:ca26 with SMTP id m1-20020a17090ab78100b002635333ca26mr3575pjr.29.1689876441923; Thu, 20 Jul 2023 11:07:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689876441; cv=none; d=google.com; s=arc-20160816; b=Aw8kTJEu/HwEm+XfCCjLE4ZVRf+iTWW9Spa0sfsxOhQFKGGnp3S8cHZRMA9XKSU4ya YY9V906eL6Wukw1D4WvTD5IS21rga/zgEkLiA5eZFc34cvmvQ0BtC+fvq+O3XUj8ysVD ymqQTXemoYEYaB2XEHeYr4cno9t8q9SQbUzngvfAvmUGup+CMovxvl+6XH4Y7Jp8Qgmn HEnqVsMVB7uCXqJ0hPchkj/HYZ9IeAdDVXUrNYIavafYSrSRPjonfdWGLHxWCD5hn6qA DZRCGN7wPJ13DjkvOkzLjU/Tu/Q4/CwAjshLVjDArPZiQzzKiMABMuAa8hHEA6ehov7Z GWLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=ZFdlPS8QU525qbseVaI8WzYFYY2eCpYrAzmO7CfAnMA=; fh=O56S7FQoQXDYw0ENjMcjZy7yKrdXtC51zI+f09bWPts=; b=hZNqzm5QLbaJBzOv4GdBN+KBNVJbu4jB/QO9LVQzyd7HFjarVWWi490RGA2X6fmJ5+ D4+PkOvay93+A4+WCT2cRi/w3PldzTqh+uCy/Qk4TsBGIsus/HIqYBLPj4GTnw9CgMO+ aWcTgf9LF12vdFuaGS+EyvYjuoF/JBkjocU3hG2x6QhJnpKSWb1QGSbt8GhxZgPRLFfv dZA+/yLBMT4Qzbmh0rapueJnG6Wk9v+aTNXqNYj2/mr01WmL43/Zv11t+nyp7x0NvoCa dZKIbtZgYzYgLaMElpwfq12cPnijSytEkKsL9rH2m7UzkfAgkZfLSn66hECBo4I8rOIW 2cPQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=gJXMrH2t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id np10-20020a17090b4c4a00b0026335941736si4409902pjb.144.2023.07.20.11.07.08; Thu, 20 Jul 2023 11:07:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=gJXMrH2t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231382AbjGTRk1 (ORCPT + 99 others); Thu, 20 Jul 2023 13:40:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231873AbjGTRkW (ORCPT ); Thu, 20 Jul 2023 13:40:22 -0400 Received: from mail-ej1-x64a.google.com (mail-ej1-x64a.google.com [IPv6:2a00:1450:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE6AC271E for ; Thu, 20 Jul 2023 10:40:13 -0700 (PDT) Received: by mail-ej1-x64a.google.com with SMTP id a640c23a62f3a-992e6840901so133130166b.0 for ; Thu, 20 Jul 2023 10:40:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689874812; x=1690479612; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZFdlPS8QU525qbseVaI8WzYFYY2eCpYrAzmO7CfAnMA=; b=gJXMrH2tT491ow+xnauMdpysCLCAJ+RDEqD/SLy8C4UJb3ZHOSt+HPScELWiPnd6Hl EgPVzLs9fAJxMtb9Wq/SxTw3WytI13SVmdB8ORp+iTzTCcxDvlsKylsoui07QxQDInAD /9FMahDgy322UqZZ75gamiyd8g0G2vz5W/jVlwICHmroyaH/dHKSYTV4aNFxQC03JJna likRyW3LmN0QMyvv1FGPqtSV36h9M+3NjKDniqhaokMRI7LAP642BRS+yNxhzF59nEa2 P45LzS/pBT/tgHOWdHlIvkWorFdwQgBTOK2XkBkRvE60ydAvvvczDdq0r/nO1GulTWSP Bbbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689874812; x=1690479612; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZFdlPS8QU525qbseVaI8WzYFYY2eCpYrAzmO7CfAnMA=; b=QFwtr8IhU4kMi/A/F/LcHSI1E2zvSMTsgdB2+NdGFKc5r4nV5UCJWI333g4td2N3xQ L5K5Ts78ofB0bAr6SrYM1UeTZUSn8AS9CmBz70PrybWPfBDPoxFSQ2b2i3Im5ON8K5YC /xscwAnGly4bJIgcxw/u2pXnjzbEFfXL175VxmbxlnzoKOsNdXY3kRUbAgGSzrPtCn2M hA90igDy5kJY/bMezl3sC2XGq+HR6+L9aFo1z4GzfgKb+2zpeuodUOnMBERP2TLmg7s6 3NMam2vaJmSFxYS1bIUPWtKnq+lCDEFSkZpxYhw65eCVmFtTXLXTnTID9PUX/O21hTQg koiQ== X-Gm-Message-State: ABy/qLYYv+bOF4+a89CFnHJPqMrjYS/i9YTImrvyqzxHbqp2laf9n2Td byOZfgBkOIM0XFEfxIAtFhyKvlsp1M8= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:c495:c29a:e275:1dfb]) (user=glider job=sendgmr) by 2002:a17:907:2ccc:b0:98e:4bf5:8fd6 with SMTP id hg12-20020a1709072ccc00b0098e4bf58fd6mr28414ejc.6.1689874812257; Thu, 20 Jul 2023 10:40:12 -0700 (PDT) Date: Thu, 20 Jul 2023 19:39:55 +0200 In-Reply-To: <20230720173956.3674987-1-glider@google.com> Mime-Version: 1.0 References: <20230720173956.3674987-1-glider@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230720173956.3674987-5-glider@google.com> Subject: [PATCH v4 4/5] arm64: mte: add a test for MTE tags compression From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771963879644601832 X-GMAIL-MSGID: 1771963879644601832 Ensure that tag sequences containing alternating values are compressed to buffers of expected size and correctly decompressed afterwards. Signed-off-by: Alexander Potapenko --- v4: - addressed comments by Andy Shevchenko: - expanded MTE to "Memory Tagging Extension" in Kconfig - changed signed variables to unsigned where applicable - added missing header dependencies - addressed comments by Yury Norov: - moved test-only declarations from mtecomp.h into this test - switched to the new "mte"-prefixed function names, dropped the mentions of "EA0" - added test_tag_to_ranges_n() v3: - addressed comments by Andy Shevchenko in another patch: - switched from u64 to unsigned long - added MODULE_IMPORT_NS(MTECOMP) - fixed includes order --- arch/arm64/Kconfig | 10 ++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/test_mtecomp.c | 217 +++++++++++++++++++++++++++++++++++ 3 files changed, 228 insertions(+) create mode 100644 arch/arm64/mm/test_mtecomp.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3ac6e302b1509..2432475003054 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2102,6 +2102,16 @@ config ARM64_MTE_COMP 128-byte tag buffers corresponding to 4K pages can be compressed to save heap memory. +config ARM64_MTE_COMP_KUNIT_TEST + tristate "Test tag compression for ARM64 Memory Tagging Extension" if !KUNIT_ALL_TESTS + default KUNIT_ALL_TESTS + depends on KUNIT && ARM64_MTE_COMP + help + Test MTE compression algorithm enabled by CONFIG_ARM64_MTE_COMP. + + Ensure that tag sequences containing alternating values are compressed + to buffers of expected size and correctly decompressed afterwards. + config ARM64_SVE bool "ARM Scalable Vector Extension support" default y diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 46778f6dd83c2..170dc62b010b9 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,7 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) += mtecomp.o +obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) += test_mtecomp.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/test_mtecomp.c b/arch/arm64/mm/test_mtecomp.c new file mode 100644 index 0000000000000..185eb6cb73650 --- /dev/null +++ b/arch/arm64/mm/test_mtecomp.c @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Test cases for MTE tags compression algorithm. + */ + +#include +#include +#include +#include +#include +#include + +#include + +/* Functions exported from mtecomp.c for this test. */ +void mte_tags_to_ranges(u8 *tags, u8 *out_tags, unsigned short *out_sizes, + size_t *out_len); +void mte_ranges_to_tags(u8 *r_tags, unsigned short *r_sizes, size_t r_len, + u8 *tags); + +/* + * Test that mte_tags_to_ranges() produces a single range for a zero-filled tag + * buffer. + */ +static void test_tags_to_ranges_zero(struct kunit *test) +{ + u8 tags[128], dtags[128]; + unsigned short r_sizes[256]; + size_t r_len = 256; + u8 r_tags[256]; + + memset(tags, 0, 128); + mte_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 1); + KUNIT_EXPECT_EQ(test, r_tags[0], 0); + KUNIT_EXPECT_EQ(test, r_sizes[0], 256); + mte_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a small number of different tags is correctly transformed into + * ranges. + */ +static void test_tags_to_ranges_simple(struct kunit *test) +{ + u8 tags[128], dtags[128]; + const u8 ex_tags[] = { 0xa, 0x0, 0xa, 0xb, 0x0 }; + const unsigned short ex_sizes[] = { 1, 2, 2, 1, 250 }; + unsigned short r_sizes[256]; + size_t r_len = 256; + u8 r_tags[256]; + + memset(tags, 0, 128); + tags[0] = 0xa0; + tags[1] = 0x0a; + tags[2] = 0xab; + mte_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 5); + KUNIT_EXPECT_EQ(test, memcmp(r_tags, ex_tags, sizeof(ex_tags)), 0); + KUNIT_EXPECT_EQ(test, memcmp(r_sizes, ex_sizes, sizeof(ex_sizes)), 0); + mte_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* Test that repeated 0xa0 byte produces 256 ranges of length 1. */ +static void test_tags_to_ranges_repeated(struct kunit *test) +{ + u8 tags[128], dtags[128]; + unsigned short r_sizes[256]; + size_t r_len = 256; + u8 r_tags[256]; + + memset(tags, 0xa0, 128); + mte_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + KUNIT_EXPECT_EQ(test, r_len, 256); + mte_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* Generate a buffer that will contain @nranges of tag ranges. */ +static void gen_tag_range_helper(u8 *tags, int nranges) +{ + unsigned int i; + + memset(tags, 0, 128); + if (nranges > 1) { + nranges--; + for (i = 0; i < nranges / 2; i++) + tags[i] = 0xab; + if (nranges % 2) + tags[nranges / 2] = 0xa0; + } +} + +/* + * Test that mte_tags_to_ranges()/mte_ranges_to_tags() work for various + * r_len values. + */ +static void test_tag_to_ranges_n(struct kunit *test) +{ + unsigned short r_sizes[256]; + u8 tags[128], dtags[128]; + unsigned int i, j, sum; + size_t r_len = 256; + u8 r_tags[256]; + + for (i = 1; i <= 256; i++) { + gen_tag_range_helper(tags, i); + mte_tags_to_ranges(tags, r_tags, r_sizes, &r_len); + sum = 0; + for (j = 0; j < r_len; j++) + sum += r_sizes[j]; + KUNIT_EXPECT_EQ(test, sum, 256); + mte_ranges_to_tags(r_tags, r_sizes, r_len, dtags); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); + } +} + +/* Test that a zero-filled array is compressed into inline storage. */ +static void test_compress_zero(struct kunit *test) +{ + u8 tags[128], dtags[128]; + unsigned long handle; + + memset(tags, 0, 128); + handle = mte_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + /* Tags are stored inline. */ + KUNIT_EXPECT_EQ(test, mte_storage_size(handle), 8); + KUNIT_EXPECT_TRUE(test, mte_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a very small number of tag ranges ends up compressed into 8 bytes. + */ +static void test_compress_simple(struct kunit *test) +{ + u8 tags[128], dtags[128]; + unsigned long handle; + + memset(tags, 0, 128); + tags[0] = 0xa0; + tags[1] = 0x0a; + tags[2] = 0xab; + + handle = mte_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + /* Tags are stored inline. */ + KUNIT_EXPECT_EQ(test, mte_storage_size(handle), 8); + KUNIT_EXPECT_TRUE(test, mte_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that a buffer containing @nranges ranges compresses into @exp_size + * bytes and decompresses into the original tag sequence. + */ +static void compress_range_helper(struct kunit *test, int nranges, + size_t exp_size) +{ + u8 tags[128], dtags[128]; + unsigned long handle; + + gen_tag_range_helper(tags, nranges); + handle = mte_compress(tags); + KUNIT_EXPECT_EQ(test, handle & BIT_ULL(63), 0); + KUNIT_EXPECT_EQ(test, mte_storage_size(handle), exp_size); + KUNIT_EXPECT_TRUE(test, mte_decompress(handle, dtags)); + KUNIT_EXPECT_EQ(test, memcmp(tags, dtags, 128), 0); +} + +/* + * Test that every number of tag ranges is correctly compressed and + * decompressed. + */ +static void test_compress_ranges(struct kunit *test) +{ + size_t exp_size; + unsigned int i; + + for (i = 1; i <= 256; i++) { + if (i < 7) + exp_size = 8; + else if (i < 12) + exp_size = 16; + else if (i < 24) + exp_size = 32; + else if (i < 47) + exp_size = 64; + else + exp_size = 128; + compress_range_helper(test, i, exp_size); + } +} + +static struct kunit_case mtecomp_test_cases[] = { + KUNIT_CASE(test_tags_to_ranges_zero), + KUNIT_CASE(test_tags_to_ranges_simple), + KUNIT_CASE(test_tags_to_ranges_repeated), + KUNIT_CASE(test_tag_to_ranges_n), + KUNIT_CASE(test_compress_zero), + KUNIT_CASE(test_compress_simple), + KUNIT_CASE(test_compress_ranges), + {} +}; + +static struct kunit_suite mtecomp_test_suite = { + .name = "mtecomp", + .test_cases = mtecomp_test_cases, +}; +kunit_test_suites(&mtecomp_test_suite); + +MODULE_IMPORT_NS(MTECOMP); +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Alexander Potapenko "); From patchwork Thu Jul 20 17:39:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 123420 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp3293324vqt; Thu, 20 Jul 2023 11:05:41 -0700 (PDT) X-Google-Smtp-Source: APBJJlHERa2d/Mv1B4NzQM1g/5Up/Uv0ya69j/CtA9zNRfTO5A6yJA7IcMO+lyWR7Zx3Y87wCiM4 X-Received: by 2002:a17:90b:17ca:b0:259:3e2a:b6d8 with SMTP id me10-20020a17090b17ca00b002593e2ab6d8mr214107pjb.17.1689876341648; Thu, 20 Jul 2023 11:05:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689876341; cv=none; d=google.com; s=arc-20160816; b=S1b7TM8n+qEN74XwC75AgHDlpPp8eIb4gt+osTE2QVgP+CAFzTA36FZz5dI59OOG6t d1d3HbXdAGHKlDjFg0xIlNzhfj5mN4tNI1ZjPhLSQ3GtJi/rLsj8YOtt8oyXjsjREjBC AlzSduNvfQ4COGbMHmG4qdhrKGCCVyC56rooxJzemM8E/AOM0PYGazwWzFslsLIBaFMR VXkXVNVKtIbY7Yyhk7bYg/yuUsa1kgXMJ88/UCtw632ZruXLJz/K0X5scG7Pfzw89BIi m6nyiyo8gic/yEuxAY6aNgsmNTcjXzdzgftBCHbYV+ksKi8ynQ1B/4pbHkWtfBsXzRIo XQ3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=lAaXjy37mqyNYrDfUW3C83NPs/bBQ+S9RZjTrhTp6QU=; fh=O56S7FQoQXDYw0ENjMcjZy7yKrdXtC51zI+f09bWPts=; b=tBwjKeqSe1klr5g8dRUHYVCE+61rijvB/2S+NsFwn6mNaqA/lGu7Z2khU4fgefZdHA kAeDj/hoPQnEH2Lw62L7r0vFMjwpyTcuS399RvYuLY8PcLr9DeF3gqy2OwSqS5dc5FRk iJtN36khQvcG2rU2cDQk+XZc6a/7l0GO92qvnluiIYtmbbdBAKE7T8SACZgBn4bIcNSU sYix4jxSAlnJPin31WPBqNWbLNeK2O9Rtp9r88EAnzuxoX4zbZWVzpOtb31vu9pl7ARj 3JO7MCnWZq6wfdwdoMTGRgOD7bh22/ntb/SQuizkfDUYWvGW6Sx6LyvdnmOViGcQ3VRq n1Cw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=zvov4G1A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a17-20020a17090a8c1100b00262e3c1c1efsi1564518pjo.79.2023.07.20.11.04.48; Thu, 20 Jul 2023 11:05:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=zvov4G1A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231892AbjGTRka (ORCPT + 99 others); Thu, 20 Jul 2023 13:40:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231908AbjGTRkX (ORCPT ); Thu, 20 Jul 2023 13:40:23 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 434002D4E for ; Thu, 20 Jul 2023 10:40:16 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-cab7304dcccso892664276.3 for ; Thu, 20 Jul 2023 10:40:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689874815; x=1690479615; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=lAaXjy37mqyNYrDfUW3C83NPs/bBQ+S9RZjTrhTp6QU=; b=zvov4G1AddGuBx6dxgnHz1dfQTZYxS2Sku8hEaIkJt2ZfSa2XFJAy105OJ3+2HbgHO CK20GvLdXNFgwWeFdV6xfHn5xrNIssdGFzaB3jwEu6NUQd8KRmhob69JtqyFihA29r8t ac9Uu92/BuSJNL70kGCoRRHf2PMVEiNkCAU7ZdLY+w378uW08vPShyu1scQ+WiThCPkC x6RbZ6Hjv+4BmyHqnZYSQkwXbjgx1g4yEMkb0971xXGk9drbWp98YC5cG/yqGVjMHy9k 0PTiA8RLSWU1L5YqF+ISCzKPKrla6lPq6N/c+Qm4KntLbx3ik5jvsA269Tnktr/4AFRa oJiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689874815; x=1690479615; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lAaXjy37mqyNYrDfUW3C83NPs/bBQ+S9RZjTrhTp6QU=; b=YdeL7snuUpYa06Ej7J3+jlyQ8Fl2lRSN88Yji/PqlE2BaLLIO/eGFsDtqCfjy2JttZ tGR2rkFooIqwiN5t0OIlb+IE34vioU/nT6iljPQBWri3+yQkMA+RdnvPcRuIFS76Bden T+6nttTsPR3f1TdzTSnh8f2Y0D+J9RSzqu++n7urakWc6RjSRjGDt8F2MiGVuzsU5Aoj H0AOhkBJE/GXXhccSfm6LFb3r45lU1mwZpm6GY5B3n5cwyk63kfdil7rOurmIIONFUWC IlJj0vaBcuq9tNa6NwG2mw2HUXEB0JHnvpcG1rI9rBd32oeWdZcx1IxKa+j6mCBwI3G2 3/mw== X-Gm-Message-State: ABy/qLa5J856U/vNh06GhLhSCyok1LPKTHmo0vuQldMw24XxQUKG0K2o VAZoQLW3CoCuFglFg9zlBF3opeMZbcE= X-Received: from glider.muc.corp.google.com ([2a00:79e0:9c:201:c495:c29a:e275:1dfb]) (user=glider job=sendgmr) by 2002:a25:2054:0:b0:c85:934:7ad2 with SMTP id g81-20020a252054000000b00c8509347ad2mr20280ybg.8.1689874815149; Thu, 20 Jul 2023 10:40:15 -0700 (PDT) Date: Thu, 20 Jul 2023 19:39:56 +0200 In-Reply-To: <20230720173956.3674987-1-glider@google.com> Mime-Version: 1.0 References: <20230720173956.3674987-1-glider@google.com> X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230720173956.3674987-6-glider@google.com> Subject: [PATCH v4 5/5] arm64: mte: add compression support to mteswap.c From: Alexander Potapenko To: glider@google.com, catalin.marinas@arm.com, will@kernel.org, pcc@google.com, andreyknvl@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, yury.norov@gmail.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, eugenis@google.com, syednwaris@gmail.com, william.gray@linaro.org X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771963774769651022 X-GMAIL-MSGID: 1771963774769651022 Define the internal mteswap.h interface: - _mte_alloc_and_save_tags() - _mte_free_saved_tags() - _mte_restore_tags() , that encapsulates saving tags for a struct page (together with memory allocation), restoring tags, and deleting the storage allocated for them. These functions accept opaque pointers, which may point to 128-byte tag buffers, as well as smaller buffers containing compressed tags, or have compressed tags stored directly in them. The existing code from mteswap.c operating with uncompressed tags is split away into mteswap_nocomp.c, and the newly introduced mteswap_comp.c provides compression support. The latter implementation is picked if CONFIG_ARM64_MTE_COMP=y. Soon after booting Android, tag compression saves ~2.5x memory previously spent by mteswap.c on tag allocations. With the growing uptime, the savings reach 20x and even more. Signed-off-by: Alexander Potapenko --- v4: - minor code simplifications suggested by Andy Shevchenko, added missing header dependencies - changed compression API names to reflect modifications made to memcomp.h (as suggested by Yury Norov) v3: - Addressed comments by Andy Shevchenko in another patch: - fixed includes order - replaced u64 with unsigned long - added MODULE_IMPORT_NS(MTECOMP) --- arch/arm64/mm/Makefile | 5 ++++ arch/arm64/mm/mteswap.c | 20 ++++++------- arch/arm64/mm/mteswap.h | 12 ++++++++ arch/arm64/mm/mteswap_comp.c | 54 ++++++++++++++++++++++++++++++++++ arch/arm64/mm/mteswap_nocomp.c | 38 ++++++++++++++++++++++++ 5 files changed, 118 insertions(+), 11 deletions(-) create mode 100644 arch/arm64/mm/mteswap.h create mode 100644 arch/arm64/mm/mteswap_comp.c create mode 100644 arch/arm64/mm/mteswap_nocomp.c diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 170dc62b010b9..46a798e2b67cb 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -11,6 +11,11 @@ obj-$(CONFIG_TRANS_TABLE) += trans_pgd-asm.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_ARM64_MTE) += mteswap.o obj-$(CONFIG_ARM64_MTE_COMP) += mtecomp.o +ifdef CONFIG_ARM64_MTE_COMP +obj-$(CONFIG_ARM64_MTE) += mteswap_comp.o +else +obj-$(CONFIG_ARM64_MTE) += mteswap_nocomp.o +endif obj-$(CONFIG_ARM64_MTE_COMP_KUNIT_TEST) += test_mtecomp.o KASAN_SANITIZE_physaddr.o += n diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index cd508ba80ab1b..9d8f87fd191a2 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -5,8 +5,11 @@ #include #include #include + #include +#include "mteswap.h" + static DEFINE_XARRAY(mte_pages); void *mte_allocate_tag_storage(void) @@ -27,20 +30,18 @@ int mte_save_tags(struct page *page) if (!page_mte_tagged(page)) return 0; - tag_storage = mte_allocate_tag_storage(); + tag_storage = _mte_alloc_and_save_tags(page); if (!tag_storage) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); - /* page_private contains the swap entry.val set in do_swap_page */ ret = xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + _mte_free_saved_tags(tag_storage); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + _mte_free_saved_tags(ret); } return 0; @@ -53,10 +54,7 @@ void mte_restore_tags(swp_entry_t entry, struct page *page) if (!tags) return; - if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); - set_page_mte_tagged(page); - } + _mte_restore_tags(tags, page); } void mte_invalidate_tags(int type, pgoff_t offset) @@ -64,7 +62,7 @@ void mte_invalidate_tags(int type, pgoff_t offset) swp_entry_t entry = swp_entry(type, offset); void *tags = xa_erase(&mte_pages, entry.val); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } void mte_invalidate_tags_area(int type) @@ -78,7 +76,7 @@ void mte_invalidate_tags_area(int type) xa_lock(&mte_pages); xas_for_each(&xa_state, tags, last_entry.val - 1) { __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + _mte_free_saved_tags(tags); } xa_unlock(&mte_pages); } diff --git a/arch/arm64/mm/mteswap.h b/arch/arm64/mm/mteswap.h new file mode 100644 index 0000000000000..4c576b76785d1 --- /dev/null +++ b/arch/arm64/mm/mteswap.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef ARCH_ARM64_MM_MTESWAP_H_ +#define ARCH_ARM64_MM_MTESWAP_H_ + +struct page; + +void *_mte_alloc_and_save_tags(struct page *page); +void _mte_free_saved_tags(void *tags); +void _mte_restore_tags(void *tags, struct page *page); + +#endif // ARCH_ARM64_MM_MTESWAP_H_ diff --git a/arch/arm64/mm/mteswap_comp.c b/arch/arm64/mm/mteswap_comp.c new file mode 100644 index 0000000000000..2c4ac5eac9e59 --- /dev/null +++ b/arch/arm64/mm/mteswap_comp.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management with compression. */ + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + u8 tags[128]; + unsigned long handle; + + mte_save_page_tags(page_address(page), tags); + handle = mte_compress(tags); + return xa_mk_value(handle); +} + +void _mte_free_saved_tags(void *storage) +{ + unsigned long handle; + int size; + + handle = xa_to_value(storage); + if (!handle) + return; + size = mte_storage_size(handle); + mte_release_handle(handle); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + unsigned long handle; + u8 tags_decomp[128]; + + handle = xa_to_value(tags); + if (!handle) + return; + if (!try_page_mte_tagging(page)) + return; + if (!mte_decompress(handle, tags_decomp)) + return; + mte_restore_page_tags(page_address(page), tags_decomp); + set_page_mte_tagged(page); +} +MODULE_IMPORT_NS(MTECOMP); diff --git a/arch/arm64/mm/mteswap_nocomp.c b/arch/arm64/mm/mteswap_nocomp.c new file mode 100644 index 0000000000000..1e665a4b5f940 --- /dev/null +++ b/arch/arm64/mm/mteswap_nocomp.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* MTE tag storage management without compression support. */ + +#include +#include +#include +#include +#include + +#include + +#include "mteswap.h" + +void *_mte_alloc_and_save_tags(struct page *page) +{ + void *storage; + + storage = mte_allocate_tag_storage(); + if (!storage) + return NULL; + + mte_save_page_tags(page_address(page), storage); + return storage; +} + +void _mte_free_saved_tags(void *storage) +{ + mte_free_tag_storage(storage); +} + +void _mte_restore_tags(void *tags, struct page *page) +{ + if (!try_page_mte_tagging(page)) + return; + mte_restore_page_tags(page_address(page), tags); + set_page_mte_tagged(page); +}