From patchwork Thu Oct 27 20:10:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 11940 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp437606wru; Thu, 27 Oct 2022 13:19:37 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5MLVGVQbjzoJqAEF86QFtJN+aGzMdbD5JM2oo5AdLWu3gl1uicZaR1D4ZRsMoqA8F+QluF X-Received: by 2002:a63:5559:0:b0:464:a987:8365 with SMTP id f25-20020a635559000000b00464a9878365mr9772386pgm.479.1666901977205; Thu, 27 Oct 2022 13:19:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666901977; cv=none; d=google.com; s=arc-20160816; b=N4HYhpWFmgK4DP3sgdjC9eEmZQve7/pD6fPwQ8IMOwkfugDX1Karn4xxKw892Wd3O9 TtfGMvacoq2uiMDWqR3xCdYvoid76sQyT+vTuWLgpEg2aYP3rZDamO1hpD70rGVChjmo iU/JE1Up9amSbt1/yddtQ1rkHV6RzKU93JPk4fKK+a80uO1EVaVAoyTFLldPjNqXebi8 p3h9a9XBtdZs3xgCySRst2ERTqES/DwEDDbhGFnhubo49fceLvKYeh+Gt/WkTY/trlaA 76DxG+vGr7lhxwNLbc2MxXjVw2iPvrVvQRpaB1DX9yUij0cD81jVmA8ZAnoKqLyiFQb/ Rhgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=OIqcZRUcgxs1qDADU8Zu6W54GU95YqrDSNAf8mGexD8=; b=YijlcZ+rwKOtzH81HHd4OEGAFxWNmotAtZqi9ceqTSDv8lEPalj+eyqaHltUaJZh71 QTUNkCTLyRJFXd9M0Bl7tk0JBkkZ0KBvEorSbmtBC1sHW2vcyajfemZyrmDMrEh3gjvR NGXt7GMkkqc4bz3ZOChfVjbX7kA6KEwRXgljkhlaaSVkIz21CbNjINh5DCIh4JAbZgHH Gi2OFcER02ZBhAP1hUMtoFuBv08Qc+ZqGifmzv8fKaA37yWfEzoLKAlPtBwWSss9Y0fU JTR3LGNHn2xmcwbSkJ0asrO+YbFUvUSlJATk+V/rQvSEFso4B2ePyO+YMbXXC3VYtNv1 5hyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=MLRzYvvq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e18-20020a17090301d200b001768452d4d0si3137332plh.30.2022.10.27.13.19.23; Thu, 27 Oct 2022 13:19:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=MLRzYvvq; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236838AbiJ0UKa (ORCPT + 99 others); Thu, 27 Oct 2022 16:10:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60484 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236477AbiJ0UKU (ORCPT ); Thu, 27 Oct 2022 16:10:20 -0400 Received: from out0.migadu.com (out0.migadu.com [94.23.1.103]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B0982D746 for ; Thu, 27 Oct 2022 13:10:18 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1666901416; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=OIqcZRUcgxs1qDADU8Zu6W54GU95YqrDSNAf8mGexD8=; b=MLRzYvvqmtPTt9XY6I6RkLN6jN4QxLSLzM/wvr5P1IMcuEDFWNep8vChlHwqmTWaDNp9/9 jKI/BWTyFEuKGJeaE5DxT63CWij1UrQQEcCguR+dW/3ib0nR9i1tnqeAyKnpVHGxBCNKim m4OkpWJ8z1uJWKYSckFDh3HkhuTAGwc= From: andrey.konovalov@linux.dev To: Marco Elver Cc: Andrey Konovalov , Alexander Potapenko , Dmitry Vyukov , Andrey Ryabinin , kasan-dev@googlegroups.com, Peter Collingbourne , Evgenii Stepanov , Florian Mayer , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH] kasan: allow sampling page_alloc allocations for HW_TAGS Date: Thu, 27 Oct 2022 22:10:09 +0200 Message-Id: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747873408041531254?= X-GMAIL-MSGID: =?utf-8?q?1747873408041531254?= From: Andrey Konovalov Add a new boot parameter called kasan.page_alloc.sample, which makes Hardware Tag-Based KASAN tag only every Nth page_alloc allocation. As Hardware Tag-Based KASAN is intended to be used in production, its performance impact is crucial. As page_alloc allocations tend to be big, tagging and checking all such allocations introduces a significant slowdown in some testing scenarios. The new flag allows to alleviate that slowdown. Enabling page_alloc sampling has a downside: KASAN will miss bad accesses to a page_alloc allocation that has not been tagged. Signed-off-by: Andrey Konovalov --- Documentation/dev-tools/kasan.rst | 4 +++ include/linux/kasan.h | 7 ++--- mm/kasan/common.c | 9 +++++-- mm/kasan/hw_tags.c | 26 +++++++++++++++++++ mm/kasan/kasan.h | 15 +++++++++++ mm/page_alloc.c | 43 +++++++++++++++++++++---------- 6 files changed, 85 insertions(+), 19 deletions(-) diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index 5c93ab915049..bd97301845ef 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -140,6 +140,10 @@ disabling KASAN altogether or controlling its features: - ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc allocations (default: ``on``). +- ``kasan.page_alloc.sample=`` makes KASAN tag only + every Nth page_alloc allocation, where N is the value of the parameter + (default: ``1``). + Error reports ~~~~~~~~~~~~~ diff --git a/include/linux/kasan.h b/include/linux/kasan.h index d811b3d7d2a1..d45d45dfd007 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -120,12 +120,13 @@ static __always_inline void kasan_poison_pages(struct page *page, __kasan_poison_pages(page, order, init); } -void __kasan_unpoison_pages(struct page *page, unsigned int order, bool init); -static __always_inline void kasan_unpoison_pages(struct page *page, +bool __kasan_unpoison_pages(struct page *page, unsigned int order, bool init); +static __always_inline bool kasan_unpoison_pages(struct page *page, unsigned int order, bool init) { if (kasan_enabled()) - __kasan_unpoison_pages(page, order, init); + return __kasan_unpoison_pages(page, order, init); + return false; } void __kasan_cache_create_kmalloc(struct kmem_cache *cache); diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 833bf2cfd2a3..1f30080a7a4c 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -95,19 +95,24 @@ asmlinkage void kasan_unpoison_task_stack_below(const void *watermark) } #endif /* CONFIG_KASAN_STACK */ -void __kasan_unpoison_pages(struct page *page, unsigned int order, bool init) +bool __kasan_unpoison_pages(struct page *page, unsigned int order, bool init) { u8 tag; unsigned long i; if (unlikely(PageHighMem(page))) - return; + return false; + + if (!kasan_sample_page_alloc()) + return false; tag = kasan_random_tag(); kasan_unpoison(set_tag(page_address(page), tag), PAGE_SIZE << order, init); for (i = 0; i < (1 << order); i++) page_kasan_tag_set(page + i, tag); + + return true; } void __kasan_poison_pages(struct page *page, unsigned int order, bool init) diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index b22c4f461cb0..aa3b5a080297 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -59,6 +59,11 @@ EXPORT_SYMBOL_GPL(kasan_mode); /* Whether to enable vmalloc tagging. */ DEFINE_STATIC_KEY_TRUE(kasan_flag_vmalloc); +/* Frequency of page_alloc allocation poisoning. */ +unsigned long kasan_page_alloc_sample = 1; + +DEFINE_PER_CPU(unsigned long, kasan_page_alloc_count); + /* kasan=off/on */ static int __init early_kasan_flag(char *arg) { @@ -122,6 +127,27 @@ static inline const char *kasan_mode_info(void) return "sync"; } +/* kasan.page_alloc.sample= */ +static int __init early_kasan_flag_page_alloc_sample(char *arg) +{ + int rv; + + if (!arg) + return -EINVAL; + + rv = kstrtoul(arg, 0, &kasan_page_alloc_sample); + if (rv) + return rv; + + if (!kasan_page_alloc_sample) { + kasan_page_alloc_sample = 1; + return -EINVAL; + } + + return 0; +} +early_param("kasan.page_alloc.sample", early_kasan_flag_page_alloc_sample); + /* * kasan_init_hw_tags_cpu() is called for each CPU. * Not marked as __init as a CPU can be hot-plugged after boot. diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index abbcc1b0eec5..ee67eb35f4a7 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -42,6 +42,9 @@ enum kasan_mode { extern enum kasan_mode kasan_mode __ro_after_init; +extern unsigned long kasan_page_alloc_sample; +DECLARE_PER_CPU(unsigned long, kasan_page_alloc_count); + static inline bool kasan_vmalloc_enabled(void) { return static_branch_likely(&kasan_flag_vmalloc); @@ -57,6 +60,13 @@ static inline bool kasan_sync_fault_possible(void) return kasan_mode == KASAN_MODE_SYNC || kasan_mode == KASAN_MODE_ASYMM; } +static inline bool kasan_sample_page_alloc(void) +{ + unsigned long *count = this_cpu_ptr(&kasan_page_alloc_count); + + return (*count)++ % kasan_page_alloc_sample == 0; +} + #else /* CONFIG_KASAN_HW_TAGS */ static inline bool kasan_async_fault_possible(void) @@ -69,6 +79,11 @@ static inline bool kasan_sync_fault_possible(void) return true; } +static inline bool kasan_sample_page_alloc(void) +{ + return true; +} + #endif /* CONFIG_KASAN_HW_TAGS */ #ifdef CONFIG_KASAN_GENERIC diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b5a6c815ae28..0b36456aedfb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1366,6 +1366,8 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) * see the comment next to it. * 3. Skipping poisoning is requested via __GFP_SKIP_KASAN_POISON, * see the comment next to it. + * 4. The allocation is excluded from being checked due to sampling, + * see the call to kasan_unpoison_pages. * * Poisoning pages during deferred memory init will greatly lengthen the * process and cause problem in large memory systems as the deferred pages @@ -2475,7 +2477,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order, { bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && !should_skip_init(gfp_flags); - bool init_tags = init && (gfp_flags & __GFP_ZEROTAGS); + bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); + bool reset_tags = !zero_tags; int i; set_page_private(page, 0); @@ -2498,30 +2501,42 @@ inline void post_alloc_hook(struct page *page, unsigned int order, */ /* - * If memory tags should be zeroed (which happens only when memory - * should be initialized as well). + * If memory tags should be zeroed + * (which happens only when memory should be initialized as well). */ - if (init_tags) { + if (zero_tags) { /* Initialize both memory and tags. */ for (i = 0; i != 1 << order; ++i) tag_clear_highpage(page + i); - /* Note that memory is already initialized by the loop above. */ + /* Take note that memory was initialized by the loop above. */ init = false; } if (!should_skip_kasan_unpoison(gfp_flags)) { - /* Unpoison shadow memory or set memory tags. */ - kasan_unpoison_pages(page, order, init); - - /* Note that memory is already initialized by KASAN. */ - if (kasan_has_integrated_init()) - init = false; - } else { - /* Ensure page_address() dereferencing does not fault. */ + /* Try unpoisoning (or setting tags) and initializing memory. */ + if (kasan_unpoison_pages(page, order, init)) { + /* Take note that memory was initialized by KASAN. */ + if (kasan_has_integrated_init()) + init = false; + /* Take note that memory tags were set by KASAN. */ + reset_tags = false; + } else { + /* + * KASAN decided to exclude this allocation from being + * poisoned due to sampling. Skip poisoning as well. + */ + SetPageSkipKASanPoison(page); + } + } + /* + * If memory tags have not been set, reset the page tags to ensure + * page_address() dereferencing does not fault. + */ + if (reset_tags) { for (i = 0; i != 1 << order; ++i) page_kasan_tag_reset(page + i); } - /* If memory is still not initialized, do it now. */ + /* If memory is still not initialized, initialize it now. */ if (init) kernel_init_pages(page, 1 << order); /* Propagate __GFP_SKIP_KASAN_POISON to page flags. */