From patchwork Wed Sep 13 17:14:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 140030 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp761904vqi; Thu, 14 Sep 2023 19:42:18 -0700 (PDT) X-Google-Smtp-Source: AGHT+IExDp+rj9jPs7PI7SZhiUJL1K2Rg8W1NGOnMgQQCOU2aBw4YRLBNZXlpuY564FyvOFWOL82 X-Received: by 2002:a05:6358:919e:b0:142:ff63:2a89 with SMTP id j30-20020a056358919e00b00142ff632a89mr709552rwa.24.1694745737854; Thu, 14 Sep 2023 19:42:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694745737; cv=none; d=google.com; s=arc-20160816; b=xlexGq3K88fmK9LIRmk33VbzMjknryoacncNopm7r6Hsu51oXMd+zMCyn7kXYEVAQ7 iAQrwhgi8CmImD8kXP700PAoi3cvnJJOl/PGRNY38LfgoRsoPC4PhYEJuLuZUk5WPueu 9Y2NjBV4PJuj0p+UFEyx5KkwHcZKKgiN3t86MFFu8OqsurHALjCUWiS66Mk5hmOxrMBi VWwRlLPsYf+f+x9FrnUI3jaa3vl9rHvSfF2sRHwar59BNRnd6AHW2m/0hVwjo17RRcvZ oxfXiMTggIOlt58s7JpLvX/VVBy9Ka432Eg/LqqyfNMDTzNexcmIHNY+hGBV9rseGLhn fxqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dA6zDrMISQ9ixV+61d0CbJSFq6N+6qPUyMdB5Zjx2Sw=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=hAaT8CRAzV47eBs1R7WV1wNrM+Z/xbjcPcFc+tNqtmNIltTjoycW39iKynsLV+pLVf FD+ITJHwSipy+pyfzoAn/tDvmkLP2cR3DZ22dX6g3ltZYoPW/q38yCVWJp2Asd6j0FWK RtNxdcKzd6sDuFQ2CpHPhfgk6e7NWW+D/oHkDTWdJM/a++CIYNatjnU/dyXkkZYSpnr6 7EVsbv4PMY3RjMCmdWIBsAOoZpDx/THEswaYIlaQB5Eb7VNs9Am6GBZJnSx3YAogcOk9 G+sueo/mmw7+vi64tb+SOIyLcTA/XAqTAwfyUQUyG7sLWNjJoUe9BFU83NVqlKqC7GSR 8IEw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=NRb1zL10; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id bk13-20020a056a02028d00b005658a6b71cdsi2456548pgb.1.2023.09.14.19.42.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Sep 2023 19:42:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=NRb1zL10; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 0B2A781410E8; Wed, 13 Sep 2023 10:15:03 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230451AbjIMRPB (ORCPT + 34 others); Wed, 13 Sep 2023 13:15:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229468AbjIMRO4 (ORCPT ); Wed, 13 Sep 2023 13:14:56 -0400 Received: from out-220.mta0.migadu.com (out-220.mta0.migadu.com [91.218.175.220]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37E0598 for ; Wed, 13 Sep 2023 10:14:52 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dA6zDrMISQ9ixV+61d0CbJSFq6N+6qPUyMdB5Zjx2Sw=; b=NRb1zL107/rDRfFWZsp0fzET+7d1A+RLuXbBLhV7sSeKYAUQEbzOobo5W1rWEVCjdjE0Dd pknESjG2AuY5ulwmpFZM53Irkq/aSdHZnmh8PdDGcyQD9/nZGmn8edi4yz2Zs/OVi2gb63 BiRpglut78hvsz84TfRJUgNxqmxUdqU= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 01/19] lib/stackdepot: check disabled flag when fetching Date: Wed, 13 Sep 2023 19:14:26 +0200 Message-Id: <66bf1f0ad22d2c49ef500893340c71355b71d092.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:15:03 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777069707134044917 X-GMAIL-MSGID: 1777069707134044917 From: Andrey Konovalov Do not try fetching a stack trace from the stack depot if the stack_depot_disabled flag is enabled. Reviewed-by: Alexander Potapenko Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 2f5aa851834e..3a945c7206f3 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -477,7 +477,7 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, */ kmsan_unpoison_memory(entries, sizeof(*entries)); - if (!handle) + if (!handle || stack_depot_disabled) return 0; if (parts.pool_index > pool_index_cached) { From patchwork Wed Sep 13 17:14:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139192 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp31513vqi; Wed, 13 Sep 2023 17:55:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE8vLklTVZhoHjC2/BiCmJcI2LlGNA3wZbwLDZesV1fX7Q75UipTlaGX0V1Tj7VayCXlsvz X-Received: by 2002:a17:902:b701:b0:1c4:335:b06d with SMTP id d1-20020a170902b70100b001c40335b06dmr854180pls.32.1694652928647; Wed, 13 Sep 2023 17:55:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694652928; cv=none; d=google.com; s=arc-20160816; b=ZC2hKqwFBwBTzttb8XPRV9HRxr6hnCy+liTEAu+kcqivSie2R27BPEfA3wg2dMPIzN qGnVeEuyuMenyXk8T2wBcC9bZAQiDS011t3Lsez1TVaX5t11FMFMcKVhj7I2I5gL5G25 Dz/wkrei3q6cS1EJkTXmpSfTouGLFnx/ATz29zusIgtwp/9qWasMDanrhM9ZnJfViQ+T mt9aDYDIp1/FVfu8AVXolKdQ6sirBy2ZixyVxmWxpJSDXtSf2RacAFUhEMrPsdQ1A/Ow 9QIYxEgb1pzU+ObbmDK0bc3Qfv0HyD2sduM/8C8bJRiVFI4pvT3st3QeX77WvprN/sHC rHfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hlykGlfL2an/VgWBLR95t+wE6CpOLlFaVT0cHbSVLN0=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=YlflyAk6n32t53xzN47UEDJhwSFMjx/Oy5VUczHXxTDztLDdzv+ftawZXaR6q1tQ+Q 1rfpJ0mAuqTgnpVwHzvI451n9IunT/0PkbOujP9JIpbE7n+B+gBvFC9kFAHtoyd25YtG Q+MJ6g8FRc0Dr7zcSyGKH/h6jypYx9WECGRJZCHz5jdpbvScqBFDgOl1Fjetl4lLGWT/ WuqeeTzEW41g/ip0heEN8mBKDvc2DdQP3pdXOSNnDS/1Mn/KhHBb5ejQrsziU2pfCNFh 8rtsmueTmp+ouYx5IW3Edt7l16vyMKh4PDXwDFacer4IKZL0KNgpj1AX5+1eDlFZGIHg CMhg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=pRcWYiA4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id n2-20020a170902d0c200b001c3ba275321si442679pln.76.2023.09.13.17.55.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 17:55:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=pRcWYiA4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6FC7A80AE81A; Wed, 13 Sep 2023 10:15:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231448AbjIMRPL (ORCPT + 34 others); Wed, 13 Sep 2023 13:15:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230386AbjIMRO7 (ORCPT ); Wed, 13 Sep 2023 13:14:59 -0400 Received: from out-228.mta0.migadu.com (out-228.mta0.migadu.com [91.218.175.228]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A725F19B1 for ; Wed, 13 Sep 2023 10:14:54 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625289; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hlykGlfL2an/VgWBLR95t+wE6CpOLlFaVT0cHbSVLN0=; b=pRcWYiA4V9DcqOTdN2gLjPxezJ2SlGOHPh1Fz59jRDjJpcStfHaMZtV5oUBDvkHZp9pcN1 2sU42kzOkV4/EmNirmCx2z3pluIo9R1jBj4CZupNvF1dF6+hUeuhGaZZv4dXI+nj7Z61iS xPrZS4MeqQS5+CE7Icb238Gz0TxGYDs= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 02/19] lib/stackdepot: simplify __stack_depot_save Date: Wed, 13 Sep 2023 19:14:27 +0200 Message-Id: <3336cf19b8e53ed5449550a085cff9bddec4c873.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:15:25 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776972389382359660 X-GMAIL-MSGID: 1776972389382359660 From: Andrey Konovalov The retval local variable in __stack_depot_save has the union type handle_parts, but the function never uses anything but the union's handle field. Define retval simply as depot_stack_handle_t to simplify the code. Reviewed-by: Alexander Potapenko Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 3a945c7206f3..0772125efe8a 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -360,7 +360,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, gfp_t alloc_flags, bool can_alloc) { struct stack_record *found = NULL, **bucket; - union handle_parts retval = { .handle = 0 }; + depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; unsigned long flags; @@ -377,7 +377,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, nr_entries = filter_irq_stacks(entries, nr_entries); if (unlikely(nr_entries == 0) || stack_depot_disabled) - goto fast_exit; + return 0; hash = hash_stack(entries, nr_entries); bucket = &stack_table[hash & stack_hash_mask]; @@ -443,9 +443,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, free_pages((unsigned long)prealloc, DEPOT_POOL_ORDER); } if (found) - retval.handle = found->handle.handle; -fast_exit: - return retval.handle; + handle = found->handle.handle; + return handle; } EXPORT_SYMBOL_GPL(__stack_depot_save); From patchwork Wed Sep 13 17:14:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139077 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp72712vqb; Wed, 13 Sep 2023 13:36:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGMnJMZRgePcKUeEPDHDBjEEr0sYY5knSSer2p0pD+3IO9eGJaOKsLJlA5KqUYlF7bAH4Cs X-Received: by 2002:a05:6a20:3d82:b0:149:602e:9239 with SMTP id s2-20020a056a203d8200b00149602e9239mr5819084pzi.21.1694637379315; Wed, 13 Sep 2023 13:36:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694637379; cv=none; d=google.com; s=arc-20160816; b=L1i9OWWY5XWAYAlCllskCYXGu5ymbegFgjvj8PqmMMO0Lo4Xd/7svXYILZEp/+q48R m6OkDcWMnfEbB5TUNPG+arBBjdjbWiF3Rn/cCZl2psYgenCU2+gDfAFAkiZ6TLKWDtjq hBu1TiWhobd2ivMkiKX6nqtau8AeLifhJkb34x8No99rRRjglFYfgAB1GXMWmwiuZ/gf YHZQiv4RWouRlp2hrxO12I5BTjE6hcc5V7d/U3E1B6pLSDNbZ/dMnSBpdAXJ/FkSbFv7 rEjxsNyn/HPqkkgEA+enRyONRCxvqhr+IWWGeje1PLo+knGtwBK82NXatxkDqGCjUuiV CbeA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eQriSK20/8KwwHOVdQKhz9C/zLOim8sff0QLVit0F7g=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=QwYAHv9bWdA8TqfSKaNPFxFWS4k3JRrETdvQscdsw+UcjqWOfWi/2Xok1CdMkSQESU IC/pliISK8g38CtqqEpUxhES7U/ezfn7E6hJsejKflPumuZMDweayb+7Bny7oE9sIpYS cQCTKHCFRuNVLM0m7AmXJ0BIE11SCBprPleWQVNHn8XhKk8eGVvrwtjcfS51XNLYM+Q+ zaCnWlnrMSeMsjYxHUyFdlVlPVMT24qaUeMbkCdrBlJvisC6jz/bh1aln/byLf1Ro74u n639CBOOJXM9xa3/t5mHkKHstYY6SPs8CRmqyqP2ofj9MvyeZOMnyde9p7XtcLqviz7T 078w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=XFAMKTE8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id d3-20020a63f243000000b0056952b496efsi10689280pgk.366.2023.09.13.13.35.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 13:36:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=XFAMKTE8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id B6D2D8113411; Wed, 13 Sep 2023 10:15:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230437AbjIMRPE (ORCPT + 34 others); Wed, 13 Sep 2023 13:15:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230347AbjIMRO7 (ORCPT ); Wed, 13 Sep 2023 13:14:59 -0400 Received: from out-218.mta0.migadu.com (out-218.mta0.migadu.com [91.218.175.218]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AFE3DD for ; Wed, 13 Sep 2023 10:14:53 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eQriSK20/8KwwHOVdQKhz9C/zLOim8sff0QLVit0F7g=; b=XFAMKTE8wAgngJGqU3Ho9bfesOIML66PlRFcmLlbqUwEjvPJBgA62jyh52QZkOwLjDPuzU AyQx3wfOttMTFmpwW6FPSwPLuzweuqHN5H2L0Pdvtlpy0XaMKleBiTQWXdXoBao5TkmHyv tHA54Ka/Rk4e/x6SqQ17ainnSJdFZLQ= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 03/19] lib/stackdepot: drop valid bit from handles Date: Wed, 13 Sep 2023 19:14:28 +0200 Message-Id: <5cbb8235fe6418f970fd0012450defdce598abcf.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:15:25 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776956084450984375 X-GMAIL-MSGID: 1776956084450984375 From: Andrey Konovalov Stack depot doesn't use the valid bit in handles in any way, so drop it. Reviewed-by: Alexander Potapenko Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 0772125efe8a..482eac40791e 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -32,13 +32,12 @@ #define DEPOT_HANDLE_BITS (sizeof(depot_stack_handle_t) * 8) -#define DEPOT_VALID_BITS 1 #define DEPOT_POOL_ORDER 2 /* Pool size order, 4 pages */ #define DEPOT_POOL_SIZE (1LL << (PAGE_SHIFT + DEPOT_POOL_ORDER)) #define DEPOT_STACK_ALIGN 4 #define DEPOT_OFFSET_BITS (DEPOT_POOL_ORDER + PAGE_SHIFT - DEPOT_STACK_ALIGN) -#define DEPOT_POOL_INDEX_BITS (DEPOT_HANDLE_BITS - DEPOT_VALID_BITS - \ - DEPOT_OFFSET_BITS - STACK_DEPOT_EXTRA_BITS) +#define DEPOT_POOL_INDEX_BITS (DEPOT_HANDLE_BITS - DEPOT_OFFSET_BITS - \ + STACK_DEPOT_EXTRA_BITS) #define DEPOT_POOLS_CAP 8192 #define DEPOT_MAX_POOLS \ (((1LL << (DEPOT_POOL_INDEX_BITS)) < DEPOT_POOLS_CAP) ? \ @@ -50,7 +49,6 @@ union handle_parts { struct { u32 pool_index : DEPOT_POOL_INDEX_BITS; u32 offset : DEPOT_OFFSET_BITS; - u32 valid : DEPOT_VALID_BITS; u32 extra : STACK_DEPOT_EXTRA_BITS; }; }; @@ -303,7 +301,6 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) stack->size = size; stack->handle.pool_index = pool_index; stack->handle.offset = pool_offset >> DEPOT_STACK_ALIGN; - stack->handle.valid = 1; stack->handle.extra = 0; memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); pool_offset += required_size; From patchwork Wed Sep 13 17:14:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139155 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp146060vqb; Wed, 13 Sep 2023 16:18:14 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEkfwaCeMvU32XV2DYcjt7k4h4aXn8QUVbIBrpR8C+wCEqexW5y5QEcl2S5fD616pAEn7GO X-Received: by 2002:a0d:dd10:0:b0:577:51cd:1b4a with SMTP id g16-20020a0ddd10000000b0057751cd1b4amr4273340ywe.41.1694647094050; Wed, 13 Sep 2023 16:18:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694647094; cv=none; d=google.com; s=arc-20160816; b=AtYD3GIyvFv+C4nG92kVky/v4vKnSnc/t9i9e6reFx5BtR1OuVzYQbs8euvzergATY Ts8GaAP12nvelMONqszBqXZSW088IOtS6r9vkirKLYrLvCBcfO0+2mEnL9kEQMyCfaza d33wh8Aa1D98D5yD+3OEHC5/EqbEAkFjK1qF0wMNUJjKj+h8PnAH/cD2246iDhaa0Pef voZCsMRro2HqzH3BzFymrM7hqBIvrYYyplctjJnKVsNqlSRlYiFA2l8pjsS8notOUUGA T9iqAZWnsPyy2thJPVFeelmJKm5LOGNjFetHfhvua08iTxzn32ABe0RNUvJLNSdQKttd VE7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9Tw+dU2o5sfOuuXbX4FJVP/5BAKc6p0/nQ7cBmUzv5Q=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=vzaEtrurA8wCF5OWnZYpyydoKMwiRCf4Oe1IvdW9gSjG1gqFf2iT4BV/1xoKTed7cN DvTSWolKIwg36UenqYKE7+JEYgHtQ0GYbvOIGcg+YdB19xqCN3582tqU7rOXaKflP0dB 9F13SyVB9A6EfovcOp//j4QbpTyoUOgS0Oy8KILOUZ37VUJFc53zulORMlW+ACRbgp7L SDRSne/mMr3Q4qSMYwGiuJ/jL1p8EB7+fsupTFiCPgQp8hdmXfZkblIG3woDbDVES4ZH Gm3T0cmtW87J6t2CxIR1zTvDmU6uiNrADCrcyzvYCWKIGY1Pp+0VPukQHTaxCYLOx9/5 R0UQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ctPU1P8P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id z2-20020a656642000000b00573ff14b9e0si154840pgv.763.2023.09.13.16.18.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 16:18:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ctPU1P8P; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id C8CAA806D7F8; Wed, 13 Sep 2023 10:15:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230486AbjIMRPH (ORCPT + 34 others); Wed, 13 Sep 2023 13:15:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230057AbjIMRO4 (ORCPT ); Wed, 13 Sep 2023 13:14:56 -0400 Received: from out-220.mta0.migadu.com (out-220.mta0.migadu.com [91.218.175.220]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 568399B for ; Wed, 13 Sep 2023 10:14:52 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9Tw+dU2o5sfOuuXbX4FJVP/5BAKc6p0/nQ7cBmUzv5Q=; b=ctPU1P8PjzANEZxxufrmMlhavpf6MSmIQazNfrOE3GkL3F/j/nvFZZSEB8ovVp30c7s6MY CBKIkfoanapClJxqvm57Tjw1Y2V8mO/SFzga+leuMilEeLfLnIn8oak/3JOrpDVDpm0SEr N49MYNSPajkSI5g7JXyddu5oLOfkpEI= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 04/19] lib/stackdepot: add depot_fetch_stack helper Date: Wed, 13 Sep 2023 19:14:29 +0200 Message-Id: <74e0a28a38b05f27f3a4b54d5fa93213672fcd30.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:15:29 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776966271020610713 X-GMAIL-MSGID: 1776966271020610713 From: Andrey Konovalov Add a helper depot_fetch_stack function that fetches the pointer to a stack record. With this change, all static depot_* functions now operate on stack pools and the exported stack_depot_* functions operate on the hash table. Reviewed-by: Alexander Potapenko Signed-off-by: Andrey Konovalov --- Changes v1->v2: - Minor comment fix as suggested by Alexander. --- lib/stackdepot.c | 45 ++++++++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 482eac40791e..9a004f15f59d 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -304,6 +304,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) stack->handle.extra = 0; memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); pool_offset += required_size; + /* * Let KMSAN know the stored stack record is initialized. This shall * prevent false positive reports if instrumented code accesses it. @@ -313,6 +314,32 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) return stack; } +static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) +{ + union handle_parts parts = { .handle = handle }; + /* + * READ_ONCE pairs with potential concurrent write in + * depot_alloc_stack(). + */ + int pool_index_cached = READ_ONCE(pool_index); + void *pool; + size_t offset = parts.offset << DEPOT_STACK_ALIGN; + struct stack_record *stack; + + if (parts.pool_index > pool_index_cached) { + WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", + parts.pool_index, pool_index_cached, handle); + return NULL; + } + + pool = stack_pools[parts.pool_index]; + if (!pool) + return NULL; + + stack = pool + offset; + return stack; +} + /* Calculates the hash for a stack. */ static inline u32 hash_stack(unsigned long *entries, unsigned int size) { @@ -456,14 +483,6 @@ EXPORT_SYMBOL_GPL(stack_depot_save); unsigned int stack_depot_fetch(depot_stack_handle_t handle, unsigned long **entries) { - union handle_parts parts = { .handle = handle }; - /* - * READ_ONCE pairs with potential concurrent write in - * depot_alloc_stack. - */ - int pool_index_cached = READ_ONCE(pool_index); - void *pool; - size_t offset = parts.offset << DEPOT_STACK_ALIGN; struct stack_record *stack; *entries = NULL; @@ -476,15 +495,7 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, if (!handle || stack_depot_disabled) return 0; - if (parts.pool_index > pool_index_cached) { - WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", - parts.pool_index, pool_index_cached, handle); - return 0; - } - pool = stack_pools[parts.pool_index]; - if (!pool) - return 0; - stack = pool + offset; + stack = depot_fetch_stack(handle); *entries = stack->entries; return stack->size; From patchwork Wed Sep 13 17:14:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139297 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp92752vqi; Wed, 13 Sep 2023 20:56:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEg10rhrDEgnLjCzOoraBfUCT+oaIjCbzRl9eLAIJ1la+NaV0k5X2W6RAkOU9BZDF+sX2p2 X-Received: by 2002:a67:fe13:0:b0:44d:4aa1:9d3f with SMTP id l19-20020a67fe13000000b0044d4aa19d3fmr4520259vsr.4.1694663803099; Wed, 13 Sep 2023 20:56:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694663803; cv=none; d=google.com; s=arc-20160816; b=fGqcOivnMMqYFW19pM5WeQ0OL39VWE1twlRRQvU4sb0Q/bldZcYzeQ32/rNwsrrOfj msp1jBSUW8iHy6n/vPTjcxVDzdMfb/xpV77KfOeRoAY0vRzg2SPui3Mh+T4rJGza8J90 4vTM+0VD8UJBb/YnNIJVl8+mbhgoV3rnQuaro/41duAkuP8mskuRSju6kiWCFJm03D6w /tbxU2B0v7nOYxwW+B05YjZhKVoBR3gAEyb9ROGIVBhKXFaXvDD2roxlsLVzwyrvGjDv Lg0vjEbyGKSG/GORfgIKlU/EnHSFPgyi4tRCwBl/wYIYgSaEKXawIayt83PvdDB5RfhN CC5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gml8DcgHPrqnYJfUQnrXQLUWDOAsSsS+HJ/vnfgqmKg=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=XnzTm3Eq60zMQDr/rd0wIJ+kmLBBmZLVdaGfzvR05j6MoDytLzue93CUOsIO+Qy93B 7ElTSV+YbaUe0rGwmyo2yDG+FxFGGCkbA7YQvYLctTJyVCqNFFMUu3Cl2qprq1OBPiLs hAYbEvkphQjhULN/uFnkVLk8Syr87VcAIJHKq6pdTn/gVi4cmII1yPl4rSjgWDkEUBeB bNteOrJZJgcJwGwp5nmGpa/QH7JGPRaHDAcaeE+PVtzP4dQMqm4WCRFo6JXnY73zs5r7 vDYio7V6nCNVz2E+jqGH9ajJQrlZhFQgmHFTPGAxwWWJrjmOl+8WCBAdnYEK/ojc/jXl 3pOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=S9us7jlk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id s18-20020a639252000000b005776a454f49si622790pgn.460.2023.09.13.20.56.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 20:56:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=S9us7jlk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 676068225CB5; Wed, 13 Sep 2023 10:15:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230410AbjIMRPJ (ORCPT + 34 others); Wed, 13 Sep 2023 13:15:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230327AbjIMRO5 (ORCPT ); Wed, 13 Sep 2023 13:14:57 -0400 Received: from out-228.mta0.migadu.com (out-228.mta0.migadu.com [91.218.175.228]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05173DC for ; Wed, 13 Sep 2023 10:14:52 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625291; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gml8DcgHPrqnYJfUQnrXQLUWDOAsSsS+HJ/vnfgqmKg=; b=S9us7jlkKOCVp2Io5I0Tx7rlmHRdUo7dp4fSfiusTxeMqFjArjUdWXOxPxa7TRsS+UTSGt szqyWIxKS4Bt5+b4uJDHImZmEPS4RWHjhTLuJQarfEMznDy8FYztx69TSQNHWeON9BhC+T q2mjKPkhq8D4BrojemcgDiouTfk4yj8= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 05/19] lib/stackdepot: use fixed-sized slots for stack records Date: Wed, 13 Sep 2023 19:14:30 +0200 Message-Id: <658f5f34d4f94721844ad8ba41452d54b4f8ace5.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:15:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776983792239275585 X-GMAIL-MSGID: 1776983792239275585 From: Andrey Konovalov Instead of storing stack records in stack depot pools one right after another, use fixed-sized slots. Add a new Kconfig option STACKDEPOT_MAX_FRAMES that allows to select the size of the slot in frames. Use 64 as the default value, which is the maximum stack trace size both KASAN and KMSAN use right now. Also add descriptions for other stack depot Kconfig options. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- Changes v1->v2: - Add and use STACKDEPOT_MAX_FRAMES Kconfig option. --- lib/Kconfig | 10 ++++++++-- lib/stackdepot.c | 13 +++++++++---- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/lib/Kconfig b/lib/Kconfig index c686f4adc124..7c32f424a6f3 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -708,13 +708,19 @@ config ARCH_STACKWALK bool config STACKDEPOT - bool + bool "Stack depot: stack trace storage that avoids duplication" select STACKTRACE config STACKDEPOT_ALWAYS_INIT - bool + bool "Always initialize stack depot during early boot" select STACKDEPOT +config STACKDEPOT_MAX_FRAMES + int "Maximum number of frames in trace saved in stack depot" + range 1 256 + default 64 + depends on STACKDEPOT + config REF_TRACKER bool depends on STACKTRACE_SUPPORT diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 9a004f15f59d..128ece21afe9 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -58,9 +58,12 @@ struct stack_record { u32 hash; /* Hash in the hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; - unsigned long entries[]; /* Variable-sized array of frames */ + unsigned long entries[CONFIG_STACKDEPOT_MAX_FRAMES]; /* Frames */ }; +#define DEPOT_STACK_RECORD_SIZE \ + ALIGN(sizeof(struct stack_record), 1 << DEPOT_STACK_ALIGN) + static bool stack_depot_disabled; static bool __stack_depot_early_init_requested __initdata = IS_ENABLED(CONFIG_STACKDEPOT_ALWAYS_INIT); static bool __stack_depot_early_init_passed __initdata; @@ -258,9 +261,7 @@ static struct stack_record * depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) { struct stack_record *stack; - size_t required_size = struct_size(stack, entries, size); - - required_size = ALIGN(required_size, 1 << DEPOT_STACK_ALIGN); + size_t required_size = DEPOT_STACK_RECORD_SIZE; /* Check if there is not enough space in the current pool. */ if (unlikely(pool_offset + required_size > DEPOT_POOL_SIZE)) { @@ -295,6 +296,10 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) if (stack_pools[pool_index] == NULL) return NULL; + /* Limit number of saved frames to CONFIG_STACKDEPOT_MAX_FRAMES. */ + if (size > CONFIG_STACKDEPOT_MAX_FRAMES) + size = CONFIG_STACKDEPOT_MAX_FRAMES; + /* Save the stack trace. */ stack = stack_pools[pool_index] + pool_offset; stack->hash = hash; From patchwork Wed Sep 13 17:14:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 138982 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9ecd:0:b0:3f2:4152:657d with SMTP id t13csp1251520vqx; Wed, 13 Sep 2023 10:41:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHxpWsB+2b+r/5QdrzYmPizRYGBUtx3RY3VasJZk22d0mPj8N19Ofr2CNrav2sSklXoGGjn X-Received: by 2002:a05:6a21:78a8:b0:14c:d494:77d1 with SMTP id bf40-20020a056a2178a800b0014cd49477d1mr3740462pzc.10.1694626900912; Wed, 13 Sep 2023 10:41:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694626900; cv=none; d=google.com; s=arc-20160816; b=0vIwV5OT5zQlQGJyf4wTqAjCM0q/DR7eImXlyRky604lrcEJ0RSmdt/xaXu75AS3Tz hlrajdMC4VbHzp6vhhxbMFjNPMT6UlYO7rJX7owc8nsSc0fzqTIyDE4gp7s7Nj1gYZYj jR7MUKrwUeLBzWQJ+i4tbhjSRPXDl98tWsULrN3nRzBE+6xHoANt5aEGC2h7fZcMUQQU U75/BWtgapFiFDkqBzK6e20igAolQyEYuMzLUkCsm8L37ePsZw3W/zZUSIF88gK2EA/4 ptOXlK5BvWtnpxUwHpdYrdwMCSIMpD7esNTtjAtF3hmmbB11m76gGyQp+uS4J96BaeGX kaFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0rCvfJyQVclVcl+nE6KRDaf9jMIyS6A7Wyxgy4VzHmE=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=LaudRh48GMQT8Qk3y+FG5ca94AgdVy3yAq/nQHJ3rlXEBeKhdBbqkjuEvOHkSeElQ3 69iHMI+ACYq8vuPSRf9UpN3kAxzqP7nC8Ex3RG8tQtpgwdz0/3caKvzK8UuAGsCv+qD6 X+88gHCC6/VNDkq7eCCyAkPBden+p1WMujhdzXqexkMpAHpq1e9kkFYVfdR5cHhv0pZ5 85y56mApwvcAf0WJ+JbR01nn7i8IPdtDRIzi6LMwd7QhyXzCC+6hKblRzpvsvWN6Btvg mhef27Idlw4gbKsg9kzhe3toCpl41fNnn37GAAPRJmmh9uRb54umai1AtMH1f/2SSbQh njNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=TYMgdFjb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id b3-20020a6567c3000000b005696a8e371csi10071256pgs.35.2023.09.13.10.41.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 10:41:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=TYMgdFjb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id ED92381CB0D1; Wed, 13 Sep 2023 10:16:11 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230057AbjIMRQB (ORCPT + 34 others); Wed, 13 Sep 2023 13:16:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229654AbjIMRQA (ORCPT ); Wed, 13 Sep 2023 13:16:00 -0400 Received: from out-222.mta1.migadu.com (out-222.mta1.migadu.com [IPv6:2001:41d0:203:375::de]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E21B98 for ; Wed, 13 Sep 2023 10:15:56 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625354; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0rCvfJyQVclVcl+nE6KRDaf9jMIyS6A7Wyxgy4VzHmE=; b=TYMgdFjbt/Eo9/c1f+k7VBMopiQ5ELQvuAFxZx2mG4w5fYla84GACRt9mHYjc8VPC6hY1n PzTFY9MQ4IbYJVEAwdIh5X1A1wBrieX7KoSLaA+ZMLXosAUWrOS734vnP0yfp7RNTCW3/Q RaHcPG2lCnUbh38nQx0ZOsudkh1SFKg= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 06/19] lib/stackdepot: fix and clean-up atomic annotations Date: Wed, 13 Sep 2023 19:14:31 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:16:12 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776945096802310581 X-GMAIL-MSGID: 1776945096802310581 From: Andrey Konovalov Simplify comments accompanying the use of atomic accesses in the stack depot code. Also drop smp_load_acquire from next_pool_required in depot_init_pool, as both depot_init_pool and the all smp_store_release's to this variable are executed under the stack depot lock. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- This patch is not strictly required, as the atomic accesses are fully removed in one of the latter patches. However, I decided to keep the patch just in case we end up needing these atomics in the following iterations of this series. Changes v1->v2: - Minor comment fix as suggested by Marco. - Drop READ_ONCE marking for next_pool_required. --- lib/stackdepot.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 128ece21afe9..babd453261f0 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -225,10 +225,8 @@ static void depot_init_pool(void **prealloc) /* * If the next pool is already initialized or the maximum number of * pools is reached, do not use the preallocated memory. - * smp_load_acquire() here pairs with smp_store_release() below and - * in depot_alloc_stack(). */ - if (!smp_load_acquire(&next_pool_required)) + if (!next_pool_required) return; /* Check if the current pool is not yet allocated. */ @@ -249,8 +247,8 @@ static void depot_init_pool(void **prealloc) * At this point, either the next pool is initialized or the * maximum number of pools is reached. In either case, take * note that initializing another pool is not required. - * This smp_store_release pairs with smp_load_acquire() above - * and in stack_depot_save(). + * smp_store_release pairs with smp_load_acquire in + * stack_depot_save. */ smp_store_release(&next_pool_required, 0); } @@ -274,15 +272,15 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) /* * Move on to the next pool. * WRITE_ONCE pairs with potential concurrent read in - * stack_depot_fetch(). + * stack_depot_fetch. */ WRITE_ONCE(pool_index, pool_index + 1); pool_offset = 0; /* * If the maximum number of pools is not reached, take note * that the next pool needs to initialized. - * smp_store_release() here pairs with smp_load_acquire() in - * stack_depot_save() and depot_init_pool(). + * smp_store_release pairs with smp_load_acquire in + * stack_depot_save. */ if (pool_index + 1 < DEPOT_MAX_POOLS) smp_store_release(&next_pool_required, 1); @@ -324,7 +322,7 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) union handle_parts parts = { .handle = handle }; /* * READ_ONCE pairs with potential concurrent write in - * depot_alloc_stack(). + * depot_alloc_stack. */ int pool_index_cached = READ_ONCE(pool_index); void *pool; @@ -413,8 +411,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, /* * Fast path: look the stack trace up without locking. - * The smp_load_acquire() here pairs with smp_store_release() to - * |bucket| below. + * smp_load_acquire pairs with smp_store_release to |bucket| below. */ found = find_stack(smp_load_acquire(bucket), entries, nr_entries, hash); if (found) @@ -424,8 +421,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * Check if another stack pool needs to be initialized. If so, allocate * the memory now - we won't be able to do that under the lock. * - * The smp_load_acquire() here pairs with smp_store_release() to - * |next_pool_inited| in depot_alloc_stack() and depot_init_pool(). + * smp_load_acquire pairs with smp_store_release in depot_alloc_stack + * and depot_init_pool. */ if (unlikely(can_alloc && smp_load_acquire(&next_pool_required))) { /* @@ -451,8 +448,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, if (new) { new->next = *bucket; /* - * This smp_store_release() pairs with - * smp_load_acquire() from |bucket| above. + * smp_store_release pairs with smp_load_acquire + * from |bucket| above. */ smp_store_release(bucket, new); found = new; From patchwork Wed Sep 13 17:14:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139060 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp46734vqb; Wed, 13 Sep 2023 12:44:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGkcNbuGsWAztbYecq+VOEGRGixIjg5FBsxPsvg1jTQDQcsUZ4wFAmT0jvPRdYAhtUHBA/7 X-Received: by 2002:a17:90b:110a:b0:268:2658:3b01 with SMTP id gi10-20020a17090b110a00b0026826583b01mr3165683pjb.39.1694634293192; Wed, 13 Sep 2023 12:44:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694634293; cv=none; d=google.com; s=arc-20160816; b=EFuebpsixPt4gjhSIWKn0/Q3qx/ySIUwNKR2T30Fxj0VbH5cZ2YGCs5J5icyt1Mh/p M3x+GFt8noE6tpxibEyBaqUPRvzWA3TSPDmQM/eiRw6pYIkaUb2BCgsyaFBaCOq2ojrg w4ZmEqPEvOJgYiKKR8PJh93i9PV6i3Q9amuBxJTyr4GckPqTkHBRHwzchFxdSGsfSTMN +vh0IfAE8A1DbTqElvx3wO3EE5iKi+1qf8krE6y1aQozdYHV/NWXoi2TLU4M2JzRWbbK Iv6ew+Fh1cwSE9tWrwEyKOP8ZeUXTSMp/7F+R9Yboi1JLyW+zmoksobfidrsTqjddgD3 /rVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gAOd+0bdipGa5TwT77P0cix2trL96KYDpIy1afA6aX4=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=gtb+4fB3v+JA5nBpJXqLrQWJomVB505Oa1k+faD4Xb3F8PweQH/S2aCH97pwX01+au OYl5/eHG8tMCxtKyhwWw4QG1kNm44sM5JN6AgYRcXu8FEWYKEn3cjBnXHDEW0FWcYV3r yO9U+TXvlt1cOGgE84//4JBCrDknII75h8FPCFENy5ucPD6ugSrFPbpAPtbvKUp9+BvD 5BgEDIAKlalfC4u5Og2UldYF960vmnWXeYS5M820vdXJxOX9hYLUZQsrfXa/U30EXh8h gP2urkvo7Oau3Ax3B5HApNaQqq34TYduPztmT4ZXbnsLu5ZmT/vg2vg9EwlciFvbOLnv CKWg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=puVK18ov; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id nv2-20020a17090b1b4200b0025359a570fesi14081pjb.19.2023.09.13.12.44.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 12:44:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=puVK18ov; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id BE85982516EF; Wed, 13 Sep 2023 10:16:11 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229654AbjIMRQC (ORCPT + 34 others); Wed, 13 Sep 2023 13:16:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229468AbjIMRQA (ORCPT ); Wed, 13 Sep 2023 13:16:00 -0400 Received: from out-222.mta1.migadu.com (out-222.mta1.migadu.com [IPv6:2001:41d0:203:375::de]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FA499B for ; Wed, 13 Sep 2023 10:15:56 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gAOd+0bdipGa5TwT77P0cix2trL96KYDpIy1afA6aX4=; b=puVK18ovwq5iY1p+xtahLNnbqGMtfLRPDwUGhyhUQSv+gXInLzJ556ocSpipYuAJUi0IMk O/5IT0uIadYTAipofo7r3zp1kwfW+8ahs4/NkRwA7zFRFCmipzk5gZ6+1AqN6CDXnhRBxH zjagfztze/wMDtvIwTVJol/qj+Z7myE= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 07/19] lib/stackdepot: rework helpers for depot_alloc_stack Date: Wed, 13 Sep 2023 19:14:32 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:16:11 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776952848217372378 X-GMAIL-MSGID: 1776952848217372378 From: Andrey Konovalov Split code in depot_alloc_stack and depot_init_pool into 3 functions: 1. depot_keep_next_pool that keeps preallocated memory for the next pool if required. 2. depot_update_pools that moves on to the next pool if there's no space left in the current pool, uses preallocated memory for the new current pool if required, and calls depot_keep_next_pool otherwise. 3. depot_alloc_stack that calls depot_update_pools and then allocates a stack record as before. This makes it somewhat easier to follow the logic of depot_alloc_stack and also serves as a preparation for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 87 +++++++++++++++++++++++++++--------------------- 1 file changed, 49 insertions(+), 38 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index babd453261f0..e85b658be050 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -219,54 +219,43 @@ int stack_depot_init(void) } EXPORT_SYMBOL_GPL(stack_depot_init); -/* Uses preallocated memory to initialize a new stack depot pool. */ -static void depot_init_pool(void **prealloc) +/* Keeps the preallocated memory to be used for the next stack depot pool. */ +static void depot_keep_next_pool(void **prealloc) { /* - * If the next pool is already initialized or the maximum number of + * If the next pool is already saved or the maximum number of * pools is reached, do not use the preallocated memory. */ if (!next_pool_required) return; - /* Check if the current pool is not yet allocated. */ - if (stack_pools[pool_index] == NULL) { - /* Use the preallocated memory for the current pool. */ - stack_pools[pool_index] = *prealloc; + /* + * Use the preallocated memory for the next pool + * as long as we do not exceed the maximum number of pools. + */ + if (pool_index + 1 < DEPOT_MAX_POOLS) { + stack_pools[pool_index + 1] = *prealloc; *prealloc = NULL; - } else { - /* - * Otherwise, use the preallocated memory for the next pool - * as long as we do not exceed the maximum number of pools. - */ - if (pool_index + 1 < DEPOT_MAX_POOLS) { - stack_pools[pool_index + 1] = *prealloc; - *prealloc = NULL; - } - /* - * At this point, either the next pool is initialized or the - * maximum number of pools is reached. In either case, take - * note that initializing another pool is not required. - * smp_store_release pairs with smp_load_acquire in - * stack_depot_save. - */ - smp_store_release(&next_pool_required, 0); } + + /* + * At this point, either the next pool is kept or the maximum + * number of pools is reached. In either case, take note that + * keeping another pool is not required. + * smp_store_release pairs with smp_load_acquire in stack_depot_save. + */ + smp_store_release(&next_pool_required, 0); } -/* Allocates a new stack in a stack depot pool. */ -static struct stack_record * -depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) +/* Updates refences to the current and the next stack depot pools. */ +static bool depot_update_pools(size_t required_size, void **prealloc) { - struct stack_record *stack; - size_t required_size = DEPOT_STACK_RECORD_SIZE; - /* Check if there is not enough space in the current pool. */ if (unlikely(pool_offset + required_size > DEPOT_POOL_SIZE)) { /* Bail out if we reached the pool limit. */ if (unlikely(pool_index + 1 >= DEPOT_MAX_POOLS)) { WARN_ONCE(1, "Stack depot reached limit capacity"); - return NULL; + return false; } /* @@ -276,9 +265,10 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) */ WRITE_ONCE(pool_index, pool_index + 1); pool_offset = 0; + /* * If the maximum number of pools is not reached, take note - * that the next pool needs to initialized. + * that the next pool needs to be initialized. * smp_store_release pairs with smp_load_acquire in * stack_depot_save. */ @@ -286,9 +276,30 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) smp_store_release(&next_pool_required, 1); } - /* Assign the preallocated memory to a pool if required. */ + /* Check if the current pool is not yet allocated. */ + if (*prealloc && stack_pools[pool_index] == NULL) { + /* Use the preallocated memory for the current pool. */ + stack_pools[pool_index] = *prealloc; + *prealloc = NULL; + return true; + } + + /* Otherwise, try using the preallocated memory for the next pool. */ if (*prealloc) - depot_init_pool(prealloc); + depot_keep_next_pool(prealloc); + return true; +} + +/* Allocates a new stack in a stack depot pool. */ +static struct stack_record * +depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) +{ + struct stack_record *stack; + size_t required_size = DEPOT_STACK_RECORD_SIZE; + + /* Update current and next pools if required and possible. */ + if (!depot_update_pools(required_size, prealloc)) + return NULL; /* Check if we have a pool to save the stack trace. */ if (stack_pools[pool_index] == NULL) @@ -322,7 +333,7 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) union handle_parts parts = { .handle = handle }; /* * READ_ONCE pairs with potential concurrent write in - * depot_alloc_stack. + * depot_update_pools. */ int pool_index_cached = READ_ONCE(pool_index); void *pool; @@ -421,8 +432,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * Check if another stack pool needs to be initialized. If so, allocate * the memory now - we won't be able to do that under the lock. * - * smp_load_acquire pairs with smp_store_release in depot_alloc_stack - * and depot_init_pool. + * smp_load_acquire pairs with smp_store_release in depot_update_pools + * and depot_keep_next_pool. */ if (unlikely(can_alloc && smp_load_acquire(&next_pool_required))) { /* @@ -459,7 +470,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * Stack depot already contains this stack trace, but let's * keep the preallocated memory for the future. */ - depot_init_pool(&prealloc); + depot_keep_next_pool(&prealloc); } raw_spin_unlock_irqrestore(&pool_lock, flags); From patchwork Wed Sep 13 17:14:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139932 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp707976vqi; Thu, 14 Sep 2023 17:15:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGFX/FRalpldb6Siv7BaFW6BqQrM3RrufQeYp3xX61rcI1EtTOapbEEbU8/m71ISd2nVTu4 X-Received: by 2002:a05:6358:e48c:b0:135:ae78:56c9 with SMTP id by12-20020a056358e48c00b00135ae7856c9mr316168rwb.6.1694736956115; Thu, 14 Sep 2023 17:15:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694736956; cv=none; d=google.com; s=arc-20160816; b=MPidx1ai6R9YXt6lqfO3lOhKwsbbfe6ex2Cq1ZsJfWEzxpWLq8wCe/FQx45EERQTzR OG3HFIbyLfypaoIlzITB905Yl/maQeoRQ3oeD3ZjX5Uu7ZRvy7hGkNgeVYJv8Wk+Wfpt J865rZVKet5IKuC95oM/F5tZPBfUwPIxvA+laCAjrPwSWxcJ81gE7D68LECRZPcf3LZG QAWcGb0Hzw7cNp2Bw80WxvfyOIGUiJ4+OgFwKyzF2RKKoBhjHFq3fiV1b/YNeZGjyZ6m GKMg5Vz9f1PFuLzHXisYj75ZbVfmYhu/5Hdko5Gp2HKJuW0ciIqNCDTbPtGPAtMrMXi8 x3gQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0/KQC+g9eK7MZLi0iRkQvuX1rPqJBRg75npInF3Ouo8=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=rY3/gQ63t8Y42iWxiB0uMGzNonMNI4VwEHeTnwjMmUKhSwhw0Ndt3Aqw63l5RBU7Wb kkjuqsCBgPLu1cbIEfmIHWPP9WZAKTS0sLFJreaXFcFYLK+ZvWFd1tBpz+Cgj/VFy/j/ +UPfHWotwnVsSJdhRmN7VdDYAqrg6tWP3QiO6CRhI6x5iS9/Sgxlb/4i3nGLa6MNDYm6 BZ/6W89R9I+17z8zbe4vNk3ALMtFrYl0QkcB7pXJNF/CTK6VbbiloVBCcjCqBmIL2pSS cKsA4eGZk/AC/ggXfskEjmncScexEateQ3X57EFMrPCZbIxfKMtGqY/PT8eQv4tTeKSZ r8uQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=n8pGZUmY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id bx29-20020a056a02051d00b0056cab36f3e1si2658627pgb.221.2023.09.14.17.15.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Sep 2023 17:15:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=n8pGZUmY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7FA6A81A1E98; Wed, 13 Sep 2023 10:16:33 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230451AbjIMRQF (ORCPT + 34 others); Wed, 13 Sep 2023 13:16:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230300AbjIMRQB (ORCPT ); Wed, 13 Sep 2023 13:16:01 -0400 Received: from out-229.mta1.migadu.com (out-229.mta1.migadu.com [95.215.58.229]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B25298 for ; Wed, 13 Sep 2023 10:15:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0/KQC+g9eK7MZLi0iRkQvuX1rPqJBRg75npInF3Ouo8=; b=n8pGZUmYCVqQ7/ite2vF9JMhwV+glcABYHAJ/a/1kWV1LzwnX+pz3GngJd9Jqqc1/Pv61I Tc5+8k/YmnNs3tTelE4rd84gD248pTV9VF628E9nZrnCTuq8FjxCXOUUKGof/AhL0LIypC 1oXVFUG4VWiBhHDjNZdJF0SfAvv8pl4= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 08/19] lib/stackdepot: rename next_pool_required to new_pool_required Date: Wed, 13 Sep 2023 19:14:33 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:16:33 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777060498548021547 X-GMAIL-MSGID: 1777060498548021547 From: Andrey Konovalov Rename next_pool_required to new_pool_required. This a purely code readability change: the following patch will change stack depot to store the pointer to the new pool in a separate variable, and "new" seems like a more logical name. Reviewed-by: Alexander Potapenko Signed-off-by: Andrey Konovalov --- lib/stackdepot.c | 45 ++++++++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index e85b658be050..e428f470faf6 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -93,12 +93,11 @@ static size_t pool_offset; static DEFINE_RAW_SPINLOCK(pool_lock); /* * Stack depot tries to keep an extra pool allocated even before it runs out - * of space in the currently used pool. - * This flag marks that this next extra pool needs to be allocated and - * initialized. It has the value 0 when either the next pool is not yet - * initialized or the limit on the number of pools is reached. + * of space in the currently used pool. This flag marks whether this extra pool + * needs to be allocated. It has the value 0 when either an extra pool is not + * yet allocated or if the limit on the number of pools is reached. */ -static int next_pool_required = 1; +static int new_pool_required = 1; static int __init disable_stack_depot(char *str) { @@ -219,18 +218,18 @@ int stack_depot_init(void) } EXPORT_SYMBOL_GPL(stack_depot_init); -/* Keeps the preallocated memory to be used for the next stack depot pool. */ -static void depot_keep_next_pool(void **prealloc) +/* Keeps the preallocated memory to be used for a new stack depot pool. */ +static void depot_keep_new_pool(void **prealloc) { /* - * If the next pool is already saved or the maximum number of + * If a new pool is already saved or the maximum number of * pools is reached, do not use the preallocated memory. */ if (!next_pool_required) return; /* - * Use the preallocated memory for the next pool + * Use the preallocated memory for the new pool * as long as we do not exceed the maximum number of pools. */ if (pool_index + 1 < DEPOT_MAX_POOLS) { @@ -239,12 +238,12 @@ static void depot_keep_next_pool(void **prealloc) } /* - * At this point, either the next pool is kept or the maximum + * At this point, either a new pool is kept or the maximum * number of pools is reached. In either case, take note that * keeping another pool is not required. * smp_store_release pairs with smp_load_acquire in stack_depot_save. */ - smp_store_release(&next_pool_required, 0); + smp_store_release(&new_pool_required, 0); } /* Updates refences to the current and the next stack depot pools. */ @@ -259,7 +258,7 @@ static bool depot_update_pools(size_t required_size, void **prealloc) } /* - * Move on to the next pool. + * Move on to the new pool. * WRITE_ONCE pairs with potential concurrent read in * stack_depot_fetch. */ @@ -268,12 +267,12 @@ static bool depot_update_pools(size_t required_size, void **prealloc) /* * If the maximum number of pools is not reached, take note - * that the next pool needs to be initialized. + * that yet another new pool needs to be allocated. * smp_store_release pairs with smp_load_acquire in * stack_depot_save. */ if (pool_index + 1 < DEPOT_MAX_POOLS) - smp_store_release(&next_pool_required, 1); + smp_store_release(&new_pool_required, 1); } /* Check if the current pool is not yet allocated. */ @@ -284,9 +283,9 @@ static bool depot_update_pools(size_t required_size, void **prealloc) return true; } - /* Otherwise, try using the preallocated memory for the next pool. */ + /* Otherwise, try using the preallocated memory for a new pool. */ if (*prealloc) - depot_keep_next_pool(prealloc); + depot_keep_new_pool(prealloc); return true; } @@ -297,7 +296,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) struct stack_record *stack; size_t required_size = DEPOT_STACK_RECORD_SIZE; - /* Update current and next pools if required and possible. */ + /* Update current and new pools if required and possible. */ if (!depot_update_pools(required_size, prealloc)) return NULL; @@ -429,13 +428,13 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, goto exit; /* - * Check if another stack pool needs to be initialized. If so, allocate - * the memory now - we won't be able to do that under the lock. + * Check if another stack pool needs to be allocated. If so, allocate + * the memory now: we won't be able to do that under the lock. * * smp_load_acquire pairs with smp_store_release in depot_update_pools - * and depot_keep_next_pool. + * and depot_keep_new_pool. */ - if (unlikely(can_alloc && smp_load_acquire(&next_pool_required))) { + if (unlikely(can_alloc && smp_load_acquire(&new_pool_required))) { /* * Zero out zone modifiers, as we don't have specific zone * requirements. Keep the flags related to allocation in atomic @@ -468,9 +467,9 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, } else if (prealloc) { /* * Stack depot already contains this stack trace, but let's - * keep the preallocated memory for the future. + * keep the preallocated memory for future. */ - depot_keep_next_pool(&prealloc); + depot_keep_new_pool(&prealloc); } raw_spin_unlock_irqrestore(&pool_lock, flags); From patchwork Wed Sep 13 17:14:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139122 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp120857vqb; Wed, 13 Sep 2023 15:23:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHJdSEDLT+sD5k/mvzofdsONPve3bCeOJUNPvCpN7S+kGg7J4eSjgr9g/aqBNtpT3uwFXgX X-Received: by 2002:a05:6a21:778c:b0:14d:d9f8:83ee with SMTP id bd12-20020a056a21778c00b0014dd9f883eemr4025595pzc.61.1694643801708; Wed, 13 Sep 2023 15:23:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694643801; cv=none; d=google.com; s=arc-20160816; b=t/pBk6nfXU+e+QNigLUWQcEhhW8C4QtYKlicT1KOCGOuAHRVLKEdKDvjT35mQehIbp fPubsQEwzV0vkOrCxFhHG4zr8p9RT4712aYqwRvS3pvKufaqoLvd4r/nCixbBMv+HLpe aiC4W6jPHnAVJElSKzaGu7CgtNjYMVsrhAP4W7tvAA/lNOl2/Oi0iJp1/VidoOkFr2O1 BfxY60h9oR0k2mDj75hoqI5mPisYYxeRFYKAW8RBQC/cfhaMO8/KCquSkYZ0SX+7blUN IgJrHf1rX+M05D8jOxPzgwSF1abDatas2MTft5l4W/YNtu7E70YuLjg9ZmU3QA8f72hw qyyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iF8d5PIu0hiypd5p1X1vzHkKz0HhegtzJ4nwY5HqH4s=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=YxZZmL8bPzJD5CLb6R9/RVWzHVHgqvIfi5zoOJ1gT7qA1P26AnS5HxmKEfxASZUvg5 tIh1EsrdFWZSFTnNvnQ4UJtZuxTiVEz1HEl8P96EIGIpznPEree291+KKv2aeORj1xlV E9IFH1Vip+yi/dZU0sZPL0L2a8x+tDFLSUB0pXbuTUhQbC/MXAtggCAIjElbGzdCQ22r lJsfe4m/7oEEHQWECz6nASDVq0X/BkuG8Z18Tn0BR3a0ntlVkf59dd3WwGs10VePhCNY 3hsvwY3IYJhuBm+i8vN+ZX6tfegpKoEDW3vS1T2YOpH6hDYESYIMRFIKV87OYsxzfLDa gNlw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=wHA2oEp9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id o13-20020a17090ab88d00b00263fc986e28si2364895pjr.24.2023.09.13.15.23.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 15:23:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=wHA2oEp9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 882FD826FB73; Wed, 13 Sep 2023 10:17:10 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230453AbjIMRQH (ORCPT + 34 others); Wed, 13 Sep 2023 13:16:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230349AbjIMRQC (ORCPT ); Wed, 13 Sep 2023 13:16:02 -0400 Received: from out-221.mta1.migadu.com (out-221.mta1.migadu.com [95.215.58.221]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5FB498 for ; Wed, 13 Sep 2023 10:15:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iF8d5PIu0hiypd5p1X1vzHkKz0HhegtzJ4nwY5HqH4s=; b=wHA2oEp9M2Png7tDBErdFmtaGLdMmLLdtrI6Au1BGNIa11sMWCsUSSkjE5ssXZQMsf89kF /ktIdpIwNBEW+HewSCwtqwLFJguCHnjOmSkrdVpWer6wxYm1qnbAq12KYjmRjrBs1FoPap FFJ3/1H7E9nEL5HyfuXogMs6GDWTN4w= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 09/19] lib/stackdepot: store next pool pointer in new_pool Date: Wed, 13 Sep 2023 19:14:34 +0200 Message-Id: <3a7056408e391ff0c66b5f50c460a7b9f796228f.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:17:10 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776962818953172052 X-GMAIL-MSGID: 1776962818953172052 From: Andrey Konovalov Instead of using the last pointer in stack_pools for storing the pointer to a new pool (which does not yet store any stack records), use a new new_pool variable. This a purely code readability change: it seems more logical to store the pointer to a pool with a special meaning in a dedicated variable. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- lib/stackdepot.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index e428f470faf6..81d8733cdbed 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -85,6 +85,8 @@ static unsigned int stack_hash_mask; /* Array of memory regions that store stack traces. */ static void *stack_pools[DEPOT_MAX_POOLS]; +/* Newly allocated pool that is not yet added to stack_pools. */ +static void *new_pool; /* Currently used pool in stack_pools. */ static int pool_index; /* Offset to the unused space in the currently used pool. */ @@ -233,7 +235,7 @@ static void depot_keep_new_pool(void **prealloc) * as long as we do not exceed the maximum number of pools. */ if (pool_index + 1 < DEPOT_MAX_POOLS) { - stack_pools[pool_index + 1] = *prealloc; + new_pool = *prealloc; *prealloc = NULL; } @@ -263,6 +265,8 @@ static bool depot_update_pools(size_t required_size, void **prealloc) * stack_depot_fetch. */ WRITE_ONCE(pool_index, pool_index + 1); + stack_pools[pool_index] = new_pool; + new_pool = NULL; pool_offset = 0; /* From patchwork Wed Sep 13 17:14:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139325 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp126477vqi; Wed, 13 Sep 2023 22:31:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF7X2+n6tj3jr68WOCypLyryVgEVbWSTq6x/Ai5yf3b5PDyNZWVm79MsjSudfGmfe9s1A8U X-Received: by 2002:a54:4416:0:b0:3a9:9bb6:7fe with SMTP id k22-20020a544416000000b003a99bb607femr5060465oiw.21.1694669476921; Wed, 13 Sep 2023 22:31:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694669476; cv=none; d=google.com; s=arc-20160816; b=hH7F3PCPknCVl+th/iANNooFipBGvfaeYJSdyd2YMCKpeVvbrNA/cwLxqvMqiiXcbT 8wfnKJ58ZHVxF67a8TS2zb19WV0hZ8vrzFVEzZtIqtok0Xw3gyy2SVyKTvZa0Dvh0JSR eVv+Bk8BixrnhbWWE7tvjXjYm0ySxtYqnIw6ytX9CQQD2/DDsS5oF/9wp4H2Bvt9zOGA JlBVTTCmvgpX1U3fKBjOxS6jtL5HeDqPc78FQpsWpIpABiTREm9lo5UKoCgoqkIRm9cN LZpijO1eqYHH6NymhY8tn4o55NONDy3k1rBA6wV6OoOjEXSPfoFUWRDonjTfYson73vG yG3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=toA317coTc48scxGmgrPy+Qu0+ZANZtHQIsZ7v5QrVU=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=qKferWGlaU8ppChI/gE90y6Y/LmTcfEuFYllAUgbNwAf8slDxkfXouyDYPBvGg8f5B 2EOSzwmEcvRx49bMzlBk1ajCkdqh3/w/2V8Eu9Vj5OOInVqjqqQYQc3SOK1r2k3i2gbH MAVAiIYPy4OlhfYxGgbaUtPyS6RT/6QUVEBwqKr3BOO6PiRfLSUTw67vUjTdAhH3hcyR BRgU6hY5SuRpdg4dQg/gmVCj/hIf1mTaEG0UGbZ0Tf07EjRzXZudAl4+oX26btrSuoCD bdHLweuS0Yn5v3ZbkU31slvmQF1sI8gnK6I1YguTO614g3jLrQNazuk4LURaIy7Wdwja DFjA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=sHXeZyLg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id g22-20020a056a000b9600b0068fef323e54si879904pfj.296.2023.09.13.22.31.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 22:31:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=sHXeZyLg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 0A08B81BEC8C; Wed, 13 Sep 2023 10:16:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230476AbjIMRQM (ORCPT + 34 others); Wed, 13 Sep 2023 13:16:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230410AbjIMRQC (ORCPT ); Wed, 13 Sep 2023 13:16:02 -0400 Received: from out-212.mta1.migadu.com (out-212.mta1.migadu.com [95.215.58.212]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0BE798 for ; Wed, 13 Sep 2023 10:15:58 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=toA317coTc48scxGmgrPy+Qu0+ZANZtHQIsZ7v5QrVU=; b=sHXeZyLgBNYMFPbyE+xXimJU/cWOgsyZs9gQcIuMN7tDfd9VZD/xdZ1RUuIm3sCJedAeFu e0aDkvFDJoyGmmMnm/t7DX3XRxFvKBMSMlx3Iyv3M86jATRb7q2FVbkDE5t+RkcwVcYLOC CxMN6cHXInTQgo1LAFhXDzMsY/RvU0M= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 10/19] lib/stackdepot: store free stack records in a freelist Date: Wed, 13 Sep 2023 19:14:35 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:16:50 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776989741564359080 X-GMAIL-MSGID: 1776989741564359080 From: Andrey Konovalov Instead of using the global pool_offset variable to find a free slot when storing a new stack record, mainlain a freelist of free slots within the allocated stack pools. A global next_stack variable is used as the head of the freelist, and the next field in the stack_record struct is reused as freelist link (when the record is not in the freelist, this field is used as a link in the hash table). This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - Fix out-of-bounds when initializing a pool. --- lib/stackdepot.c | 131 +++++++++++++++++++++++++++++------------------ 1 file changed, 82 insertions(+), 49 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 81d8733cdbed..ca8e6fee0cb4 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -54,8 +54,8 @@ union handle_parts { }; struct stack_record { - struct stack_record *next; /* Link in the hash table */ - u32 hash; /* Hash in the hash table */ + struct stack_record *next; /* Link in hash table or freelist */ + u32 hash; /* Hash in hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; unsigned long entries[CONFIG_STACKDEPOT_MAX_FRAMES]; /* Frames */ @@ -87,10 +87,10 @@ static unsigned int stack_hash_mask; static void *stack_pools[DEPOT_MAX_POOLS]; /* Newly allocated pool that is not yet added to stack_pools. */ static void *new_pool; -/* Currently used pool in stack_pools. */ -static int pool_index; -/* Offset to the unused space in the currently used pool. */ -static size_t pool_offset; +/* Number of pools in stack_pools. */ +static int pools_num; +/* Next stack in the freelist of stack records within stack_pools. */ +static struct stack_record *next_stack; /* Lock that protects the variables above. */ static DEFINE_RAW_SPINLOCK(pool_lock); /* @@ -220,6 +220,42 @@ int stack_depot_init(void) } EXPORT_SYMBOL_GPL(stack_depot_init); +/* Initializes a stack depol pool. */ +static void depot_init_pool(void *pool) +{ + const int records_in_pool = DEPOT_POOL_SIZE / DEPOT_STACK_RECORD_SIZE; + int i, offset; + + /* Initialize handles and link stack records to each other. */ + for (i = 0, offset = 0; + offset <= DEPOT_POOL_SIZE - DEPOT_STACK_RECORD_SIZE; + i++, offset += DEPOT_STACK_RECORD_SIZE) { + struct stack_record *stack = pool + offset; + + stack->handle.pool_index = pools_num; + stack->handle.offset = offset >> DEPOT_STACK_ALIGN; + stack->handle.extra = 0; + + if (i < records_in_pool - 1) + stack->next = (void *)stack + DEPOT_STACK_RECORD_SIZE; + else + stack->next = NULL; + } + + /* Link stack records into the freelist. */ + WARN_ON(next_stack); + next_stack = pool; + + /* Save reference to the pool to be used by depot_fetch_stack. */ + stack_pools[pools_num] = pool; + + /* + * WRITE_ONCE pairs with potential concurrent read in + * depot_fetch_stack. + */ + WRITE_ONCE(pools_num, pools_num + 1); +} + /* Keeps the preallocated memory to be used for a new stack depot pool. */ static void depot_keep_new_pool(void **prealloc) { @@ -234,7 +270,7 @@ static void depot_keep_new_pool(void **prealloc) * Use the preallocated memory for the new pool * as long as we do not exceed the maximum number of pools. */ - if (pool_index + 1 < DEPOT_MAX_POOLS) { + if (pools_num < DEPOT_MAX_POOLS) { new_pool = *prealloc; *prealloc = NULL; } @@ -249,45 +285,42 @@ static void depot_keep_new_pool(void **prealloc) } /* Updates refences to the current and the next stack depot pools. */ -static bool depot_update_pools(size_t required_size, void **prealloc) +static bool depot_update_pools(void **prealloc) { - /* Check if there is not enough space in the current pool. */ - if (unlikely(pool_offset + required_size > DEPOT_POOL_SIZE)) { - /* Bail out if we reached the pool limit. */ - if (unlikely(pool_index + 1 >= DEPOT_MAX_POOLS)) { - WARN_ONCE(1, "Stack depot reached limit capacity"); - return false; - } + /* Check if we still have objects in the freelist. */ + if (next_stack) + goto out_keep_prealloc; - /* - * Move on to the new pool. - * WRITE_ONCE pairs with potential concurrent read in - * stack_depot_fetch. - */ - WRITE_ONCE(pool_index, pool_index + 1); - stack_pools[pool_index] = new_pool; + /* Check if we have a new pool saved and use it. */ + if (new_pool) { + depot_init_pool(new_pool); new_pool = NULL; - pool_offset = 0; - /* - * If the maximum number of pools is not reached, take note - * that yet another new pool needs to be allocated. - * smp_store_release pairs with smp_load_acquire in - * stack_depot_save. - */ - if (pool_index + 1 < DEPOT_MAX_POOLS) + /* Take note that we might need a new new_pool. */ + if (pools_num < DEPOT_MAX_POOLS) smp_store_release(&new_pool_required, 1); + + /* Try keeping the preallocated memory for new_pool. */ + goto out_keep_prealloc; + } + + /* Bail out if we reached the pool limit. */ + if (unlikely(pools_num >= DEPOT_MAX_POOLS)) { + WARN_ONCE(1, "Stack depot reached limit capacity"); + return false; } - /* Check if the current pool is not yet allocated. */ - if (*prealloc && stack_pools[pool_index] == NULL) { - /* Use the preallocated memory for the current pool. */ - stack_pools[pool_index] = *prealloc; + /* Check if we have preallocated memory and use it. */ + if (*prealloc) { + depot_init_pool(*prealloc); *prealloc = NULL; return true; } - /* Otherwise, try using the preallocated memory for a new pool. */ + return false; + +out_keep_prealloc: + /* Keep the preallocated memory for a new pool if required. */ if (*prealloc) depot_keep_new_pool(prealloc); return true; @@ -298,35 +331,35 @@ static struct stack_record * depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) { struct stack_record *stack; - size_t required_size = DEPOT_STACK_RECORD_SIZE; /* Update current and new pools if required and possible. */ - if (!depot_update_pools(required_size, prealloc)) + if (!depot_update_pools(prealloc)) return NULL; - /* Check if we have a pool to save the stack trace. */ - if (stack_pools[pool_index] == NULL) + /* Check if we have a stack record to save the stack trace. */ + stack = next_stack; + if (!stack) return NULL; + /* Advance the freelist. */ + next_stack = stack->next; + /* Limit number of saved frames to CONFIG_STACKDEPOT_MAX_FRAMES. */ if (size > CONFIG_STACKDEPOT_MAX_FRAMES) size = CONFIG_STACKDEPOT_MAX_FRAMES; /* Save the stack trace. */ - stack = stack_pools[pool_index] + pool_offset; + stack->next = NULL; stack->hash = hash; stack->size = size; - stack->handle.pool_index = pool_index; - stack->handle.offset = pool_offset >> DEPOT_STACK_ALIGN; - stack->handle.extra = 0; + /* stack->handle is already filled in by depot_init_pool. */ memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); - pool_offset += required_size; /* * Let KMSAN know the stored stack record is initialized. This shall * prevent false positive reports if instrumented code accesses it. */ - kmsan_unpoison_memory(stack, required_size); + kmsan_unpoison_memory(stack, DEPOT_STACK_RECORD_SIZE); return stack; } @@ -336,16 +369,16 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) union handle_parts parts = { .handle = handle }; /* * READ_ONCE pairs with potential concurrent write in - * depot_update_pools. + * depot_init_pool. */ - int pool_index_cached = READ_ONCE(pool_index); + int pools_num_cached = READ_ONCE(pools_num); void *pool; size_t offset = parts.offset << DEPOT_STACK_ALIGN; struct stack_record *stack; - if (parts.pool_index > pool_index_cached) { + if (parts.pool_index > pools_num_cached) { WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", - parts.pool_index, pool_index_cached, handle); + parts.pool_index, pools_num_cached, handle); return NULL; } From patchwork Wed Sep 13 17:14:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139187 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp29576vqi; Wed, 13 Sep 2023 17:50:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHzPDSDMTiQhwNMXx96m8+75vzvl0z+DJ8por+H5jOExJ3nt4ZKGvG5iogN5O1mOKHZOXME X-Received: by 2002:a67:e919:0:b0:44e:c24b:3358 with SMTP id c25-20020a67e919000000b0044ec24b3358mr4100690vso.21.1694652619091; Wed, 13 Sep 2023 17:50:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694652619; cv=none; d=google.com; s=arc-20160816; b=wIvfa2/JqWUKsUqquk+Fp6Sp08JoWXe+N2O1F/kf2uZJH4dOo4PALFQQcbs1JAnsxq jgcfWoscrCS3n0/7saUZIFQH7kM6tMhZmb1UXVT88SaJVlsO63zO2z2OvZG6h4cAKLtX 4Y5Jw57b/u0VAann8yWtQlQl6IUO/v8iH6uib3hHSVzcYs4c58bCySH8YiuqaFML+wa1 c4J8A4gp06BFeuqiJQvbTJocBRXDr2pmSUsmnL2SEIkMvdGyDfAtxPAJsUcbuhF9Ab7B rJ+3fUTuc1KVrzA/JrObEKUilH8ZQqI5zFHXoBgXuyav2GQ9VxGNkDM/CMnTCgxxzGbE 6yOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CyWM6kbTs8GrrlAHDj3nsmHfEdZOGlJXB22VnHOfO0M=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=ey58VLxK/MxxgX13Q52QVDryZmvQB0Wr0t3gkRrHOY4uS9VAKPRtjbbdfCa28ZD7yG OR5NPiy2spa20oPp5iKsTKnul1oWcTkXek54tZfeka84jCvEaazLEg7p9JM2/+tF9nEX oJu+qtKAvJ6dN/+yrlVZ+xtJ+AsOP1i2bxhB9N017LUj2IMv2Kpm9jTnWcHKnEY2UPpc rO+j3aitiOPdyLhqQS+Jg2ePk3AGexubco78RhcpP+QoxXnF5mY/EMQtevl96gEUMXf1 +NjhnZ3IwkVNAc1NCcJEr7lPFniMhxbntzV4PQBAJD0AQ/F40w991PKc0hIdrvVUG3Il yy/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=L1ENvY04; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id s28-20020a63525c000000b00577fc59373fsi18434pgl.296.2023.09.13.17.50.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 17:50:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=L1ENvY04; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 625AC821A15E; Wed, 13 Sep 2023 10:57:55 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231448AbjIMRQQ (ORCPT + 34 others); Wed, 13 Sep 2023 13:16:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230412AbjIMRQD (ORCPT ); Wed, 13 Sep 2023 13:16:03 -0400 Received: from out-218.mta1.migadu.com (out-218.mta1.migadu.com [IPv6:2001:41d0:203:375::da]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69F8B9B for ; Wed, 13 Sep 2023 10:15:59 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625357; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CyWM6kbTs8GrrlAHDj3nsmHfEdZOGlJXB22VnHOfO0M=; b=L1ENvY040875Cm/nc0yhrIVtsqNyOhOIXKPx6nLMXWM8PkG8yixRDVFUggO39BNfhc33bX af0kEGdnfi4PSIOXOiDRNlbzmSLurESzcExebX+lARTNP00ezblXTQhVuwb+ijuW97gd3o YCV/x3uEcgpqwXtFTWUA9zbuwGx2h5I= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 11/19] lib/stackdepot: use read/write lock Date: Wed, 13 Sep 2023 19:14:36 +0200 Message-Id: <5c5eca8a53ea53352794de57c87440ec509c9bbc.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:57:55 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776972064848075080 X-GMAIL-MSGID: 1776972064848075080 From: Andrey Konovalov Currently, stack depot uses the following locking scheme: 1. Lock-free accesses when looking up a stack record, which allows to have multiple users to look up records in parallel; 2. Spinlock for protecting the stack depot pools and the hash table when adding a new record. For implementing the eviction of stack traces from stack depot, the lock-free approach is not going to work anymore, as we will need to be able to also remove records from the hash table. Convert the spinlock into a read/write lock, and drop the atomic accesses, as they are no longer required. Looking up stack traces is now protected by the read lock and adding new records - by the write lock. One of the following patches will add a new function for evicting stack records, which will be protected by the write lock as well. With this change, multiple users can still look up records in parallel. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - Add lockdep_assert annotations. --- lib/stackdepot.c | 86 ++++++++++++++++++++++++++---------------------- 1 file changed, 47 insertions(+), 39 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index ca8e6fee0cb4..0b4591475d4f 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -91,15 +92,15 @@ static void *new_pool; static int pools_num; /* Next stack in the freelist of stack records within stack_pools. */ static struct stack_record *next_stack; -/* Lock that protects the variables above. */ -static DEFINE_RAW_SPINLOCK(pool_lock); /* * Stack depot tries to keep an extra pool allocated even before it runs out * of space in the currently used pool. This flag marks whether this extra pool * needs to be allocated. It has the value 0 when either an extra pool is not * yet allocated or if the limit on the number of pools is reached. */ -static int new_pool_required = 1; +static bool new_pool_required = true; +/* Lock that protects the variables above. */ +static DEFINE_RWLOCK(pool_rwlock); static int __init disable_stack_depot(char *str) { @@ -226,6 +227,8 @@ static void depot_init_pool(void *pool) const int records_in_pool = DEPOT_POOL_SIZE / DEPOT_STACK_RECORD_SIZE; int i, offset; + lockdep_assert_held_write(&pool_rwlock); + /* Initialize handles and link stack records to each other. */ for (i = 0, offset = 0; offset <= DEPOT_POOL_SIZE - DEPOT_STACK_RECORD_SIZE; @@ -248,22 +251,19 @@ static void depot_init_pool(void *pool) /* Save reference to the pool to be used by depot_fetch_stack. */ stack_pools[pools_num] = pool; - - /* - * WRITE_ONCE pairs with potential concurrent read in - * depot_fetch_stack. - */ - WRITE_ONCE(pools_num, pools_num + 1); + pools_num++; } /* Keeps the preallocated memory to be used for a new stack depot pool. */ static void depot_keep_new_pool(void **prealloc) { + lockdep_assert_held_write(&pool_rwlock); + /* * If a new pool is already saved or the maximum number of * pools is reached, do not use the preallocated memory. */ - if (!next_pool_required) + if (!new_pool_required) return; /* @@ -279,14 +279,15 @@ static void depot_keep_new_pool(void **prealloc) * At this point, either a new pool is kept or the maximum * number of pools is reached. In either case, take note that * keeping another pool is not required. - * smp_store_release pairs with smp_load_acquire in stack_depot_save. */ - smp_store_release(&new_pool_required, 0); + new_pool_required = false; } /* Updates refences to the current and the next stack depot pools. */ static bool depot_update_pools(void **prealloc) { + lockdep_assert_held_write(&pool_rwlock); + /* Check if we still have objects in the freelist. */ if (next_stack) goto out_keep_prealloc; @@ -298,7 +299,7 @@ static bool depot_update_pools(void **prealloc) /* Take note that we might need a new new_pool. */ if (pools_num < DEPOT_MAX_POOLS) - smp_store_release(&new_pool_required, 1); + new_pool_required = true; /* Try keeping the preallocated memory for new_pool. */ goto out_keep_prealloc; @@ -332,6 +333,8 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) { struct stack_record *stack; + lockdep_assert_held_write(&pool_rwlock); + /* Update current and new pools if required and possible. */ if (!depot_update_pools(prealloc)) return NULL; @@ -367,18 +370,15 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) { union handle_parts parts = { .handle = handle }; - /* - * READ_ONCE pairs with potential concurrent write in - * depot_init_pool. - */ - int pools_num_cached = READ_ONCE(pools_num); void *pool; size_t offset = parts.offset << DEPOT_STACK_ALIGN; struct stack_record *stack; - if (parts.pool_index > pools_num_cached) { + lockdep_assert_held(&pool_rwlock); + + if (parts.pool_index > pools_num) { WARN(1, "pool index %d out of bounds (%d) for stack id %08x\n", - parts.pool_index, pools_num_cached, handle); + parts.pool_index, pools_num, handle); return NULL; } @@ -420,6 +420,8 @@ static inline struct stack_record *find_stack(struct stack_record *bucket, { struct stack_record *found; + lockdep_assert_held(&pool_rwlock); + for (found = bucket; found; found = found->next) { if (found->hash == hash && found->size == size && @@ -437,6 +439,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; + bool need_alloc = false; unsigned long flags; u32 hash; @@ -456,22 +459,26 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, hash = hash_stack(entries, nr_entries); bucket = &stack_table[hash & stack_hash_mask]; - /* - * Fast path: look the stack trace up without locking. - * smp_load_acquire pairs with smp_store_release to |bucket| below. - */ - found = find_stack(smp_load_acquire(bucket), entries, nr_entries, hash); - if (found) + read_lock_irqsave(&pool_rwlock, flags); + + /* Fast path: look the stack trace up without full locking. */ + found = find_stack(*bucket, entries, nr_entries, hash); + if (found) { + read_unlock_irqrestore(&pool_rwlock, flags); goto exit; + } + + /* Take note if another stack pool needs to be allocated. */ + if (new_pool_required) + need_alloc = true; + + read_unlock_irqrestore(&pool_rwlock, flags); /* - * Check if another stack pool needs to be allocated. If so, allocate - * the memory now: we won't be able to do that under the lock. - * - * smp_load_acquire pairs with smp_store_release in depot_update_pools - * and depot_keep_new_pool. + * Allocate memory for a new pool if required now: + * we won't be able to do that under the lock. */ - if (unlikely(can_alloc && smp_load_acquire(&new_pool_required))) { + if (unlikely(can_alloc && need_alloc)) { /* * Zero out zone modifiers, as we don't have specific zone * requirements. Keep the flags related to allocation in atomic @@ -485,7 +492,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, prealloc = page_address(page); } - raw_spin_lock_irqsave(&pool_lock, flags); + write_lock_irqsave(&pool_rwlock, flags); found = find_stack(*bucket, entries, nr_entries, hash); if (!found) { @@ -494,11 +501,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, if (new) { new->next = *bucket; - /* - * smp_store_release pairs with smp_load_acquire - * from |bucket| above. - */ - smp_store_release(bucket, new); + *bucket = new; found = new; } } else if (prealloc) { @@ -509,7 +512,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, depot_keep_new_pool(&prealloc); } - raw_spin_unlock_irqrestore(&pool_lock, flags); + write_unlock_irqrestore(&pool_rwlock, flags); exit: if (prealloc) { /* Stack depot didn't use this memory, free it. */ @@ -533,6 +536,7 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, unsigned long **entries) { struct stack_record *stack; + unsigned long flags; *entries = NULL; /* @@ -544,8 +548,12 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, if (!handle || stack_depot_disabled) return 0; + read_lock_irqsave(&pool_rwlock, flags); + stack = depot_fetch_stack(handle); + read_unlock_irqrestore(&pool_rwlock, flags); + *entries = stack->entries; return stack->size; } From patchwork Wed Sep 13 17:14:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139225 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp50473vqi; Wed, 13 Sep 2023 18:46:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGVQMg9JkW+pAW0WSsvcuS1Bo+Xnr4ai/5lmae/aB4dY89FhrPhONAdz2HgOchKPIKpjSlv X-Received: by 2002:a17:90a:598a:b0:26d:1986:f7ec with SMTP id l10-20020a17090a598a00b0026d1986f7ecmr4124279pji.1.1694655989456; Wed, 13 Sep 2023 18:46:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694655989; cv=none; d=google.com; s=arc-20160816; b=ZbYpL975c1mGhugNooovzaeq6vM+BqczqGpJ3ZGoQU6nQJRbsKMaAPie9o+sk+EA5l 17yf0gJ7lu6Md08BZb0DwtXLsk52Ja3+q6ELdxrzuPOiy6G/UZJ7lo/zhazmQlYfBcqz ppY8VaEVvqJFZ3rHxjgTB1iP+rZZTQ20c1d87cmmEzpnHttxOkHk85lIgQ0aYbdV2Eoc jrtyvhmKdsig1ByPFhF8gD4SufiiiwdPpCAn9p4ShsW/mGkymG8kNzoCW/IWtEuON33F yDd6IY9GKee5e+cM6UR4CO1MgKp9LZ8mUYyrgQKA4qE+xD8D+er1bn+dFmjCwlkkIa9I LY6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4oWt0Nx5j2++AoKVMMksnS7f1HGHVd77ggPp9l5g/Uk=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=klD5BeAxsnz2YZg00H96EJ/0R9ioclvZGfWQB9/xRnk4DlOXiX7svenddL/+bAqTg/ xCaKVz/i8zYR2uDxiIK4JWcfbMERc3MF1d18dgP8ZeSZOBbk5Pas5f+0YCajyZPycvkf fasydgj1q77S13PYeEIYWVPHM6eOmnO+sclFk9HrQ03Y5FagtJIqbwMHKMbPM3ue66F8 rvB5/MItJMJdCjHm+l27GfivYFECAdffUKNv1hKLRimVkYA/XbeRN0PNz7G6FKIzxljE zYFghL/2GF4+dL0JwhKaYUimQdgwH1EiJNukDxsATchsso7OlLhAsiqC4WBo+5yPlxHc Glyg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=R3Jti2k9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id s8-20020a17090a764800b002532c9b252asi552385pjl.73.2023.09.13.18.46.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 18:46:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=R3Jti2k9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 8C0F381D2D27; Wed, 13 Sep 2023 10:17:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231871AbjIMRR0 (ORCPT + 34 others); Wed, 13 Sep 2023 13:17:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231549AbjIMRRL (ORCPT ); Wed, 13 Sep 2023 13:17:11 -0400 Received: from out-212.mta1.migadu.com (out-212.mta1.migadu.com [IPv6:2001:41d0:203:375::d4]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 991A41BC3 for ; Wed, 13 Sep 2023 10:17:01 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4oWt0Nx5j2++AoKVMMksnS7f1HGHVd77ggPp9l5g/Uk=; b=R3Jti2k9kB0yy9qtF62OlZ0XsrhyZBxEgfMpCsSO0Xtmx5zn9sCpRYhM/HxR9dhZXP7w4C yeT9mFQVFs12hS4aIQ0U95P2I39dlKW9Yh997irLZ5Edx98Q9noaq/ksCuEF2tKwOTWVLN 0q5css86Vw7XJNBgfVgPCjGAbeXVxCE= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 12/19] lib/stackdepot: use list_head for stack record links Date: Wed, 13 Sep 2023 19:14:37 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:17:38 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776975598697584711 X-GMAIL-MSGID: 1776975598697584711 From: Andrey Konovalov Switch stack_record to use list_head for links in the hash table and in the freelist. This will allow removing entries from the hash table buckets. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov --- Changes v1->v2: - Use list_head instead of open-coding backward links. --- lib/stackdepot.c | 77 ++++++++++++++++++++++++++---------------------- 1 file changed, 42 insertions(+), 35 deletions(-) diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 0b4591475d4f..1b08897ebd2b 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -55,7 +56,7 @@ union handle_parts { }; struct stack_record { - struct stack_record *next; /* Link in hash table or freelist */ + struct list_head list; /* Links in hash table or freelist */ u32 hash; /* Hash in hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; @@ -77,21 +78,21 @@ static bool __stack_depot_early_init_passed __initdata; /* Initial seed for jhash2. */ #define STACK_HASH_SEED 0x9747b28c -/* Hash table of pointers to stored stack traces. */ -static struct stack_record **stack_table; +/* Hash table of stored stack records. */ +static struct list_head *stack_table; /* Fixed order of the number of table buckets. Used when KASAN is enabled. */ static unsigned int stack_bucket_number_order; /* Hash mask for indexing the table. */ static unsigned int stack_hash_mask; -/* Array of memory regions that store stack traces. */ +/* Array of memory regions that store stack records. */ static void *stack_pools[DEPOT_MAX_POOLS]; /* Newly allocated pool that is not yet added to stack_pools. */ static void *new_pool; /* Number of pools in stack_pools. */ static int pools_num; -/* Next stack in the freelist of stack records within stack_pools. */ -static struct stack_record *next_stack; +/* Freelist of stack records within stack_pools. */ +static LIST_HEAD(free_stacks); /* * Stack depot tries to keep an extra pool allocated even before it runs out * of space in the currently used pool. This flag marks whether this extra pool @@ -123,6 +124,15 @@ void __init stack_depot_request_early_init(void) __stack_depot_early_init_requested = true; } +/* Initialize list_head's within the hash table. */ +static void init_stack_table(unsigned long entries) +{ + unsigned long i; + + for (i = 0; i < entries; i++) + INIT_LIST_HEAD(&stack_table[i]); +} + /* Allocates a hash table via memblock. Can only be used during early boot. */ int __init stack_depot_early_init(void) { @@ -152,10 +162,10 @@ int __init stack_depot_early_init(void) entries = 1UL << stack_bucket_number_order; pr_info("allocating hash table via alloc_large_system_hash\n"); stack_table = alloc_large_system_hash("stackdepot", - sizeof(struct stack_record *), + sizeof(struct list_head), entries, STACK_HASH_TABLE_SCALE, - HASH_EARLY | HASH_ZERO, + HASH_EARLY, NULL, &stack_hash_mask, 1UL << STACK_BUCKET_NUMBER_ORDER_MIN, @@ -165,6 +175,7 @@ int __init stack_depot_early_init(void) stack_depot_disabled = true; return -ENOMEM; } + init_stack_table(entries); return 0; } @@ -205,7 +216,7 @@ int stack_depot_init(void) entries = 1UL << STACK_BUCKET_NUMBER_ORDER_MAX; pr_info("allocating hash table of %lu entries via kvcalloc\n", entries); - stack_table = kvcalloc(entries, sizeof(struct stack_record *), GFP_KERNEL); + stack_table = kvcalloc(entries, sizeof(struct list_head), GFP_KERNEL); if (!stack_table) { pr_err("hash table allocation failed, disabling\n"); stack_depot_disabled = true; @@ -213,6 +224,7 @@ int stack_depot_init(void) goto out_unlock; } stack_hash_mask = entries - 1; + init_stack_table(entries); out_unlock: mutex_unlock(&stack_depot_init_mutex); @@ -224,30 +236,24 @@ EXPORT_SYMBOL_GPL(stack_depot_init); /* Initializes a stack depol pool. */ static void depot_init_pool(void *pool) { - const int records_in_pool = DEPOT_POOL_SIZE / DEPOT_STACK_RECORD_SIZE; - int i, offset; + int offset; lockdep_assert_held_write(&pool_rwlock); - /* Initialize handles and link stack records to each other. */ - for (i = 0, offset = 0; - offset <= DEPOT_POOL_SIZE - DEPOT_STACK_RECORD_SIZE; - i++, offset += DEPOT_STACK_RECORD_SIZE) { + WARN_ON(!list_empty(&free_stacks)); + + /* Initialize handles and link stack records into the freelist. */ + for (offset = 0; offset <= DEPOT_POOL_SIZE - DEPOT_STACK_RECORD_SIZE; + offset += DEPOT_STACK_RECORD_SIZE) { struct stack_record *stack = pool + offset; stack->handle.pool_index = pools_num; stack->handle.offset = offset >> DEPOT_STACK_ALIGN; stack->handle.extra = 0; - if (i < records_in_pool - 1) - stack->next = (void *)stack + DEPOT_STACK_RECORD_SIZE; - else - stack->next = NULL; + list_add(&stack->list, &free_stacks); } - /* Link stack records into the freelist. */ - WARN_ON(next_stack); - next_stack = pool; /* Save reference to the pool to be used by depot_fetch_stack. */ stack_pools[pools_num] = pool; @@ -289,7 +295,7 @@ static bool depot_update_pools(void **prealloc) lockdep_assert_held_write(&pool_rwlock); /* Check if we still have objects in the freelist. */ - if (next_stack) + if (!list_empty(&free_stacks)) goto out_keep_prealloc; /* Check if we have a new pool saved and use it. */ @@ -340,19 +346,18 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) return NULL; /* Check if we have a stack record to save the stack trace. */ - stack = next_stack; - if (!stack) + if (list_empty(&free_stacks)) return NULL; - /* Advance the freelist. */ - next_stack = stack->next; + /* Get and unlink the first entry from the freelist. */ + stack = list_first_entry(&free_stacks, struct stack_record, list); + list_del(&stack->list); /* Limit number of saved frames to CONFIG_STACKDEPOT_MAX_FRAMES. */ if (size > CONFIG_STACKDEPOT_MAX_FRAMES) size = CONFIG_STACKDEPOT_MAX_FRAMES; /* Save the stack trace. */ - stack->next = NULL; stack->hash = hash; stack->size = size; /* stack->handle is already filled in by depot_init_pool. */ @@ -414,15 +419,17 @@ int stackdepot_memcmp(const unsigned long *u1, const unsigned long *u2, } /* Finds a stack in a bucket of the hash table. */ -static inline struct stack_record *find_stack(struct stack_record *bucket, +static inline struct stack_record *find_stack(struct list_head *bucket, unsigned long *entries, int size, u32 hash) { + struct list_head *pos; struct stack_record *found; lockdep_assert_held(&pool_rwlock); - for (found = bucket; found; found = found->next) { + list_for_each(pos, bucket) { + found = list_entry(pos, struct stack_record, list); if (found->hash == hash && found->size == size && !stackdepot_memcmp(entries, found->entries, size)) @@ -435,7 +442,8 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, unsigned int nr_entries, gfp_t alloc_flags, bool can_alloc) { - struct stack_record *found = NULL, **bucket; + struct list_head *bucket; + struct stack_record *found = NULL; depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; @@ -462,7 +470,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, read_lock_irqsave(&pool_rwlock, flags); /* Fast path: look the stack trace up without full locking. */ - found = find_stack(*bucket, entries, nr_entries, hash); + found = find_stack(bucket, entries, nr_entries, hash); if (found) { read_unlock_irqrestore(&pool_rwlock, flags); goto exit; @@ -494,14 +502,13 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, write_lock_irqsave(&pool_rwlock, flags); - found = find_stack(*bucket, entries, nr_entries, hash); + found = find_stack(bucket, entries, nr_entries, hash); if (!found) { struct stack_record *new = depot_alloc_stack(entries, nr_entries, hash, &prealloc); if (new) { - new->next = *bucket; - *bucket = new; + list_add(&new->list, bucket); found = new; } } else if (prealloc) { From patchwork Wed Sep 13 17:14:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp45977vqb; Wed, 13 Sep 2023 12:43:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE9M6dIfwhFd/PHN8ipcexyq0N6csdWr3PYuXZjGndthXNd2VC1SV368Zqa3YXpMBmlX2ro X-Received: by 2002:a05:6a20:ceaf:b0:14d:e075:fc5d with SMTP id if47-20020a056a20ceaf00b0014de075fc5dmr2796525pzb.40.1694634191326; Wed, 13 Sep 2023 12:43:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694634191; cv=none; d=google.com; s=arc-20160816; b=D0gqhEFEyGgTYoB8gGEyyDzUQOPYghBY8n8XDu5hZDpTV+lqHUgo7AgCqap1q84ROf IJh3EjBC7eVOHYJPvLTZ6E1M1rPVDEUNSlYvD1Cf9NVnpbI3URmkyMA+bJWs3hC1P23v bAP7VlBw3kDLRze5eFy0x6v+bfmeYZPDgx+E+LMRyKw/Z8rCqKk1QX60MrWG8w23kqYK ZJcWcmaPJkOWsp9cG1HEIqUEcZAgvoUZkpmdF6ESjSnXvvtn78wJpXs4FMesgQqMdOh/ CDH6BVkaMHQArlxnKfw6NlcbbfXO1n5wkf+xrMcvWQX79iOOCHxDhc6uj2tXIpkg+XT7 Weng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DC9UmNBNkxmGXhCZnNWnsoHzitcm2pHd/lCALP3GjjY=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=Wdub+hsCF5rKtjRVW61KGIs5d6f/1Z5BS/TRNy2+GKLQqndpoEsc0TUdoIluzFR6BX iMLAJSYF804OlU86j/UAOV6sjWGzw4ZEym9bbdf9qKyR39ZgsGP8Di3wLhxUzG2nEsdI DB8LH/V0CXnkjb/Q5qPty9w4eT6/lImU+SpVz7zaDJWy41wlx8Uh8BdvmhuhXlU3pkQt DyJd6y+yf9wBFTtLLg1K0p2uJIBMFXiENbNIvQvpL6uYirg9r4BZSER9mt8l/kxIx7uq kVSh+7DKPeo2uqSwc3hyhFciYZjYDkT/rqGQiZH+aaJBirk5ci4IrPVsfgp2R/S9Seau d++w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=riuufySF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id cl8-20020a056a0032c800b006902507d409si520318pfb.174.2023.09.13.12.43.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 12:43:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=riuufySF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 58B1882E6C1D; Wed, 13 Sep 2023 10:18:16 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231748AbjIMRR2 (ORCPT + 34 others); Wed, 13 Sep 2023 13:17:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231696AbjIMRRL (ORCPT ); Wed, 13 Sep 2023 13:17:11 -0400 Received: from out-229.mta1.migadu.com (out-229.mta1.migadu.com [95.215.58.229]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 095181BCD for ; Wed, 13 Sep 2023 10:17:01 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625420; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DC9UmNBNkxmGXhCZnNWnsoHzitcm2pHd/lCALP3GjjY=; b=riuufySFxtO02s+5vb9CAH57DenGXrZTUkunEd+rXmQlCvLne7cXHWGh8H/JxSrJaHldo1 +Ti+81/7EXiWikASrBXA2uiFwVtaXJN9opiGB+/Z3fX20LDW1ZVMd8ZuCg94ye7sNtPe/p DZTAmOjByDol9k695KLblDDTg6NOK9s= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 13/19] kmsan: use stack_depot_save instead of __stack_depot_save Date: Wed, 13 Sep 2023 19:14:38 +0200 Message-Id: <7f6b58fda637238ffc0c240e7fd3b3a6673d9d91.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:18:16 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776952741758328306 X-GMAIL-MSGID: 1776952741758328306 From: Andrey Konovalov Make KMSAN use stack_depot_save instead of __stack_depot_save, as it always passes true to __stack_depot_save as the last argument. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - This is a new patch. --- mm/kmsan/core.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/kmsan/core.c b/mm/kmsan/core.c index 3adb4c1d3b19..5d942f19d12a 100644 --- a/mm/kmsan/core.c +++ b/mm/kmsan/core.c @@ -76,7 +76,7 @@ depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, /* Don't sleep. */ flags &= ~(__GFP_DIRECT_RECLAIM | __GFP_KSWAPD_RECLAIM); - handle = __stack_depot_save(entries, nr_entries, flags, true); + handle = stack_depot_save(entries, nr_entries, flags); return stack_depot_set_extra_bits(handle, extra); } @@ -250,11 +250,10 @@ depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id) /* * @entries is a local var in non-instrumented code, so KMSAN does not * know it is initialized. Explicitly unpoison it to avoid false - * positives when __stack_depot_save() passes it to instrumented code. + * positives when stack_depot_save() passes it to instrumented code. */ kmsan_internal_unpoison_memory(entries, sizeof(entries), false); - handle = __stack_depot_save(entries, ARRAY_SIZE(entries), __GFP_HIGH, - true); + handle = stack_depot_save(entries, ARRAY_SIZE(entries), __GFP_HIGH); return stack_depot_set_extra_bits(handle, extra_bits); } From patchwork Wed Sep 13 17:14:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139161 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp152270vqb; Wed, 13 Sep 2023 16:33:18 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHppd1ZVdVtrCWm1V6f2GkiiU6fLw8+p6UkxeZAZyYO1K3/iisxHT43muZlZuIapvaJDBBM X-Received: by 2002:a05:6808:1a0e:b0:3a7:1278:1899 with SMTP id bk14-20020a0568081a0e00b003a712781899mr5237561oib.15.1694647998310; Wed, 13 Sep 2023 16:33:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694647998; cv=none; d=google.com; s=arc-20160816; b=SSSatomTrWaakb6Y40+We+PYTrS0Eajn7FqrvLYb1P83KpNj6oiJFVI+p6sImDTA6D YERK4vK5CmrM1ROMxr0g3qy0Pmo8iPd+fOVwxjYPqtWtjunPOpzUizsJjyVuMOwtlFv7 mieWkD9Sj9yHnUzsGKl/YAf78qVFPzskGZ5R7BI8mCZONm8GNoVn0wSwUJZlSg3hjaJ0 3yLwQvlHJXDzUlEr0g/3ThgrPPY8cIt65FfrZtklIhGl6BtKFRwd9eiufyn1sjzpqso2 o9Ig2A2lulq/dwI8UzYs9Xng/+Iy0GUIzf2nAp6zOIF/ACB/86upTzBMWXEhbrXTCkcA hcWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Y6OFfwxwVZx7oqRr59fNDzzOOQmJw0k24eVuDSDilTk=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=sOAaQJDHPf6sat0ZEAc3ChUD7SfRuQxrmwdmRZIsUVsL+miKAUP27VGH7pK4q/w4dt Q0QxY9duykoIaayB6kYcnOrvh8JLGt40zUFRZlOsrkj5JRHlidRsHHyAAAIuzd16Tws5 SjCAfsuQXUupEmMDJGJnfQVmBSvdsVxYk5W8B8EBxikUOhPI8n/W6zH+3BkHkqrcuxc6 5iAbRXdLINN6WnVK/AcswdbVPIXUh5OMnBFy9j5YgvyhwpUKljb8+KCpNhjmPmMUNLpJ Fblzm0rJftLnGlh/JciXQSgKEthsHWCJl/Sup/4r2rE+fXmisKkYELs6r6I/j276Rso7 Z/8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=iZqHPx4f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id bx32-20020a056a02052000b00565f2d55050si209056pgb.737.2023.09.13.16.33.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 16:33:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=iZqHPx4f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 4C0558225CA6; Wed, 13 Sep 2023 10:17:38 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231876AbjIMRRa (ORCPT + 34 others); Wed, 13 Sep 2023 13:17:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231598AbjIMRRM (ORCPT ); Wed, 13 Sep 2023 13:17:12 -0400 Received: from out-222.mta1.migadu.com (out-222.mta1.migadu.com [95.215.58.222]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B095D212A for ; Wed, 13 Sep 2023 10:17:02 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625421; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y6OFfwxwVZx7oqRr59fNDzzOOQmJw0k24eVuDSDilTk=; b=iZqHPx4f/wcAAX6htIe3HmT20tHPT3znA2vDH4S/em2UcGFdml1qecOGFN5tBuJcXsaeSs NiEGa/AdC8wq/6VxUz6W4Joy5rfNfXtgBfp4X3vBzhkONxZ85SEY4j5PzMbE8moO/6iQxW P9SPTh0BEOQwZjBHsaixmdVGcJAg7ZU= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 14/19] lib/stackdepot, kasan: add flags to __stack_depot_save and rename Date: Wed, 13 Sep 2023 19:14:39 +0200 Message-Id: <2a161c99c47a45f8e9f7a21a732c60f0cd674a66.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:17:38 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776967219516350022 X-GMAIL-MSGID: 1776967219516350022 From: Andrey Konovalov Change the bool can_alloc argument of __stack_depot_save to a u32 argument that accepts a set of flags. The following patch will add another flag to stack_depot_save_flags besides the existing STACK_DEPOT_FLAG_CAN_ALLOC. Also rename the function to stack_depot_save_flags, as __stack_depot_save is a cryptic name, Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - This is a new patch. --- include/linux/stackdepot.h | 36 +++++++++++++++++++++++++----------- lib/stackdepot.c | 16 +++++++++++----- mm/kasan/common.c | 7 ++++--- mm/kasan/generic.c | 9 +++++---- mm/kasan/kasan.h | 2 +- mm/kasan/tags.c | 3 ++- 6 files changed, 48 insertions(+), 25 deletions(-) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index e58306783d8e..0b262e14144e 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -32,6 +32,17 @@ typedef u32 depot_stack_handle_t; */ #define STACK_DEPOT_EXTRA_BITS 5 +typedef u32 depot_flags_t; + +/* + * Flags that can be passed to stack_depot_save_flags(); see the comment next + * to its declaration for more details. + */ +#define STACK_DEPOT_FLAG_CAN_ALLOC ((depot_flags_t)0x0001) + +#define STACK_DEPOT_FLAGS_NUM 1 +#define STACK_DEPOT_FLAGS_MASK ((depot_flags_t)((1 << STACK_DEPOT_FLAGS_NUM) - 1)) + /* * Using stack depot requires its initialization, which can be done in 3 ways: * @@ -69,31 +80,34 @@ static inline int stack_depot_early_init(void) { return 0; } #endif /** - * __stack_depot_save - Save a stack trace to stack depot + * stack_depot_save_flags - Save a stack trace to stack depot * * @entries: Pointer to the stack trace * @nr_entries: Number of frames in the stack * @alloc_flags: Allocation GFP flags - * @can_alloc: Allocate stack pools (increased chance of failure if false) + * @depot_flags: Stack depot flags + * + * Saves a stack trace from @entries array of size @nr_entries. * - * Saves a stack trace from @entries array of size @nr_entries. If @can_alloc is - * %true, stack depot can replenish the stack pools in case no space is left - * (allocates using GFP flags of @alloc_flags). If @can_alloc is %false, avoids - * any allocations and fails if no space is left to store the stack trace. + * If STACK_DEPOT_FLAG_CAN_ALLOC is set in @depot_flags, stack depot can + * replenish the stack pools in case no space is left (allocates using GFP + * flags of @alloc_flags). Otherwise, stack depot avoids any allocations and + * fails if no space is left to store the stack trace. * * If the provided stack trace comes from the interrupt context, only the part * up to the interrupt entry is saved. * - * Context: Any context, but setting @can_alloc to %false is required if + * Context: Any context, but setting STACK_DEPOT_FLAG_CAN_ALLOC is required if * alloc_pages() cannot be used from the current context. Currently * this is the case for contexts where neither %GFP_ATOMIC nor * %GFP_NOWAIT can be used (NMI, raw_spin_lock). * * Return: Handle of the stack struct stored in depot, 0 on failure */ -depot_stack_handle_t __stack_depot_save(unsigned long *entries, - unsigned int nr_entries, - gfp_t gfp_flags, bool can_alloc); +depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, + unsigned int nr_entries, + gfp_t gfp_flags, + depot_flags_t depot_flags); /** * stack_depot_save - Save a stack trace to stack depot @@ -103,7 +117,7 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, * @alloc_flags: Allocation GFP flags * * Context: Contexts where allocations via alloc_pages() are allowed. - * See __stack_depot_save() for more details. + * See stack_depot_save_flags() for more details. * * Return: Handle of the stack trace stored in depot, 0 on failure */ diff --git a/lib/stackdepot.c b/lib/stackdepot.c index 1b08897ebd2b..e5121225f124 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -438,19 +438,24 @@ static inline struct stack_record *find_stack(struct list_head *bucket, return NULL; } -depot_stack_handle_t __stack_depot_save(unsigned long *entries, - unsigned int nr_entries, - gfp_t alloc_flags, bool can_alloc) +depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, + unsigned int nr_entries, + gfp_t alloc_flags, + depot_flags_t depot_flags) { struct list_head *bucket; struct stack_record *found = NULL; depot_stack_handle_t handle = 0; struct page *page = NULL; void *prealloc = NULL; + bool can_alloc = depot_flags & STACK_DEPOT_FLAG_CAN_ALLOC; bool need_alloc = false; unsigned long flags; u32 hash; + if (depot_flags & ~STACK_DEPOT_FLAGS_MASK) + return 0; + /* * If this stack trace is from an interrupt, including anything before * interrupt entry usually leads to unbounded stack depot growth. @@ -529,13 +534,14 @@ depot_stack_handle_t __stack_depot_save(unsigned long *entries, handle = found->handle.handle; return handle; } -EXPORT_SYMBOL_GPL(__stack_depot_save); +EXPORT_SYMBOL_GPL(stack_depot_save_flags); depot_stack_handle_t stack_depot_save(unsigned long *entries, unsigned int nr_entries, gfp_t alloc_flags) { - return __stack_depot_save(entries, nr_entries, alloc_flags, true); + return stack_depot_save_flags(entries, nr_entries, alloc_flags, + STACK_DEPOT_FLAG_CAN_ALLOC); } EXPORT_SYMBOL_GPL(stack_depot_save); diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 256930da578a..825a0240ec02 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include @@ -37,19 +38,19 @@ struct slab *kasan_addr_to_slab(const void *addr) return NULL; } -depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc) +depot_stack_handle_t kasan_save_stack(gfp_t flags, depot_flags_t depot_flags) { unsigned long entries[KASAN_STACK_DEPTH]; unsigned int nr_entries; nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 0); - return __stack_depot_save(entries, nr_entries, flags, can_alloc); + return stack_depot_save_flags(entries, nr_entries, flags, depot_flags); } void kasan_set_track(struct kasan_track *track, gfp_t flags) { track->pid = current->pid; - track->stack = kasan_save_stack(flags, true); + track->stack = kasan_save_stack(flags, STACK_DEPOT_FLAG_CAN_ALLOC); } #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index 4d837ab83f08..5d168c9afb32 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include #include @@ -472,7 +473,7 @@ size_t kasan_metadata_size(struct kmem_cache *cache, bool in_object) sizeof(struct kasan_free_meta) : 0); } -static void __kasan_record_aux_stack(void *addr, bool can_alloc) +static void __kasan_record_aux_stack(void *addr, depot_flags_t depot_flags) { struct slab *slab = kasan_addr_to_slab(addr); struct kmem_cache *cache; @@ -489,17 +490,17 @@ static void __kasan_record_aux_stack(void *addr, bool can_alloc) return; alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0]; - alloc_meta->aux_stack[0] = kasan_save_stack(0, can_alloc); + alloc_meta->aux_stack[0] = kasan_save_stack(0, depot_flags); } void kasan_record_aux_stack(void *addr) { - return __kasan_record_aux_stack(addr, true); + return __kasan_record_aux_stack(addr, STACK_DEPOT_FLAG_CAN_ALLOC); } void kasan_record_aux_stack_noalloc(void *addr) { - return __kasan_record_aux_stack(addr, false); + return __kasan_record_aux_stack(addr, 0); } void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index f70e3d7a602e..de3206e11888 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -370,7 +370,7 @@ static inline void kasan_init_cache_meta(struct kmem_cache *cache, unsigned int static inline void kasan_init_object_meta(struct kmem_cache *cache, const void *object) { } #endif -depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc); +depot_stack_handle_t kasan_save_stack(gfp_t flags, depot_flags_t depot_flags); void kasan_set_track(struct kasan_track *track, gfp_t flags); void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags); void kasan_save_free_info(struct kmem_cache *cache, void *object); diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index 7dcfe341d48e..4fd32121b0fd 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -101,7 +102,7 @@ static void save_stack_info(struct kmem_cache *cache, void *object, struct kasan_stack_ring_entry *entry; void *old_ptr; - stack = kasan_save_stack(gfp_flags, true); + stack = kasan_save_stack(gfp_flags, STACK_DEPOT_FLAG_CAN_ALLOC); /* * Prevent save_stack_info() from modifying stack ring From patchwork Wed Sep 13 17:14:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139079 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp74779vqb; Wed, 13 Sep 2023 13:41:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGI7LZtr5oASSN7Dtes09U4T8MuVz4DKhv39d7fClwY0iR8qff/p59nxEgGjKKxlY4DXZx7 X-Received: by 2002:a05:6871:5c7:b0:1c0:1424:5e6c with SMTP id v7-20020a05687105c700b001c014245e6cmr4079020oan.17.1694637669849; Wed, 13 Sep 2023 13:41:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694637669; cv=none; d=google.com; s=arc-20160816; b=0qRZa/YixNa6qVdtDESi7dqH0iirVvua5jmlgiDwIlHH+iLr9E/4DBIGSuWPbtQDMc 84iAIDDaTHHmjLvtEifxJXj6DmDecBG21GoC4i1Rj9azjzkXqYhWC3YGPe9Rn/dehWzF 04jQItkEuoGhlroZiU0cYJp0+XHaJf1LX12PC4iNQDtmLvIONexFNVDWKhFEvtuHqDmK ETaFG3nukPeQzTEtl2HZXid3fUB4HFWn/0SJsziPaLObysfU3My6ZYGTB3KP/TjkoZMv 7CUaqOPCqTkc6XdWHz8UD/E+tsGWpQDt+vryZkFOEErYQMRlmXUVeX+XRDemaaJA5KTz e0/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iWiSl9lpWFlYTb7e84Z8P7HowCNwXEfuw/46H+rnea4=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=UEJQT/a7xp1Z77DrhYe4QN4kEQmLvdOhBA/3Z3MvGxMFzKkBervdGX2VidkDoza7Wz 9173KN+ObDAqsy7dPTi7zoBTTx0sitB396t938jVY5d2UKPTs1x743FgAxeoLt0xufTk VsO9IsA6m7noA2WBzPaJZItcdtKt6PnpPX1iEgd+B79WszsI2EJEyB5lL5BMInYgGkOE gWwn9PvZYDJ8XrOOBryeKXwo8Jg5ChW3rQHDyEecqdz/DeRE2YtzNSTnoW2wk8X7JwP6 VJLurE9j9rjnqcPlkLdyXDV73hxvosADgdbLkGs9daByqaYF0KYP+m0xuDAuumavj7bI Bmbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=rPLyIHlB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id m1-20020a632601000000b0056417c8a310si10425470pgm.204.2023.09.13.13.40.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 13:41:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=rPLyIHlB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id E2B9F81129BD; Wed, 13 Sep 2023 10:17:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231892AbjIMRRd (ORCPT + 34 others); Wed, 13 Sep 2023 13:17:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231807AbjIMRRN (ORCPT ); Wed, 13 Sep 2023 13:17:13 -0400 Received: from out-227.mta1.migadu.com (out-227.mta1.migadu.com [95.215.58.227]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F5682134 for ; Wed, 13 Sep 2023 10:17:03 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625421; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iWiSl9lpWFlYTb7e84Z8P7HowCNwXEfuw/46H+rnea4=; b=rPLyIHlBbsaWO60jp650y+oiahqwCIrJ0H6GB7tgNClg622J9RmQzHLj0jZ3Mte7u2yjvx FoH7zGtya7Ki6Pakd+45misXntm3w9rYsO9/Q7XXHlRkAQ3iapWRAe7bP72+7cHr7LmOPW waAC4TKXVoO/HQ0rZPchMh9OZqw2yzk= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 15/19] lib/stackdepot: add refcount for records Date: Wed, 13 Sep 2023 19:14:40 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:17:37 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776956388808209076 X-GMAIL-MSGID: 1776956388808209076 From: Andrey Konovalov Add a reference counter for how many times a stack records has been added to stack depot. Add a new STACK_DEPOT_FLAG_GET flag to stack_depot_save_flags that instructs the stack depot to increment the refcount. Do not yet decrement the refcount; this is implemented in one of the following patches. Do not yet enable any users to use the flag to avoid overflowing the refcount. This is preparatory patch for implementing the eviction of stack records from the stack depot. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - Add forgotten refcount_inc() under write lock. - Add STACK_DEPOT_FLAG_GET flag for stack_depot_save_flags. --- include/linux/stackdepot.h | 13 ++++++++++--- lib/stackdepot.c | 12 ++++++++++-- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index 0b262e14144e..611716702d73 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -39,8 +39,9 @@ typedef u32 depot_flags_t; * to its declaration for more details. */ #define STACK_DEPOT_FLAG_CAN_ALLOC ((depot_flags_t)0x0001) +#define STACK_DEPOT_FLAG_GET ((depot_flags_t)0x0002) -#define STACK_DEPOT_FLAGS_NUM 1 +#define STACK_DEPOT_FLAGS_NUM 2 #define STACK_DEPOT_FLAGS_MASK ((depot_flags_t)((1 << STACK_DEPOT_FLAGS_NUM) - 1)) /* @@ -94,6 +95,9 @@ static inline int stack_depot_early_init(void) { return 0; } * flags of @alloc_flags). Otherwise, stack depot avoids any allocations and * fails if no space is left to store the stack trace. * + * If STACK_DEPOT_FLAG_GET is set in @depot_flags, stack depot will increment + * the refcount on the saved stack trace if it already exists in stack depot. + * * If the provided stack trace comes from the interrupt context, only the part * up to the interrupt entry is saved. * @@ -116,8 +120,11 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, * @nr_entries: Number of frames in the stack * @alloc_flags: Allocation GFP flags * - * Context: Contexts where allocations via alloc_pages() are allowed. - * See stack_depot_save_flags() for more details. + * Does not increment the refcount on the saved stack trace; see + * stack_depot_save_flags() for more details. + * + * Context: Contexts where allocations via alloc_pages() are allowed; + * see stack_depot_save_flags() for more details. * * Return: Handle of the stack trace stored in depot, 0 on failure */ diff --git a/lib/stackdepot.c b/lib/stackdepot.c index e5121225f124..e2c622054265 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -60,6 +61,7 @@ struct stack_record { u32 hash; /* Hash in hash table */ u32 size; /* Number of stored frames */ union handle_parts handle; + refcount_t count; unsigned long entries[CONFIG_STACKDEPOT_MAX_FRAMES]; /* Frames */ }; @@ -361,6 +363,7 @@ depot_alloc_stack(unsigned long *entries, int size, u32 hash, void **prealloc) stack->hash = hash; stack->size = size; /* stack->handle is already filled in by depot_init_pool. */ + refcount_set(&stack->count, 1); memcpy(stack->entries, entries, flex_array_size(stack, entries, size)); /* @@ -477,6 +480,8 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, /* Fast path: look the stack trace up without full locking. */ found = find_stack(bucket, entries, nr_entries, hash); if (found) { + if (depot_flags & STACK_DEPOT_FLAG_GET) + refcount_inc(&found->count); read_unlock_irqrestore(&pool_rwlock, flags); goto exit; } @@ -516,12 +521,15 @@ depot_stack_handle_t stack_depot_save_flags(unsigned long *entries, list_add(&new->list, bucket); found = new; } - } else if (prealloc) { + } else { + if (depot_flags & STACK_DEPOT_FLAG_GET) + refcount_inc(&found->count); /* * Stack depot already contains this stack trace, but let's * keep the preallocated memory for future. */ - depot_keep_new_pool(&prealloc); + if (prealloc) + depot_keep_new_pool(&prealloc); } write_unlock_irqrestore(&pool_rwlock, flags); From patchwork Wed Sep 13 17:14:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139188 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp30085vqi; Wed, 13 Sep 2023 17:51:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEPXCdEiJXuek6Q2BivV9BAZCvW3Z9p60J29bs48RgbJzte8sCkpzM1qQmCsoMXpJ2eeOe6 X-Received: by 2002:a17:902:7446:b0:1bc:9794:22ef with SMTP id e6-20020a170902744600b001bc979422efmr3952991plt.1.1694652701448; Wed, 13 Sep 2023 17:51:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694652701; cv=none; d=google.com; s=arc-20160816; b=JjOnRuAT27Yhbd2/GiNgv2YhZ+MSYXcMgwulgrhisRcSvB2HrNUmkqM/D1alclvbLi Fwzo7E7WE8JUwX/F6ivgG82LbL4bm1cAEc+wO0c3jQxAC5mc8kge6jZTeKb6fMkgAFfK eMRpYxt7/KGkZ5EWxDT0HaJeO0b9o2YqZduH6sSt2bUkFSvbhzXnqotfeDFUN53Nl451 eLFFIGNYScdGZ+tHGstHuhYq7v2uvMj16SAZ9KD09XOX8bAlYALpfsV4y0dOy7zMZUqC QJHnDuhVDpz7LJfd2d6pUIoxlcWJ2bLjemZ8qmutxztbD8+1xnvSOhh0ge1QAlT0vPjS fHqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AuXPIx9M1sa3aovsp0XdlaM599Z45TGxV3EusWHlaB4=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=VBatGnqIJAzMLaXpi1K3Ba6TgUA7lC+VTswGZqMpKuv4ermDiBdnWrZhZgGOP8PiX0 4nbv8IEht75a5UJOhUpzhC3Eqsjss6e9utquBAQuyIiAQL9YtB0ux71MJiqUNN8jDCxN LLiXj4CeCNG5HJk20no68P7jI9mToL7J1D1SGlsSoahgAzV95Dv5uExFTVhruUjyhA7v CZ8jMNN+OJGSXHIITYojN0baYo7kGpkwloXdgBDMWcDGKwX548bstRZX1FuVaY4oK4QA 0xcEQ1B7JokB81o7SUs57o2LaMtIxsNJU5wlZDR0iKzi1yB47/hAJx3Laee/7uJRJ1L0 WrlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=F2Ii6+s+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id h8-20020a170902f54800b001b8a2774d60si505135plf.486.2023.09.13.17.51.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 17:51:41 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=F2Ii6+s+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 7096C823367F; Wed, 13 Sep 2023 10:17:39 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230451AbjIMRRi (ORCPT + 34 others); Wed, 13 Sep 2023 13:17:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231577AbjIMRRO (ORCPT ); Wed, 13 Sep 2023 13:17:14 -0400 Received: from out-220.mta1.migadu.com (out-220.mta1.migadu.com [IPv6:2001:41d0:203:375::dc]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C95EA213F for ; Wed, 13 Sep 2023 10:17:03 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AuXPIx9M1sa3aovsp0XdlaM599Z45TGxV3EusWHlaB4=; b=F2Ii6+s+cdiC4H7z0xYGquHm32HzsvJdDEssh13Gmfxh6dgxf0ImSZW90i2I0Ul8XHXFZN EabRGGiqHKx8/fjEiRgDu0SjJmNsgtQaiOLP1FFmQ02FkMsAmcJU65Zs5fSzoSf3rveOpt b+/xe3k7MacpwOlBvRZDIYPeX/d3pF0= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 16/19] lib/stackdepot: allow users to evict stack traces Date: Wed, 13 Sep 2023 19:14:41 +0200 Message-Id: <1a3ea1e6b82f95eea644875756e12daba6c4cc7f.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:17:39 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776972151120190874 X-GMAIL-MSGID: 1776972151120190874 From: Andrey Konovalov Add stack_depot_put, a function that decrements the reference counter on a stack record and removes it from the stack depot once the counter reaches 0. Internally, when removing a stack record, the function unlinks it from the hash table bucket and returns to the freelist. With this change, the users of stack depot can call stack_depot_put when keeping a stack trace in the stack depot is not needed anymore. This allows avoiding polluting the stack depot with irrelevant stack traces and thus have more space to store the relevant ones before the stack depot reaches its capacity. Signed-off-by: Andrey Konovalov --- Changes v1->v2: - Comments fixes as suggested by Marco. - Add lockdep_assert annotation. - Adapt to using list_head's. - Rename stack_depot_evict to stack_depot_put. --- include/linux/stackdepot.h | 14 ++++++++++++++ lib/stackdepot.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index 611716702d73..a6796f178913 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -97,6 +97,8 @@ static inline int stack_depot_early_init(void) { return 0; } * * If STACK_DEPOT_FLAG_GET is set in @depot_flags, stack depot will increment * the refcount on the saved stack trace if it already exists in stack depot. + * Users of this flag must also call stack_depot_put() when keeping the stack + * trace is no longer required to avoid overflowing the refcount. * * If the provided stack trace comes from the interrupt context, only the part * up to the interrupt entry is saved. @@ -162,6 +164,18 @@ void stack_depot_print(depot_stack_handle_t stack); int stack_depot_snprint(depot_stack_handle_t handle, char *buf, size_t size, int spaces); +/** + * stack_depot_put - Drop a reference to a stack trace from stack depot + * + * @handle: Stack depot handle returned from stack_depot_save() + * + * The stack trace is evicted from stack depot once all references to it have + * been dropped (once the number of stack_depot_evict() calls matches the + * number of stack_depot_save_flags() calls with STACK_DEPOT_FLAG_GET set for + * this stack trace). + */ +void stack_depot_put(depot_stack_handle_t handle); + /** * stack_depot_set_extra_bits - Set extra bits in a stack depot handle * diff --git a/lib/stackdepot.c b/lib/stackdepot.c index e2c622054265..56f2abc03717 100644 --- a/lib/stackdepot.c +++ b/lib/stackdepot.c @@ -398,6 +398,14 @@ static struct stack_record *depot_fetch_stack(depot_stack_handle_t handle) return stack; } +/* Links stack into the freelist. */ +static void depot_free_stack(struct stack_record *stack) +{ + lockdep_assert_held_write(&pool_rwlock); + + list_add(&stack->list, &free_stacks); +} + /* Calculates the hash for a stack. */ static inline u32 hash_stack(unsigned long *entries, unsigned int size) { @@ -580,6 +588,33 @@ unsigned int stack_depot_fetch(depot_stack_handle_t handle, } EXPORT_SYMBOL_GPL(stack_depot_fetch); +void stack_depot_put(depot_stack_handle_t handle) +{ + struct stack_record *stack; + unsigned long flags; + + if (!handle || stack_depot_disabled) + return; + + write_lock_irqsave(&pool_rwlock, flags); + + stack = depot_fetch_stack(handle); + if (WARN_ON(!stack)) + goto out; + + if (refcount_dec_and_test(&stack->count)) { + /* Unlink stack from the hash table. */ + list_del(&stack->list); + + /* Free stack. */ + depot_free_stack(stack); + } + +out: + write_unlock_irqrestore(&pool_rwlock, flags); +} +EXPORT_SYMBOL_GPL(stack_depot_put); + void stack_depot_print(depot_stack_handle_t stack) { unsigned long *entries; From patchwork Wed Sep 13 17:14:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139008 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp2437vqb; Wed, 13 Sep 2023 11:26:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE0j4Pt+yNoL7qW2zi/RhY7JmWCaS3UQxIYldzPE1jJ5wNnsAhGSi59T0yQh2/Xhy9rfuCG X-Received: by 2002:a05:6a20:560f:b0:13b:d080:aa03 with SMTP id ir15-20020a056a20560f00b0013bd080aa03mr2989531pzc.60.1694629617405; Wed, 13 Sep 2023 11:26:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694629617; cv=none; d=google.com; s=arc-20160816; b=lEhpCDe8ljfezuR2LfavZHWe5S0u2n0Q1TzZ4aWWV1zoL6JAiz4LQV4fGwdF3wzJX4 tCfQTwSuft+e7z90bLRO3ernUeQ9ERWOiGPF2H8EiSzbmSEDVRHBZU3fnDJMxvKlpgcI HeSr9T2u7oYY5iWO/+z0SBut3QCrRYrseXKilnu2/kGM74qWZRxJt8Poh9S/U00zIdkx ZMpi6ClNHKp1VThYs+jnlImSXAh7nb1Uydu/wfFbe4m/xJuMhCNHzqf8B7jP0kVRLN/1 /khrtPLuBh7R1bYZvZvw5SqBwxFrM6OHqqhAobtwgFDSlbRceKhY/OEmoyizDSkK48/L Gp6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=P7zzoSFiMvzmGoSw/qnaD/dTzTEPv955C3ixzUuxzB0=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=e2Tz9g0ly3i5VpAqOGez0+38BKFKZ0qDf3XvwJMdhmwpjTAsbcHR5xOonqg2PKtICW idvxqA7ew25yqjgvG5fNUxGH2t1xT5MDX2S8iFvHMb6zMDWvaNEBZgQiMEFYBYASTO4C hxqL+E3aLbY54kBw0XEy1aRek2VPRDCQXwYiAC4uTKMkc5aZYBDXQITaYPphFGUps0+v usloh2egt4FEUOlSXuy/uQcX26ymQPQ6IiUFvnQ/Tj6kO5Kctggz5tStLmYdj0Idrjnt oxQOw0KigqzpIbd8j8Zsmm7PP/JrYNN4dGa75KUHuVTMLpk/KUkK1XltjqY6C9pNdv+L L1vg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=dUDWfm9G; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id h71-20020a63834a000000b0057777451a73si6209474pge.703.2023.09.13.11.26.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 11:26:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=dUDWfm9G; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id A0457830D343; Wed, 13 Sep 2023 10:17:46 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231255AbjIMRRm (ORCPT + 34 others); Wed, 13 Sep 2023 13:17:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231757AbjIMRRP (ORCPT ); Wed, 13 Sep 2023 13:17:15 -0400 Received: from out-220.mta1.migadu.com (out-220.mta1.migadu.com [IPv6:2001:41d0:203:375::dc]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 422D72685 for ; Wed, 13 Sep 2023 10:17:04 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625422; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P7zzoSFiMvzmGoSw/qnaD/dTzTEPv955C3ixzUuxzB0=; b=dUDWfm9GQJCwXn5Ns2lo8cMMrhaO+r45+sngvwNP0vGullsCgE1WTcWwBv1boZZKILIxby Iwf95QD/YaEE0XqYwS1ZkwuDWLFwJogmyau0zsHzueAIHLAi5uTqqsd2dYHi8ddYBLSeuh ZXP72m6TnkiIFBuVA1VHgbYB3a0DFuM= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 17/19] kasan: remove atomic accesses to stack ring entries Date: Wed, 13 Sep 2023 19:14:42 +0200 Message-Id: <556085476eb7d2e3703d62dc2fa920931aadf459.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:17:46 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776947945585375501 X-GMAIL-MSGID: 1776947945585375501 From: Andrey Konovalov Remove the atomic accesses to entry fields in save_stack_info and kasan_complete_mode_report_info for tag-based KASAN modes. These atomics are not required, as the read/write lock prevents the entries from being read (in kasan_complete_mode_report_info) while being written (in save_stack_info) and the try_cmpxchg prevents the same entry from being rewritten (in save_stack_info) in the unlikely case of wrapping during writing. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - This is a new patch. --- mm/kasan/report_tags.c | 25 +++++++------------------ mm/kasan/tags.c | 13 +++++-------- 2 files changed, 12 insertions(+), 26 deletions(-) diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c index 8b8bfdb3cfdb..78abdcde5da9 100644 --- a/mm/kasan/report_tags.c +++ b/mm/kasan/report_tags.c @@ -31,10 +31,6 @@ void kasan_complete_mode_report_info(struct kasan_report_info *info) unsigned long flags; u64 pos; struct kasan_stack_ring_entry *entry; - void *ptr; - u32 pid; - depot_stack_handle_t stack; - bool is_free; bool alloc_found = false, free_found = false; if ((!info->cache || !info->object) && !info->bug_type) { @@ -61,18 +57,11 @@ void kasan_complete_mode_report_info(struct kasan_report_info *info) entry = &stack_ring.entries[i % stack_ring.size]; - /* Paired with smp_store_release() in save_stack_info(). */ - ptr = (void *)smp_load_acquire(&entry->ptr); - - if (kasan_reset_tag(ptr) != info->object || - get_tag(ptr) != get_tag(info->access_addr)) + if (kasan_reset_tag(entry->ptr) != info->object || + get_tag(entry->ptr) != get_tag(info->access_addr)) continue; - pid = READ_ONCE(entry->pid); - stack = READ_ONCE(entry->stack); - is_free = READ_ONCE(entry->is_free); - - if (is_free) { + if (entry->is_free) { /* * Second free of the same object. * Give up on trying to find the alloc entry. @@ -80,8 +69,8 @@ void kasan_complete_mode_report_info(struct kasan_report_info *info) if (free_found) break; - info->free_track.pid = pid; - info->free_track.stack = stack; + info->free_track.pid = entry->pid; + info->free_track.stack = entry->stack; free_found = true; /* @@ -95,8 +84,8 @@ void kasan_complete_mode_report_info(struct kasan_report_info *info) if (alloc_found) break; - info->alloc_track.pid = pid; - info->alloc_track.stack = stack; + info->alloc_track.pid = entry->pid; + info->alloc_track.stack = entry->stack; alloc_found = true; /* diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index 4fd32121b0fd..b6c017e670d8 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -121,15 +121,12 @@ static void save_stack_info(struct kmem_cache *cache, void *object, if (!try_cmpxchg(&entry->ptr, &old_ptr, STACK_RING_BUSY_PTR)) goto next; /* Busy slot. */ - WRITE_ONCE(entry->size, cache->object_size); - WRITE_ONCE(entry->pid, current->pid); - WRITE_ONCE(entry->stack, stack); - WRITE_ONCE(entry->is_free, is_free); + entry->size = cache->object_size; + entry->pid = current->pid; + entry->stack = stack; + entry->is_free = is_free; - /* - * Paired with smp_load_acquire() in kasan_complete_mode_report_info(). - */ - smp_store_release(&entry->ptr, (s64)object); + entry->ptr = object; read_unlock_irqrestore(&stack_ring.lock, flags); } From patchwork Wed Sep 13 17:14:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139029 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp10597vqb; Wed, 13 Sep 2023 11:42:05 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF7hS22u/MPVF7YC3W89Vpl8CDghfYIL6DPE6xR9eCsEvTQ46os4Umq/jYsC3S1osHYKiQr X-Received: by 2002:a17:902:dacd:b0:1c3:3f7b:9cf1 with SMTP id q13-20020a170902dacd00b001c33f7b9cf1mr4908623plx.0.1694630524910; Wed, 13 Sep 2023 11:42:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694630524; cv=none; d=google.com; s=arc-20160816; b=XqGYi3EkU6P1rkwjILUkOn+mSE3MRDEeCdaAz1qA5ZFd0sINDH+5PbS5WKRYW+B4pn wtPW3jdACFoVzn77GjBLjZnO5EA/sUYbSzPHuFEY7jCthx3Iei6YERyYqcNL0w9Zke2d m2aLCLlKoaWqyx8xxT/UPeeU1hD7jWY7JihZ5nCHj8Z5SHhR/CPHYN79SnQLO2UbwEci k1djK/3D67XvBbawtKiaDFo9tfFlHHW2zz8g9+ybzsoUmI98S3siyZkdmsJRUTlTjbKl jx56rndSU5Z8iAVSCbqWK08rTQmf++KnCLH2596A+yCYBE2K2nJncogbbfQcqCS2vKgY k7XQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=gGKJxtryqSg7zyzOR+J62w3w1hKrmZ/huaWffkkgHB8=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=RAhJqh4Z2q3+XxTWKSEOB2DgxOjAMa+TntACN0PET+UDUrXf6anhgtynFXHIFZrKJw d8hyOkecI0HdiN0amVy0Eq9RUay8SAgghMA3eVJK0POCnK3nwIAWylJJbFX9lsBOAWSv 8XM7LhOK30vpchTYDNUjLlhX3R8yfISa1lu8SQ6s4NIcd5QJpyTk0aVvT88G5mjMjjH7 jW9eeT+J1L9ETekii9U5Yi+3bKLxS1tKpqJp9GskwVWevqPnCqUibhvbjd+lStwZ7qxJ DGkUI7N0E0WLjUFDrJj6j0ot+aBIWENZKZic9SsZyvZJrMB5EaAQ7ArOn9ePaCfjRipu 5piQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Kykiqu4a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id cp6-20020a170902e78600b001b89ce303e4si10716151plb.195.2023.09.13.11.42.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 11:42:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Kykiqu4a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 65C308311B5B; Wed, 13 Sep 2023 10:18:54 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231777AbjIMRSt (ORCPT + 34 others); Wed, 13 Sep 2023 13:18:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231202AbjIMRSd (ORCPT ); Wed, 13 Sep 2023 13:18:33 -0400 Received: from out-227.mta1.migadu.com (out-227.mta1.migadu.com [95.215.58.227]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64A0A1FD4 for ; Wed, 13 Sep 2023 10:18:06 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gGKJxtryqSg7zyzOR+J62w3w1hKrmZ/huaWffkkgHB8=; b=Kykiqu4aZApYf03YebHa3Dl/Iiz3dyOXmVXrrBHmpydAQBwIdJhL3HLDTzwCu05bpW7SeU 7532l7l5s0ccuWBVga6HibBP/hh1Jt3T/r7Xexqvuo0EKwyd5lqVv2v2D3IiN0BK5IbTfw g5lfY7rNPM1SE6bV/kcFjLK6Jf0r1ds= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 18/19] kasan: check object_size in kasan_complete_mode_report_info Date: Wed, 13 Sep 2023 19:14:43 +0200 Message-Id: <293d73bcd89932bc026263d3df8ee281ad3f621f.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:18:54 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776948897101690339 X-GMAIL-MSGID: 1776948897101690339 From: Andrey Konovalov Check the object size when looking up entries in the stack ring. If the size of the object for which a report is being printed does not match the size of the object for which a stack trace has been saved in the stack ring, the saved stack trace is irrelevant. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - This is a new patch. --- mm/kasan/report_tags.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c index 78abdcde5da9..98c238ba3545 100644 --- a/mm/kasan/report_tags.c +++ b/mm/kasan/report_tags.c @@ -58,7 +58,8 @@ void kasan_complete_mode_report_info(struct kasan_report_info *info) entry = &stack_ring.entries[i % stack_ring.size]; if (kasan_reset_tag(entry->ptr) != info->object || - get_tag(entry->ptr) != get_tag(info->access_addr)) + get_tag(entry->ptr) != get_tag(info->access_addr) || + info->cache->object_size != entry->size) continue; if (entry->is_free) { From patchwork Wed Sep 13 17:14:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: andrey.konovalov@linux.dev X-Patchwork-Id: 139037 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp16303vqb; Wed, 13 Sep 2023 11:52:40 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH7sq1WO1ybhUeP9ogaMWcABeznPVR0hwNH11mIuRNF8uOj6eeptosLD3ki7LZfqoSuwHHc X-Received: by 2002:a17:902:e84a:b0:1b5:64a4:bea0 with SMTP id t10-20020a170902e84a00b001b564a4bea0mr4356793plg.10.1694631160396; Wed, 13 Sep 2023 11:52:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694631160; cv=none; d=google.com; s=arc-20160816; b=TBRWV+zUaXQxROkI0EaHTlXmW2Xwlrva06NKsT6Pu1W5lJ1tLXxMCUXwNV75rAvnv1 xEPsd+ZNuexwOwLKOEkTvwLcaRz2wtoiIuv/io5pOCkIkIA39FvTr5gUQ/sbXyGlUBrX f74hW3MNgiTrtK5KFi0fPb4J+R3Jvg+u5C8kw10wN4AdpwEVRilgVDBwN8abeImefVI0 gwVz28zrCCYvfBDyRCnvCftagdNs932YwRXicpXnq3SrMA0axbnp914w8sumn87tVkNp 3wdwxJ2U5mDI5SL8e0mgdr0GzTAvE0TUzYX7vol0iGUtE8bYGfclquwMR0hCO9AeYmxq 1aSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qjH3MV6DYl1G1UYfCD5hIyhq0uUm2VbQwACKU2fMzJc=; fh=nfQSbTp1dWHt2Ier1Up8UhVNdXOqoJJLNvTrpT3jJEk=; b=0ftS7AN3FcruSJkwb1H8wN8SJOrHs3caUV5LEgfK2/5CWXEQjZ60Pv+PxhViU9rtU5 P5814AGCXHiHnp3hTLXVFB+b3h6yRo7ZdKHk6wMVTfvx5Xs9raD6r+W0Nh5bjasUIBv3 56FYyIcnci0j8KjOOnfFxW4PeEBf9JfMajWFB67EY/h8i/Po+Vss91bNZw1rlOZXUydZ lIWJXiy5bT0oTE34XsCvQ5RxUU+YTB6A6R6wFoPL4e3/x1pH/L7xU1JVtISAUpMl0lAH FfgZq4iwWqwnOhbK+sSGVY+nqrkbc9XRXvqsoemEor5t8Ib08f2XjDjMd7Dgd5U77MXz 6dMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=aOUyG6zN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id p11-20020a170902eacb00b001b89b77ce99si10533386pld.458.2023.09.13.11.52.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 11:52:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=aOUyG6zN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id B8F4182771A3; Wed, 13 Sep 2023 10:19:09 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231431AbjIMRSw (ORCPT + 34 others); Wed, 13 Sep 2023 13:18:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231425AbjIMRSd (ORCPT ); Wed, 13 Sep 2023 13:18:33 -0400 Received: from out-214.mta1.migadu.com (out-214.mta1.migadu.com [IPv6:2001:41d0:203:375::d6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 159AA2108 for ; Wed, 13 Sep 2023 10:18:06 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1694625485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qjH3MV6DYl1G1UYfCD5hIyhq0uUm2VbQwACKU2fMzJc=; b=aOUyG6zNU4qOExhkFzpcOcmJrKwkx0ZsS/bVVWhskgw/aBPt6S1+VAaKP4oT7FsM6oyJVg +IUqSRli1eaRlhDIRzKFieirB2EVqYsJdtwVbWJ4TY/mTzb+JzNUxW7Lbi7K2Q5uY06kf1 yBI+2ElH2T2LDtZB7yOsWKeboKVByXg= From: andrey.konovalov@linux.dev To: Marco Elver , Alexander Potapenko Cc: Andrey Konovalov , Dmitry Vyukov , Vlastimil Babka , kasan-dev@googlegroups.com, Evgenii Stepanov , Oscar Salvador , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrey Konovalov Subject: [PATCH v2 19/19] kasan: use stack_depot_put for tag-based modes Date: Wed, 13 Sep 2023 19:14:44 +0200 Message-Id: <6e2367e7693aa107f05c649abe06180fff847bb4.1694625260.git.andreyknvl@google.com> In-Reply-To: References: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:19:09 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776949564094579167 X-GMAIL-MSGID: 1776949564094579167 From: Andrey Konovalov Make tag-based KASAN modes to evict stack traces from the stack depot once they are evicted from the stack ring. Internally, pass STACK_DEPOT_FLAG_GET to stack_depot_save_flags (via kasan_save_stack) to increment the refcount when saving a new entry to stack ring and call stack_depot_put when removing an entry from stack ring. Signed-off-by: Andrey Konovalov Reviewed-by: Alexander Potapenko --- Changes v1->v2: - Adapt to the stack depot API change. - Drop READ_ONCE when reading entry->stack. --- mm/kasan/report_tags.c | 1 + mm/kasan/tags.c | 10 ++++++++-- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c index 98c238ba3545..55154743f915 100644 --- a/mm/kasan/report_tags.c +++ b/mm/kasan/report_tags.c @@ -7,6 +7,7 @@ #include #include "kasan.h" +#include "../slab.h" extern struct kasan_stack_ring stack_ring; diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index b6c017e670d8..739ae997463d 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -97,12 +97,13 @@ static void save_stack_info(struct kmem_cache *cache, void *object, gfp_t gfp_flags, bool is_free) { unsigned long flags; - depot_stack_handle_t stack; + depot_stack_handle_t stack, old_stack; u64 pos; struct kasan_stack_ring_entry *entry; void *old_ptr; - stack = kasan_save_stack(gfp_flags, STACK_DEPOT_FLAG_CAN_ALLOC); + stack = kasan_save_stack(gfp_flags, + STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_GET); /* * Prevent save_stack_info() from modifying stack ring @@ -121,6 +122,8 @@ static void save_stack_info(struct kmem_cache *cache, void *object, if (!try_cmpxchg(&entry->ptr, &old_ptr, STACK_RING_BUSY_PTR)) goto next; /* Busy slot. */ + old_stack = entry->stack; + entry->size = cache->object_size; entry->pid = current->pid; entry->stack = stack; @@ -129,6 +132,9 @@ static void save_stack_info(struct kmem_cache *cache, void *object, entry->ptr = object; read_unlock_irqrestore(&stack_ring.lock, flags); + + if (old_stack) + stack_depot_put(old_stack); } void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)