From patchwork Wed Nov 29 09:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 171232 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a5a7:0:b0:403:3b70:6f57 with SMTP id d7csp246117vqn; Wed, 29 Nov 2023 02:34:52 -0800 (PST) X-Google-Smtp-Source: AGHT+IHH73GCmCX2EYnkcMF6PNfMWAVvQHq6a7nXU5iDSF4tsjIGzXdtdj5ymYT0f/fbL1MwGVYZ X-Received: by 2002:a92:d5cf:0:b0:35c:d27c:431b with SMTP id d15-20020a92d5cf000000b0035cd27c431bmr10069595ilq.11.1701254091963; Wed, 29 Nov 2023 02:34:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701254091; cv=none; d=google.com; s=arc-20160816; b=wf1WD6/z1yUvQtgMnuVS5N1VHpDRQTZlk6idzCWCkqjVzop37XLQ208yTqKANT6Di8 hdrCoMG0xZ6XCAS+bbRXo+ux/tbvMzR0CwfmHm/2UMxuCKgLg0zYg/xicaahS4DAFsb5 DIlERgGyQOr/ZSV2W5YHrRdxolbtOo6w6aBx5QN+fFZMRT48dXJ8Ss9OhpPp+kUba3X+ eJb3sqptzQR3Ein4H0dVLWU4I7KtKQ1HbxZK75RtPo72B9RrMiHaS31D94t738zpylOs m/ybccK60dBHlv8qMR3RfnEKvu6D+9GgaVh/jauXpOdToSZvo9r0vhnuXXYoMVqK+hN+ SXlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:in-reply-to:references:message-id :content-transfer-encoding:mime-version:subject:date:from :dkim-signature:dkim-signature; bh=OzJx1XLcKpfOVPPxGeC45HXa1Ww2Cw/dmnDirEUeMrA=; fh=BM+YXXPDItJMOh98QtZoCEH1RXzAZsCss4yDzENT5Z4=; b=mg1BilieOGYSRrZtpoIWUJLUfiCV+/VAjw7liwU4oqT6nF53e0rkobT6I+agpqh+0i jiWQTPsz1dk4ubwNFqGdL4rikdI8lnndg90/tGd0AEc3oZVMNyww4S+FKMZ7tyz6Q3+w QkeJ9BfeuwRS2BosL5Q1PtsJBCK0oeCA6jDM//Li9nsNy+RC/KaGCfoyUJC4FHXrDhqV +6Ngqr1u1hK05TVfK5XESneuDxCNHbHM4cvxmtIqNQ7t1aaBSNWoskGMsr1oU9k3F0yo JtYnkB/SQLnTVYh665eaEsGtRKH2hqSRF4Nl2kCQSsMcW9f2KZ4LxiR9pHsA2S7itH5s zrUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=inSOz4yT; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=Fx7hfg5V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id bw28-20020a056a02049c00b005b88dfa676esi14709160pgb.158.2023.11.29.02.34.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Nov 2023 02:34:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.cz header.s=susede2_rsa header.b=inSOz4yT; dkim=neutral (no key) header.i=@suse.cz header.s=susede2_ed25519 header.b=Fx7hfg5V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id CDD8B80B31C1; Wed, 29 Nov 2023 02:34:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232004AbjK2JyR (ORCPT + 99 others); Wed, 29 Nov 2023 04:54:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231246AbjK2Jxw (ORCPT ); Wed, 29 Nov 2023 04:53:52 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC0831BC1 for ; Wed, 29 Nov 2023 01:53:39 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id F1F931F8BE; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1701251618; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OzJx1XLcKpfOVPPxGeC45HXa1Ww2Cw/dmnDirEUeMrA=; b=inSOz4yT8TFgPaEVNqR0q6mY6UH6gJHUbkvbIB60MQaG+WfZTTe8fyxGmqgFcEPzn+zbkH X4wFuffCvNZPe31CV8gw6BNVz2L7dpmQFwcoVuq+TvUiUMdeAYgBVI4N/mYaqnMfUmr0nc /CknMU6FEMRIquT7ifcgQ/wnwv5HELU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1701251618; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OzJx1XLcKpfOVPPxGeC45HXa1Ww2Cw/dmnDirEUeMrA=; b=Fx7hfg5Vv0T5+dLFQWNf1/uBwvIhSApKaJpzlzazw7YdiXTEwqcjlUJKSONFGFXMIuuIIj UDa9Bi0THVElqdBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id AC77013AA1; Wed, 29 Nov 2023 09:53:37 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id aFDSKSEKZ2UrfQAAD6G6ig (envelope-from ); Wed, 29 Nov 2023 09:53:37 +0000 From: Vlastimil Babka Date: Wed, 29 Nov 2023 10:53:34 +0100 Subject: [PATCH RFC v3 9/9] maple_tree: replace preallocation with slub percpu array prefill MIME-Version: 1.0 Message-Id: <20231129-slub-percpu-caches-v3-9-6bcf536772bc@suse.cz> References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> To: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" Cc: Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Marco Elver , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com X-Mailer: b4 0.12.4 Authentication-Results: smtp-out2.suse.de; none X-Spam-Level: X-Spamd-Result: default: False [0.20 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_ENVRCPT(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; BAYES_HAM(-0.00)[29.13%]; RCVD_COUNT_THREE(0.00)[3]; R_RATELIMIT(0.00)[to_ip_from(RLtz7ce9b89hw8xzamye9qeynd)]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; NEURAL_HAM_SHORT(-0.20)[-1.000]; RCPT_COUNT_TWELVE(0.00)[16]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[linux-foundation.org,linux.dev,gmail.com,google.com,kvack.org,vger.kernel.org,lists.infradead.org,googlegroups.com]; RCVD_TLS_ALL(0.00)[]; SUSPICIOUS_RECIPS(1.50)[] X-Spam-Score: 0.20 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 29 Nov 2023 02:34:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783894210890529132 X-GMAIL-MSGID: 1783894210890529132 With the percpu array we can try not doing the preallocations in maple tree, and instead make sure the percpu array is prefilled, and using GFP_ATOMIC in places that relied on the preallocation (in case we miss or fail trylock on the array), i.e. mas_store_prealloc(). For now simply add __GFP_NOFAIL there as well. --- lib/maple_tree.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index f5c0bca2c5d7..d84a0c0fe83b 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5452,7 +5452,12 @@ void mas_store_prealloc(struct ma_state *mas, void *entry) mas_wr_store_setup(&wr_mas); trace_ma_write(__func__, mas, 0, entry); + +retry: mas_wr_store_entry(&wr_mas); + if (unlikely(mas_nomem(mas, GFP_ATOMIC | __GFP_NOFAIL))) + goto retry; + MAS_WR_BUG_ON(&wr_mas, mas_is_err(mas)); mas_destroy(mas); } @@ -5471,8 +5476,6 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) MA_WR_STATE(wr_mas, mas, entry); unsigned char node_size; int request = 1; - int ret; - if (unlikely(!mas->index && mas->last == ULONG_MAX)) goto ask_now; @@ -5512,16 +5515,8 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp) /* node store, slot store needs one node */ ask_now: - mas_node_count_gfp(mas, request, gfp); - if (likely(!mas_is_err(mas))) - return 0; + return kmem_cache_prefill_percpu_array(maple_node_cache, request, gfp); - mas_set_alloc_req(mas, 0); - ret = xa_err(mas->node); - mas_reset(mas); - mas_destroy(mas); - mas_reset(mas); - return ret; } EXPORT_SYMBOL_GPL(mas_preallocate);