Message ID | alpine.LSU.2.20.2311071651280.15233@wotan.suse.de |
---|---|
State | Accepted |
Headers |
Return-Path: <binutils-bounces+ouuuleilei=gmail.com@sourceware.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:aa0b:0:b0:403:3b70:6f57 with SMTP id k11csp363178vqo; Tue, 7 Nov 2023 08:51:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IFnjR4e+xdP4RQZPKvr8Umn8fVwHSiCXZKmd+eXj8jn4HgsMtu2HFw+pgYbUUMhcm7DoHwR X-Received: by 2002:a05:620a:1726:b0:778:8d72:91ea with SMTP id az38-20020a05620a172600b007788d7291eamr13390132qkb.25.1699375906357; Tue, 07 Nov 2023 08:51:46 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1699375906; cv=pass; d=google.com; s=arc-20160816; b=cTRgKmwchq4jI7leO5nqoUsUXVNEOYpLTaGsrMIhssHMzWNPaMulbjWwNULPZ+Nld9 WwJKLPLfaAKM6i5mYUbAo4lWS+cjiTGFa27ctEvbc7AgRoimSUdl3IdXieF755Fuctq2 Z8ISCLoMJzRKUw0+NSjMRiOLA4LIJ+oGtsyvUOh+QBh22ItoPX2yjzyxJ6TmrWo+EMl/ 13WJI75J3SuqVl8TQZ9OpInS90w7zmrW7Rynbebr9ovmMpmgKhnnCRX9zt60Omb83w/X YCuiST0bHOe2qBqbW69r8As7CJt0e2ZlY+frc+kyrn9vWangkISmby2hbz4J8ssfJeFa 7hHQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mime-version:user-agent :message-id:subject:to:from:date:dkim-signature:dkim-signature :arc-filter:dmarc-filter:delivered-to; bh=sH6aNs8EM293wG5VfOz2Nd5HSRkfyjxJdebi/03rA8c=; fh=NLxAvL/bDfPg4AGOtxqvQlND8vazkZrNzKLY8+LAbBY=; b=mmoYrZZmx5xDw6h58XWWV6wkn9nxQcnCgZ6AhZmprCmpIzz05pu0jZ0MvunndZGlVK x1pR5Ty/7FT2qq1nmiQiJ67qn50WwSdLB3tBTqyN/tma9XeQqCysDBNix7ybUk0yXnxC 7KzlGgzjqwavtXjJWq8Qjt3YIptVxt8o6kL+oilso7f1WwIOValUY0FJGGkZupoeBTPq AXdrQOLGNnRSVVmyYrGO21yXtqOC/kXQwZyZIA0K0NzbtEgRtSjeRCIZ3X6Gknt5sQ8t gamFmrzMbKs6Q5T4Q3JVmTuqM7XXJQSz4tg+LVySgmI+kJYDpVIOdVj367oBEth6gfAm q95Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Kgr5V52f; dkim=neutral (no key) header.i=@suse.de; arc=pass (i=1); spf=pass (google.com: domain of binutils-bounces+ouuuleilei=gmail.com@sourceware.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="binutils-bounces+ouuuleilei=gmail.com@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (server2.sourceware.org. [8.43.85.97]) by mx.google.com with ESMTPS id u14-20020a05620a084e00b007789a6caf87si72010qku.557.2023.11.07.08.51.46 for <ouuuleilei@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Nov 2023 08:51:46 -0800 (PST) Received-SPF: pass (google.com: domain of binutils-bounces+ouuuleilei=gmail.com@sourceware.org designates 8.43.85.97 as permitted sender) client-ip=8.43.85.97; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Kgr5V52f; dkim=neutral (no key) header.i=@suse.de; arc=pass (i=1); spf=pass (google.com: domain of binutils-bounces+ouuuleilei=gmail.com@sourceware.org designates 8.43.85.97 as permitted sender) smtp.mailfrom="binutils-bounces+ouuuleilei=gmail.com@sourceware.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 222BF38582A3 for <ouuuleilei@gmail.com>; Tue, 7 Nov 2023 16:51:46 +0000 (GMT) X-Original-To: binutils@sourceware.org Delivered-To: binutils@sourceware.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by sourceware.org (Postfix) with ESMTPS id DE6233858002 for <binutils@sourceware.org>; Tue, 7 Nov 2023 16:51:40 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DE6233858002 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org DE6233858002 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2001:67c:2178:6::1d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699375902; cv=none; b=OCj8/7pJlqcl+nhjlH5hDKreiARw2p0C21LDkM4oUxoYr0kN0hLxKSr7ySBMdG2JO8g0TQ1est7Fc1ZvJkpVUw2TX8ydE9OO04sT5Qo7ClvLxZTZRlLOWf+eKlwvFGXdT6RHB7eYlAT4tZ0K/qUAW01ciOpQnXX+kEGFop2T9JM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1699375902; c=relaxed/simple; bh=XsMyMvEsBvzbfzeeSKFHx4ICPdXs53aKgfUHLGq6ap0=; h=DKIM-Signature:DKIM-Signature:Date:From:To:Subject:Message-ID: MIME-Version; b=NDYYvs3dEzQXzlojHhK5jAbaoM2Hc2udURP4gAD+9OazrA82cdDAV7ldcDhx84ENZXasFh1Q/gpFv2on5550QyR8kHIF4EBhb3MjR38mjJBpmZfapE+ZiYRMXCpoHgKqbPeHhAmjVQNL9apgKVMrb39KQPZt+nQlpuYb2DdUFq4= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 1F9EB1F37C for <binutils@sourceware.org>; Tue, 7 Nov 2023 16:51:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1699375900; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=sH6aNs8EM293wG5VfOz2Nd5HSRkfyjxJdebi/03rA8c=; b=Kgr5V52ftOsAoZZNJZnnEQ2j78VqpMN4bnP5EGgWQHgTs3CAtL3Mss4uTqfNr3+vwJmJ5q fBOYBzRgDCTw2h87HSnl7uNJsyKOkwZb/FYAdOnqp889AXQtMeYc6Rnvf2NY+ePE+OYv9O TxZFmovAj8rV595liA4iQ5vzin1otUU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1699375900; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type; bh=sH6aNs8EM293wG5VfOz2Nd5HSRkfyjxJdebi/03rA8c=; b=I4R5CrijDznwKdOg9qOqGyw6pRiBNlyuAN8XSWZ+cKMaGdaCwhLXWwhJFOc6sRdNcoL/pw YwUDg+lwSZBGmZDg== Received: from wotan.suse.de (wotan.suse.de [10.160.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id DE6522C16D for <binutils@sourceware.org>; Tue, 7 Nov 2023 16:51:39 +0000 (UTC) Received: by wotan.suse.de (Postfix, from userid 10510) id 1233166A4; Tue, 7 Nov 2023 16:51:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by wotan.suse.de (Postfix) with ESMTP id 10FAF6588 for <binutils@sourceware.org>; Tue, 7 Nov 2023 16:51:40 +0000 (UTC) Date: Tue, 7 Nov 2023 16:51:39 +0000 (UTC) From: Michael Matz <matz@suse.de> To: binutils@sourceware.org Subject: bfd: use less memory in string merging Message-ID: <alpine.LSU.2.20.2311071651280.15233@wotan.suse.de> User-Agent: Alpine 2.20 (LSU 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-9.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: binutils@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Binutils mailing list <binutils.sourceware.org> List-Unsubscribe: <https://sourceware.org/mailman/options/binutils>, <mailto:binutils-request@sourceware.org?subject=unsubscribe> List-Archive: <https://sourceware.org/pipermail/binutils/> List-Post: <mailto:binutils@sourceware.org> List-Help: <mailto:binutils-request@sourceware.org?subject=help> List-Subscribe: <https://sourceware.org/mailman/listinfo/binutils>, <mailto:binutils-request@sourceware.org?subject=subscribe> Errors-To: binutils-bounces+ouuuleilei=gmail.com@sourceware.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781924790676519780 X-GMAIL-MSGID: 1781924790676519780 |
Series |
bfd: use less memory in string merging
|
|
Checks
Context | Check | Description |
---|---|---|
snail/binutils-gdb-check | success | Github commit url |
Commit Message
Michael Matz
Nov. 7, 2023, 4:51 p.m. UTC
the offset-to-entry mappings are allocated in blocks, which may become a bit wasteful in case there are extremely many small input files or sections. This made it so that a large project (Qt5WebEngine) didn't build anymore on x86 32bit due to address space limits. It barely fit into address space before the new string merging, and then got pushed over the limit by this. So instead of leaving the waste reallocate the maps to their final size once known. Now the link barely fits again. bfd/ * merge.c (record_section): Reallocate offset maps to their final size. --- regtested on many targets, okay for master? bfd/merge.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
Comments
On 07.11.2023 17:51, Michael Matz wrote: > --- a/bfd/merge.c > +++ b/bfd/merge.c > @@ -767,6 +767,18 @@ record_section (struct sec_merge_info *sinfo, > > free (contents); > contents = NULL; > + > + /* We allocate the ofsmap arrays in blocks of 2048 elements. > + In case we have very many small input files/sections, > + this might waste large amounts of memory, so reallocate these > + arrays here to their true size. */ > + amt = secinfo->noffsetmap + 1; > + secinfo->map_ofs = bfd_realloc (secinfo->map_ofs, > + amt * sizeof(secinfo->map_ofs[0])); > + BFD_ASSERT (secinfo->map_ofs); > + secinfo->map = bfd_realloc (secinfo->map, amt * sizeof(secinfo->map[0])); > + BFD_ASSERT (secinfo->map); Re-use of the same block when shrinking isn't guaranteed, so depending on the underlying allocator this may actually add memory pressure (and the allocations may therefore also fail). I think it would be nice to be independent of such an implementation detail of the underlying library. (It may also be worthwhile then to shrink the larger of the two arrays first. Otoh the comment ahead of mapofs_type already indicates that this type may need widening at some point.) Jan
Hello, On Wed, 8 Nov 2023, Jan Beulich wrote: > > + /* We allocate the ofsmap arrays in blocks of 2048 elements. > > + In case we have very many small input files/sections, > > + this might waste large amounts of memory, so reallocate these > > + arrays here to their true size. */ > > + amt = secinfo->noffsetmap + 1; > > + secinfo->map_ofs = bfd_realloc (secinfo->map_ofs, > > + amt * sizeof(secinfo->map_ofs[0])); > > + BFD_ASSERT (secinfo->map_ofs); > > + secinfo->map = bfd_realloc (secinfo->map, amt * sizeof(secinfo->map[0])); > > + BFD_ASSERT (secinfo->map); > > Re-use of the same block when shrinking isn't guaranteed, so depending > on the underlying allocator this may actually add memory pressure (and > the allocations may therefore also fail). That's true, strictly speaking, but when this doesn't average out over thousands of blocks then it's such a low quality malloc(3) that it will have many other problems as well. Certainly with the cases that lead me to the above (linking running nearly out of 32bit address space). So I had that worry as well and rejected it. > I think it would be nice to be independent of such an implementation > detail of the underlying library. Yes. But do you mean it as pre-requisite for the patch? In that case I'll try something about a bucket allocator for the offsetmap blocks, though I think it's a bit on the extreme to work around lousy mallocs in current times. > (It may also be worthwhile then to shrink the larger of the > two arrays first. Otoh the comment ahead of mapofs_type already > indicates that this type may need widening at some point.) That is true nevertheless, so consider this changed in the patch. Ciao, Michael.
On 08.11.2023 14:39, Michael Matz wrote: > Hello, > > On Wed, 8 Nov 2023, Jan Beulich wrote: > >>> + /* We allocate the ofsmap arrays in blocks of 2048 elements. >>> + In case we have very many small input files/sections, >>> + this might waste large amounts of memory, so reallocate these >>> + arrays here to their true size. */ >>> + amt = secinfo->noffsetmap + 1; >>> + secinfo->map_ofs = bfd_realloc (secinfo->map_ofs, >>> + amt * sizeof(secinfo->map_ofs[0])); >>> + BFD_ASSERT (secinfo->map_ofs); >>> + secinfo->map = bfd_realloc (secinfo->map, amt * sizeof(secinfo->map[0])); >>> + BFD_ASSERT (secinfo->map); >> >> Re-use of the same block when shrinking isn't guaranteed, so depending >> on the underlying allocator this may actually add memory pressure (and >> the allocations may therefore also fail). > > That's true, strictly speaking, but when this doesn't average out over > thousands of blocks then it's such a low quality malloc(3) that it will > have many other problems as well. Certainly with the cases that lead me > to the above (linking running nearly out of 32bit address space). So I > had that worry as well and rejected it. > >> I think it would be nice to be independent of such an implementation >> detail of the underlying library. > > Yes. But do you mean it as pre-requisite for the patch? In that case > I'll try something about a bucket allocator for the offsetmap blocks, > though I think it's a bit on the extreme to work around lousy mallocs in > current times. I definitely wouldn't go as far as asking for such a rework. What I'd like to see though is that the realloc() return values be latched into a local, and instead of failing upon the function returning NULL the old pointers in the struct simply not be updated. Preferably with that adjustment okay to put in. Jan >> (It may also be worthwhile then to shrink the larger of the >> two arrays first. Otoh the comment ahead of mapofs_type already >> indicates that this type may need widening at some point.) > > That is true nevertheless, so consider this changed in the patch. > > > Ciao, > Michael.
Hey, On Thu, 9 Nov 2023, Jan Beulich wrote: > >> I think it would be nice to be independent of such an implementation > >> detail of the underlying library. > > > > Yes. But do you mean it as pre-requisite for the patch? In that case > > I'll try something about a bucket allocator for the offsetmap blocks, > > though I think it's a bit on the extreme to work around lousy mallocs in > > current times. > > I definitely wouldn't go as far as asking for such a rework. What I'd > like to see though is that the realloc() return values be latched into > a local, and instead of failing upon the function returning NULL the > old pointers in the struct simply not be updated. Preferably with that > adjustment okay to put in. Oh, that makes sense, yes. (The contract on the bfd_realloc wrapper is a bit unhelpful here, it invariably will set bfd_error_no_memory when returning NULL, but I still agree with you that it's nicer to not overwrite the existing pointer when realloc didn't work). Ciao, Michael. > > Jan > > >> (It may also be worthwhile then to shrink the larger of the > >> two arrays first. Otoh the comment ahead of mapofs_type already > >> indicates that this type may need widening at some point.) > > > > That is true nevertheless, so consider this changed in the patch. > > > > > > Ciao, > > Michael. >
diff --git a/bfd/merge.c b/bfd/merge.c index 4aa2f838679..ccefb707c47 100644 --- a/bfd/merge.c +++ b/bfd/merge.c @@ -767,6 +767,18 @@ record_section (struct sec_merge_info *sinfo, free (contents); contents = NULL; + + /* We allocate the ofsmap arrays in blocks of 2048 elements. + In case we have very many small input files/sections, + this might waste large amounts of memory, so reallocate these + arrays here to their true size. */ + amt = secinfo->noffsetmap + 1; + secinfo->map_ofs = bfd_realloc (secinfo->map_ofs, + amt * sizeof(secinfo->map_ofs[0])); + BFD_ASSERT (secinfo->map_ofs); + secinfo->map = bfd_realloc (secinfo->map, amt * sizeof(secinfo->map[0])); + BFD_ASSERT (secinfo->map); + /*printf ("ZZZ %s:%s %u entries\n", sec->owner->filename, sec->name, (unsigned)secinfo->noffsetmap);*/