perf data: Increase RLIMIT_NOFILE limit when open too many files in perf_data__create_dir()
Message ID | 20231013075945.698874-1-yangjihong1@huawei.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp1730035vqb; Fri, 13 Oct 2023 01:01:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHaJ6Amcr5zbYWfitUB0mFyPdxwOA8CzmxmqZUCO5zNEbVe1j+W1ZVPzHKvwlbk/q0uP6ls X-Received: by 2002:a05:6a20:1595:b0:163:ab09:195d with SMTP id h21-20020a056a20159500b00163ab09195dmr29937054pzj.0.1697184104820; Fri, 13 Oct 2023 01:01:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697184104; cv=none; d=google.com; s=arc-20160816; b=u21Qfol0rvidsv/D0B2X9cSTrS5LUIY7kAdC4OAIjri3MP+GQ+z+T4u6Sxj+h0+9fa RWgqHy8DUK5tQsPQ607lBZjTiYxeu33l7HwdZW370TStiCCucLM02OKTNilmdihaPEIi BuhKmd4iKxeWfOw9H+u7yDCAQswAePM70eLs/5Ig4JSLOe3m7JDjJozfDVIH3bC66z5Z 0bLyE2e7OuokaxxB4UB84mbUnMN5Ns0J2d+l8J6slB7Chs84RuMW6fyjU9P1mFFOr+BZ 25qnIJ7CcRqYm+MyvSHVP14MtRCYR7FmKY9IlV5MxoJA+FDAyMOvyGqnQ1FdPTkt26sR URxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=kkFMdBzvq7puihhd/UCW9F+8y7Sm+2Iqt5F+u17LN2s=; fh=JXx1mv1fB650HoTkc5GIgpDQ5Ey550GcDyaNZ/s6bYc=; b=K2fAKegvJ5713ZdFYuLSDO/9Eh9haRfuvCaWf2VVtVjEpllPwdPHr2ifRZp7hFwysi 9w9tS6xDI+yKnFTtHXFBWGQGokiD5u9TJWrp6nV3lK5xhJVrWo/s1jNjEt7PR1Gvo9km LrkLkUoGSjthrpSfkvNhxjltcc1SLeo4VH0JN0njfTDc6H0ClhRlcraSjoWiLX/MDMSx 4uKVMb/8CCmzdUeGjtq4GC7o6DBpjW8aH5BseJLRQkmtQjEcesXKRLrgbqaIm9ACH4Ub TlT1M3C3Gk9VGuSvag8UwxppPNLqjD0Wu1VjpgsTV+/YmFyPVqoYM6IIZ9Mhfj2ahW2P ay9A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id a19-20020a170902b59300b001c747cafce9si3686775pls.231.2023.10.13.01.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Oct 2023 01:01:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=huawei.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 2C11781B454C; Fri, 13 Oct 2023 01:01:42 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229992AbjJMIBe (ORCPT <rfc822;rua109.linux@gmail.com> + 19 others); Fri, 13 Oct 2023 04:01:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229863AbjJMIBa (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Fri, 13 Oct 2023 04:01:30 -0400 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D959C2; Fri, 13 Oct 2023 01:01:28 -0700 (PDT) Received: from kwepemm000003.china.huawei.com (unknown [172.30.72.56]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4S6JlP2TdXzVlWf; Fri, 13 Oct 2023 15:57:53 +0800 (CST) Received: from ubuntu2204.huawei.com (10.67.174.22) by kwepemm000003.china.huawei.com (7.193.23.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.31; Fri, 13 Oct 2023 16:01:25 +0800 From: Yang Jihong <yangjihong1@huawei.com> To: <peterz@infradead.org>, <mingo@redhat.com>, <acme@kernel.org>, <mark.rutland@arm.com>, <alexander.shishkin@linux.intel.com>, <jolsa@kernel.org>, <namhyung@kernel.org>, <irogers@google.com>, <adrian.hunter@intel.com>, <linux-perf-users@vger.kernel.org>, <linux-kernel@vger.kernel.org> CC: <yangjihong1@huawei.com> Subject: [PATCH] perf data: Increase RLIMIT_NOFILE limit when open too many files in perf_data__create_dir() Date: Fri, 13 Oct 2023 07:59:45 +0000 Message-ID: <20231013075945.698874-1-yangjihong1@huawei.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 7BIT Content-Type: text/plain; charset=US-ASCII X-Originating-IP: [10.67.174.22] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To kwepemm000003.china.huawei.com (7.193.23.66) X-CFilter-Loop: Reflected X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 13 Oct 2023 01:01:42 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1779626519836281339 X-GMAIL-MSGID: 1779626519836281339 |
Series |
perf data: Increase RLIMIT_NOFILE limit when open too many files in perf_data__create_dir()
|
|
Commit Message
Yang Jihong
Oct. 13, 2023, 7:59 a.m. UTC
If using parallel threads to collect data, perf record needs at least 6 fds
per CPU. (one for sys_perf_event_open, four for pipe msg and ack of the
pipe, see record__thread_data_open_pipes(), and one for open perf.data.XXX)
For an environment with more than 100 cores, if perf record uses both
`-a` and `--threads` options, it is easy to exceed the upper limit of the
file descriptor number, when we run out of them try to increase the limits.
Before:
$ ulimit -n
1024
$ lscpu | grep 'On-line CPU(s)'
On-line CPU(s) list: 0-159
$ perf record --threads -a sleep 1
Failed to create data directory: Too many open files
After:
$ ulimit -n
1024
$ lscpu | grep 'On-line CPU(s)'
On-line CPU(s) list: 0-159
$ perf record --threads -a sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.394 MB perf.data (1576 samples) ]
Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
---
tools/perf/util/data.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
Comments
Hello, On Fri, Oct 13, 2023 at 1:01 AM Yang Jihong <yangjihong1@huawei.com> wrote: > > If using parallel threads to collect data, perf record needs at least 6 fds > per CPU. (one for sys_perf_event_open, four for pipe msg and ack of the > pipe, see record__thread_data_open_pipes(), and one for open perf.data.XXX) Yep, probably one more for the dummy event. > For an environment with more than 100 cores, if perf record uses both > `-a` and `--threads` options, it is easy to exceed the upper limit of the > file descriptor number, when we run out of them try to increase the limits. > > Before: > $ ulimit -n > 1024 > $ lscpu | grep 'On-line CPU(s)' > On-line CPU(s) list: 0-159 > $ perf record --threads -a sleep 1 > Failed to create data directory: Too many open files > > After: > $ ulimit -n > 1024 > $ lscpu | grep 'On-line CPU(s)' > On-line CPU(s) list: 0-159 > $ perf record --threads -a sleep 1 > [ perf record: Woken up 1 times to write data ] > [ perf record: Captured and wrote 0.394 MB perf.data (1576 samples) ] > Signed-off-by: Yang Jihong <yangjihong1@huawei.com> > --- > tools/perf/util/data.c | 12 ++++++++++++ > 1 file changed, 12 insertions(+) > > diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c > index fc16299c915f..098f9e3bb2e7 100644 > --- a/tools/perf/util/data.c > +++ b/tools/perf/util/data.c > @@ -17,6 +17,7 @@ > #include "util.h" // rm_rf_perf_data() > #include "debug.h" > #include "header.h" > +#include "evsel.h" > #include <internal/lib.h> > > static void close_dir(struct perf_data_file *files, int nr) > @@ -35,6 +36,7 @@ void perf_data__close_dir(struct perf_data *data) > > int perf_data__create_dir(struct perf_data *data, int nr) > { > + enum rlimit_action set_rlimit = NO_CHANGE; > struct perf_data_file *files = NULL; > int i, ret; > > @@ -54,11 +56,21 @@ int perf_data__create_dir(struct perf_data *data, int nr) > goto out_err; > } > > +retry_open: > ret = open(file->path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR); > if (ret < 0) { > + /* > + * If using parallel threads to collect data, > + * perf record needs at least 6 fds per CPU. > + * When we run out of them try to increase the limits. > + */ > + if (errno == EMFILE && evsel__increase_rlimit(&set_rlimit)) It seems weird that we have this helper with evsel prefix and it does nothing with evsel. But it's a separate concern, so Acked-by: Namhyung Kim <namhyung@kernel.org> Thanks, Namhyung > + goto retry_open; > + > ret = -errno; > goto out_err; > } > + set_rlimit = NO_CHANGE; > > file->fd = ret; > } > -- > 2.34.1 >
On Fri, 13 Oct 2023 07:59:45 +0000, Yang Jihong wrote: > If using parallel threads to collect data, perf record needs at least 6 fds > per CPU. (one for sys_perf_event_open, four for pipe msg and ack of the > pipe, see record__thread_data_open_pipes(), and one for open perf.data.XXX) > For an environment with more than 100 cores, if perf record uses both > `-a` and `--threads` options, it is easy to exceed the upper limit of the > file descriptor number, when we run out of them try to increase the limits. > > [...] Applied to perf-tools-next, thanks!
Hello, On 2023/10/20 13:59, Namhyung Kim wrote: > Hello, > > On Fri, Oct 13, 2023 at 1:01 AM Yang Jihong <yangjihong1@huawei.com> wrote: >> >> If using parallel threads to collect data, perf record needs at least 6 fds >> per CPU. (one for sys_perf_event_open, four for pipe msg and ack of the >> pipe, see record__thread_data_open_pipes(), and one for open perf.data.XXX) > > Yep, probably one more for the dummy event. > >> For an environment with more than 100 cores, if perf record uses both >> `-a` and `--threads` options, it is easy to exceed the upper limit of the >> file descriptor number, when we run out of them try to increase the limits. >> >> Before: >> $ ulimit -n >> 1024 >> $ lscpu | grep 'On-line CPU(s)' >> On-line CPU(s) list: 0-159 >> $ perf record --threads -a sleep 1 >> Failed to create data directory: Too many open files >> >> After: >> $ ulimit -n >> 1024 >> $ lscpu | grep 'On-line CPU(s)' >> On-line CPU(s) list: 0-159 >> $ perf record --threads -a sleep 1 >> [ perf record: Woken up 1 times to write data ] >> [ perf record: Captured and wrote 0.394 MB perf.data (1576 samples) ] >> Signed-off-by: Yang Jihong <yangjihong1@huawei.com> >> --- >> tools/perf/util/data.c | 12 ++++++++++++ >> 1 file changed, 12 insertions(+) >> >> diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c >> index fc16299c915f..098f9e3bb2e7 100644 >> --- a/tools/perf/util/data.c >> +++ b/tools/perf/util/data.c >> @@ -17,6 +17,7 @@ >> #include "util.h" // rm_rf_perf_data() >> #include "debug.h" >> #include "header.h" >> +#include "evsel.h" >> #include <internal/lib.h> >> >> static void close_dir(struct perf_data_file *files, int nr) >> @@ -35,6 +36,7 @@ void perf_data__close_dir(struct perf_data *data) >> >> int perf_data__create_dir(struct perf_data *data, int nr) >> { >> + enum rlimit_action set_rlimit = NO_CHANGE; >> struct perf_data_file *files = NULL; >> int i, ret; >> >> @@ -54,11 +56,21 @@ int perf_data__create_dir(struct perf_data *data, int nr) >> goto out_err; >> } >> >> +retry_open: >> ret = open(file->path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR); >> if (ret < 0) { >> + /* >> + * If using parallel threads to collect data, >> + * perf record needs at least 6 fds per CPU. >> + * When we run out of them try to increase the limits. >> + */ >> + if (errno == EMFILE && evsel__increase_rlimit(&set_rlimit)) > > It seems weird that we have this helper with evsel prefix and > it does nothing with evsel. But it's a separate concern, so > > Acked-by: Namhyung Kim <namhyung@kernel.org> Thanks for the Acked-by tag. Uh... The name of this helper doesn't seem to be appropriate. Okay, I'll submit a patch to fix this helper name. Thanks, Yang
diff --git a/tools/perf/util/data.c b/tools/perf/util/data.c index fc16299c915f..098f9e3bb2e7 100644 --- a/tools/perf/util/data.c +++ b/tools/perf/util/data.c @@ -17,6 +17,7 @@ #include "util.h" // rm_rf_perf_data() #include "debug.h" #include "header.h" +#include "evsel.h" #include <internal/lib.h> static void close_dir(struct perf_data_file *files, int nr) @@ -35,6 +36,7 @@ void perf_data__close_dir(struct perf_data *data) int perf_data__create_dir(struct perf_data *data, int nr) { + enum rlimit_action set_rlimit = NO_CHANGE; struct perf_data_file *files = NULL; int i, ret; @@ -54,11 +56,21 @@ int perf_data__create_dir(struct perf_data *data, int nr) goto out_err; } +retry_open: ret = open(file->path, O_RDWR|O_CREAT|O_TRUNC, S_IRUSR|S_IWUSR); if (ret < 0) { + /* + * If using parallel threads to collect data, + * perf record needs at least 6 fds per CPU. + * When we run out of them try to increase the limits. + */ + if (errno == EMFILE && evsel__increase_rlimit(&set_rlimit)) + goto retry_open; + ret = -errno; goto out_err; } + set_rlimit = NO_CHANGE; file->fd = ret; }