Message ID | 20231207125716.1947860-1-tmricht@linux.ibm.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp4759284vqy; Thu, 7 Dec 2023 04:57:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IGKtbszzzQpJvn2H7ViCznlxPh7UHg5iVPh8/gaag2BoZDI70kYsK2AGcYycvnzZ8mLmvZM X-Received: by 2002:a05:6a21:3102:b0:18b:30e2:7e55 with SMTP id yz2-20020a056a21310200b0018b30e27e55mr4079591pzb.46.1701953868381; Thu, 07 Dec 2023 04:57:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701953868; cv=none; d=google.com; s=arc-20160816; b=CH8zluBS9pvrcpivnGXPN286GIGd56s6igfHQK95KRDNSzcrNsQwQuGhsJpy6UCiM1 S5QDJpKH5Mu8PBvhdbXHKg6NLemIXpJnvUKxqQbmGMb6VZ0THHiU3vpKRCfbfNmJHY5g YCdqec42yP8KSsmsr5ZphaWYqJ2sweKW7iBDhWnBhcLohFZwa/Eog8KFEDw4aVHYzYQn SbUj4CA6O/v2n+pOfT4LFokcwwavtdcIuooBTe2A9wPcbJkQJimJcdJ8xUZoNJAGWga6 Qs7PwJUIhthaN1M6d3Duyws3bh6ZO2FGshz4TgDKWgz7p14uz/I1nASSuqbd4ExBTZ3w ZjNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=HjIMDBErvcFf1wv9hqQUMOA9EbkqzxRH/JoQ6aYpg14=; fh=5a+Wx783XmQPihkKeBQBI7cD6zjOCSCFwDahEdzHe4w=; b=xU9C3dgaHbwDTDVVeFA8ltJ6XGjn5sSEHb1FvhbXy0RguJzYASlx8VhbjNDxy4+WnY NT2ToMyejQVWy4gGvQLg0cxTm0SxuLu5GsVatPNwnb2yNug+Av6tcGUyjAXxa2yIVSRJ rL0ImKzj7NSzSTOVfv2psSD/e3sgMiO3G+gR9/CBJ5N3Twr05nrLsJpbUFQz+R06qYSO 8K243S3Qs7MSti2maOhAYN1u7GwDna9A0JexZPIvfGAz0mIQTSvgzdzeD+pgn2Duz22Y zitKjPiY14dW3cCvaHihbyxyZeGVtHY5LvUTcpoiGpQGpLESZDL/J1p0XS4LP8Pyu0B1 iqrg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=bEh6pzQr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id b5-20020a63eb45000000b005b95ccd1b4dsi1157880pgk.82.2023.12.07.04.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 04:57:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@ibm.com header.s=pp1 header.b=bEh6pzQr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=NONE dis=NONE) header.from=ibm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id ADF468060349; Thu, 7 Dec 2023 04:57:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbjLGM51 (ORCPT <rfc822;pusanteemu@gmail.com> + 99 others); Thu, 7 Dec 2023 07:57:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47764 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232546AbjLGM5Z (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Thu, 7 Dec 2023 07:57:25 -0500 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 929B1D5E; Thu, 7 Dec 2023 04:57:30 -0800 (PST) Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7Cr1qs027097; Thu, 7 Dec 2023 12:57:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=HjIMDBErvcFf1wv9hqQUMOA9EbkqzxRH/JoQ6aYpg14=; b=bEh6pzQrIfTkOLO0V9/d1Qq0VEh+D4VuHsCnadzoxUbtv8OV+YHzw6fz5wuAxFAGjLH1 b3dwl7XWgnfO2HkuNwV+3ebx6LW3QA46FnS2Iy5xHrVCNvj6ytmyc1NSVgAv4aCTuZ6T j6FIGeh5Yum9vGwb8+MdPa2VDaRpvX/i4tjkhgWgOF84AdbiwJHjt83PBrAvlPU+5HFn EFDSONVwWfvV1yWoX7dE4y68GdhCF9pa+YcKI8dOPbanEDNxxnoXXDK4Bq4znXqw0ZBD xKICO5zw9PQ7Ou592wsa6Evm0yxrc+x6FqvDVzm3g28e0S7NBYiHNFgZM3xVGTM2lCqd uw== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3uueep84ck-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 07 Dec 2023 12:57:24 +0000 Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 3B7B5ied015401; Thu, 7 Dec 2023 12:57:24 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 3utavkk5yu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 07 Dec 2023 12:57:24 +0000 Received: from smtpav01.fra02v.mail.ibm.com (smtpav01.fra02v.mail.ibm.com [10.20.54.100]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 3B7CvLRG28574272 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 7 Dec 2023 12:57:21 GMT Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7614120043; Thu, 7 Dec 2023 12:57:21 +0000 (GMT) Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4DEA520040; Thu, 7 Dec 2023 12:57:21 +0000 (GMT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by smtpav01.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 7 Dec 2023 12:57:21 +0000 (GMT) From: Thomas Richter <tmricht@linux.ibm.com> To: linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, acme@kernel.org, namhyung@kernel.org Cc: svens@linux.ibm.com, gor@linux.ibm.com, sumanthk@linux.ibm.com, hca@linux.ibm.com, Thomas Richter <tmricht@linux.ibm.com> Subject: [PATCH] perf test: Fix fails of perf stat --bpf-counters --for-each-cgroup on s390 Date: Thu, 7 Dec 2023 13:57:16 +0100 Message-Id: <20231207125716.1947860-1-tmricht@linux.ibm.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: ct-Akwe7L1CJCRtQHzLZKpa85UhOFwas X-Proofpoint-ORIG-GUID: ct-Akwe7L1CJCRtQHzLZKpa85UhOFwas X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-07_10,2023-12-07_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 adultscore=0 mlxlogscore=999 mlxscore=0 bulkscore=0 impostorscore=0 phishscore=0 suspectscore=0 priorityscore=1501 clxscore=1011 malwarescore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312070106 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 07 Dec 2023 04:57:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784627979793553804 X-GMAIL-MSGID: 1784627979793553804 |
Series |
perf test: Fix fails of perf stat --bpf-counters --for-each-cgroup on s390
|
|
Commit Message
Thomas Richter
Dec. 7, 2023, 12:57 p.m. UTC
On s390 this test fails very often, as can be observed in the output below. This is caused by the second test function check_cpu_list_counted(). The perf stat is triggered for 2 CPUs 0 and 1. On s390, which usually has a lot more CPUs, most often this ends up in no counter increments on these 2 CPUs 0 and 1. Fix this and trigger explicit workload on CPU 0 and 1 for systemd. This is a better approach than calculating a long list of CPUs (which is basicly the same as option -a), or wait a longer period of time. Output before: # for i in $(seq 10) > do ./perf test 100 > done 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! 100: perf stat --bpf-counters --for-each-cgroup test : Ok # Output after: # for i in $(seq 10); do ./perf test 100; done 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok 100: perf stat --bpf-counters --for-each-cgroup test : Ok # Signed-off-by: Thomas Richter <tmricht@linux.ibm.com> --- tools/perf/tests/shell/stat_bpf_counters_cgrp.sh | 1 + 1 file changed, 1 insertion(+)
Comments
Hello, On Thu, Dec 7, 2023 at 4:57 AM Thomas Richter <tmricht@linux.ibm.com> wrote: > > On s390 this test fails very often, as can be observed in the output > below. This is caused by the second test function > check_cpu_list_counted(). The perf stat is triggered for 2 CPUs > 0 and 1. On s390, which usually has a lot more CPUs, most often > this ends up in no counter increments on these 2 CPUs 0 and 1. > > Fix this and trigger explicit workload on CPU 0 and 1 for > systemd. This is a better approach than calculating a long > list of CPUs (which is basicly the same as option -a), or > wait a longer period of time. > > Output before: > # for i in $(seq 10) > > do ./perf test 100 > > done > 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : FAILED! > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > # > > Output after: > # for i in $(seq 10); > do ./perf test 100; > done > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > 100: perf stat --bpf-counters --for-each-cgroup test : Ok > # > > Signed-off-by: Thomas Richter <tmricht@linux.ibm.com> > --- > tools/perf/tests/shell/stat_bpf_counters_cgrp.sh | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh b/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh > index e75d0780dc78..f67602321403 100755 > --- a/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh > +++ b/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh > @@ -60,6 +60,7 @@ check_system_wide_counted() > > check_cpu_list_counted() > { > + taskset -c 0,1 systemctl daemon-reexec & Thanks for the patch. But I think it should support machines without systemd (or maybe with old versions). Also probably you want to reset the behavior after the test. I think we can just run some built-in test workload like `perf test -w thloop`. Thanks, Namhyung > check_cpu_list_counted_output=$(perf stat -C 0,1 --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, taskset -c 1 sleep 1 2>&1) > if echo ${check_cpu_list_counted_output} | grep -q -F "<not "; then > echo "Some CPU events are not counted" > -- > 2.43.0 >
On 12/8/23 00:26, Namhyung Kim wrote: > Thanks for the patch. But I think it should support > machines without systemd (or maybe with old versions). > > Also probably you want to reset the behavior after > the test. I think we can just run some built-in test > workload like `perf test -w thloop`. > > Thanks, > Namhyung Thanks for our feedback. Well regarding the use of systemd daemon-reexec the manual says this command restarts the systemd triggered processes. There is nothing to reset. All ports stay active while the command is processed. I tried your 'perf test -w thloop`, but that did not trigger anything on system.slice. I do not understand enough about cgroups and system.slice, but I am under the impression, that the system.slice just increment counters when executed by processes under systemd control. Maybe I am wrong. The only other workload which always incremented system.slice counters was 'ssh localhost ls -l', which involves local login and a running sshd. Thanks for your advice on how to continue on this.
On 12/8/23 12:07, Thomas Richter wrote: > On 12/8/23 00:26, Namhyung Kim wrote: > >> Thanks for the patch. But I think it should support >> machines without systemd (or maybe with old versions). >> >> Also probably you want to reset the behavior after >> the test. I think we can just run some built-in test >> workload like `perf test -w thloop`. >> >> Thanks, >> Namhyung > > Thanks for our feedback. > Well regarding the use of systemd daemon-reexec the manual says > this command restarts the systemd triggered processes. > There is nothing to reset. All ports stay active while the command > is processed. > > I tried your 'perf test -w thloop`, but that did not trigger > anything on system.slice. > > I do not understand enough about cgroups and system.slice, but I am > under the impression, that the system.slice just increment counters > when executed by processes under systemd control. Maybe I am wrong. > > The only other workload which always incremented system.slice counters > was 'ssh localhost ls -l', which involves local login and a running sshd. > > Thanks for your advice on how to continue on this. > > I have done some reading and found this: Special Slice Units There are four ".slice" units which form the basis of the hierarchy for assignment of resources for services, users, and virtual machines or containers. See systemd.slice(7) for details about slice units. -.slice The root slice is the root of the slice hierarchy. It usually does not contain units directly, but may be used to set defaults for the whole tree. Added in version 206. system.slice By default, all system services started by systemd are found in this slice. Added in version 206. So it looks like system.slice attached counters get only incremented when systemd controlled processes do some work,
On Fri, Dec 8, 2023 at 3:30 AM Thomas Richter <tmricht@linux.ibm.com> wrote: > > On 12/8/23 12:07, Thomas Richter wrote: > > On 12/8/23 00:26, Namhyung Kim wrote: > > > >> Thanks for the patch. But I think it should support > >> machines without systemd (or maybe with old versions). > >> > >> Also probably you want to reset the behavior after > >> the test. I think we can just run some built-in test > >> workload like `perf test -w thloop`. > >> > >> Thanks, > >> Namhyung > > > > Thanks for our feedback. > > Well regarding the use of systemd daemon-reexec the manual says > > this command restarts the systemd triggered processes. > > There is nothing to reset. All ports stay active while the command > > is processed. > > > > I tried your 'perf test -w thloop`, but that did not trigger > > anything on system.slice. > > > > I do not understand enough about cgroups and system.slice, but I am > > under the impression, that the system.slice just increment counters > > when executed by processes under systemd control. Maybe I am wrong. Ah, you're right. It needs to run the task somewhere in the system.slice. Then it'd be hard to get a proper cgroup name generally. Hmm.. My concern was it'd bind system daemons on the CPU 0 and 1 after the test. Probably you could run it at the end of the test again without taskset. > > > > The only other workload which always incremented system.slice counters > > was 'ssh localhost ls -l', which involves local login and a running sshd. But it won't work if the system doesn't have sshd. Thanks, Namhyung
diff --git a/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh b/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh index e75d0780dc78..f67602321403 100755 --- a/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh +++ b/tools/perf/tests/shell/stat_bpf_counters_cgrp.sh @@ -60,6 +60,7 @@ check_system_wide_counted() check_cpu_list_counted() { + taskset -c 0,1 systemctl daemon-reexec & check_cpu_list_counted_output=$(perf stat -C 0,1 --bpf-counters --for-each-cgroup ${test_cgroups} -e cpu-clock -x, taskset -c 1 sleep 1 2>&1) if echo ${check_cpu_list_counted_output} | grep -q -F "<not "; then echo "Some CPU events are not counted"