From patchwork Wed Jan 18 11:37:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: tip-bot2 for Thomas Gleixner X-Patchwork-Id: 45204 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2304903wrn; Wed, 18 Jan 2023 04:20:38 -0800 (PST) X-Google-Smtp-Source: AMrXdXuazhv4kUUAch6F8KPQctxxyNUG6JM+JXft4ZayL/lz1Qr7uFRUUf6RLf+M866qV635P9J6 X-Received: by 2002:a17:906:a04d:b0:84d:4325:7f7a with SMTP id bg13-20020a170906a04d00b0084d43257f7amr2598884ejb.65.1674044438322; Wed, 18 Jan 2023 04:20:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674044438; cv=none; d=google.com; s=arc-20160816; b=fjXizcJ/9aM0Zvb9ColdRuch7G96CYz8e1LwdprNG45aM+yMQISFFbUpIZomuhkmgH MncvXxlAVryZrWI6y7mTWE34f97/G4KYwC63yvb4u3eP5G9l4Coid/YkHegHOXVgLcI2 wvtYmXlDrmVdadinZO+52LPrxl+xb1Ig/4kvzVl9NazxyAVY4A1GPpzpB4Cs9NWw7cEb 1JwnKKl3TbhJKAu8E5nyUq0C0PGQ6Kuvr6efplEVjL6SGQ5EQIJFtEt1+lEM8JzhyO2/ FnPyhr6X7BUfetlzL7w8EvsCoKh5wsAYUaYQ6aWhkxNlXraZIUr+R2q4n4gwjuHozWHm X5tA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:robot-unsubscribe :robot-id:message-id:mime-version:references:in-reply-to:cc:subject :to:reply-to:sender:from:dkim-signature:dkim-signature:date; bh=5XOWWhVeVAOMndfqq8i24CNy11DrE3XLqdZTXMAnqN8=; b=M/j6CAw0ka2k1plSkt8ahcKTdi5lCX+Qavua/f4LBMthJTkAzXS4dYlHqkpZuDTFGr 4BkL0+++ISxtKX4HpY/xVwqLMUhPiJJhhUSkxUmJgf/xni6fnel4JD8jcZBPautGsJVv IcyyhGE2FTVgIdkDS7+D4Jbi/6MmuUIxqTB589gIDnwIL5bBU1o8UKvjZc7GlsMyw5V3 lEB9qjH9olAbBgVMKWmdJNfY93vjv+ThTjrKzGzQLdpYkpw7GCAvruXcG7M4nvvqcC/X v8R2GBxv+OEUazFG07imhhFfZoyppDyZ3sI3blZ45oSFd/Jy84kBpdRELf1LUhmqPVf8 WdXA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=WqHZvzjh; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id di6-20020a170906730600b00857c01c6b3fsi28937776ejc.164.2023.01.18.04.20.14; Wed, 18 Jan 2023 04:20:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=WqHZvzjh; dkim=neutral (no key) header.i=@linutronix.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230363AbjARMQo (ORCPT + 99 others); Wed, 18 Jan 2023 07:16:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230321AbjARMQG (ORCPT ); Wed, 18 Jan 2023 07:16:06 -0500 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD924875BF; Wed, 18 Jan 2023 03:37:53 -0800 (PST) Date: Wed, 18 Jan 2023 11:37:51 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1674041872; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5XOWWhVeVAOMndfqq8i24CNy11DrE3XLqdZTXMAnqN8=; b=WqHZvzjhTdWZsNYNrIKU83e66QbSNwqjgZrHZo75ibugKQrq7y81ZE7kwut/WV2jqfcUtT X3aAqnqMbVHBJw2q9PWbhGd66Naqsx5f9JXIuZBc7l363l9jVPk5fq6hhDmTe7eZWPWDVg nCAGhcnHVWIKMoV8V9eEpLKfUYaupeMT/IjPVcAx5EdPJBlgVZS14GnYWKKvFzS8ZWUx7X FJJds45OnsfSpJ3cJc18lxwKh3IA6qcr7hgq8/jfS7lz/Q1vHj5jvkF7xufFw53VgI2mCU 1wYO0APOcT9iW7oDThhI0s91h7S3A/gx1bFAY5Z9DHeWsWhF2b7zcHYGMIt2EA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1674041872; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5XOWWhVeVAOMndfqq8i24CNy11DrE3XLqdZTXMAnqN8=; b=0EBcDAg+zhZ2uG2J6kQ38Pa8bkOR6LdNOOK2DrH7pFZowsKrUDK2TB/OFVmX670VAEMc/P /PUTtPISt+ayIoCg== From: "tip-bot2 for Namhyung Kim" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: perf/core] perf/core: Set data->sample_flags in perf_prepare_sample() Cc: Peter Zijlstra , Namhyung Kim , Ingo Molnar , Jiri Olsa , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20230118060559.615653-6-namhyung@kernel.org> References: <20230118060559.615653-6-namhyung@kernel.org> MIME-Version: 1.0 Message-ID: <167404187184.4906.18122536452263093575.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755340431153700099?= X-GMAIL-MSGID: =?utf-8?q?1755362821122661930?= The following commit has been merged into the perf/core branch of tip: Commit-ID: bb447c27a4674628ea50341cfa4b31618f2010af Gitweb: https://git.kernel.org/tip/bb447c27a4674628ea50341cfa4b31618f2010af Author: Namhyung Kim AuthorDate: Tue, 17 Jan 2023 22:05:56 -08:00 Committer: Ingo Molnar CommitterDate: Wed, 18 Jan 2023 11:57:20 +01:00 perf/core: Set data->sample_flags in perf_prepare_sample() The perf_prepare_sample() function sets the perf_sample_data according to the attr->sample_type before copying it to the ring buffer. But BPF also wants to access the sample data so it needs to prepare the sample even before the regular path. That means perf_prepare_sample() can be called more than once. Set the data->sample_flags consistently so that it can indicate which fields are set already and skip them if sets. Also update the filtered_sample_type to have the dependent flags to reduce the number of branches. Suggested-by: Peter Zijlstra Signed-off-by: Namhyung Kim Signed-off-by: Ingo Molnar Tested-by: Jiri Olsa Acked-by: Jiri Olsa Acked-by: Peter Zijlstra Link: https://lore.kernel.org/r/20230118060559.615653-6-namhyung@kernel.org --- kernel/events/core.c | 85 ++++++++++++++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 20 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index bd20705..7135cb9 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7046,12 +7046,21 @@ out_put: ring_buffer_put(rb); } +/* + * A set of common sample data types saved even for non-sample records + * when event->attr.sample_id_all is set. + */ +#define PERF_SAMPLE_ID_ALL (PERF_SAMPLE_TID | PERF_SAMPLE_TIME | \ + PERF_SAMPLE_ID | PERF_SAMPLE_STREAM_ID | \ + PERF_SAMPLE_CPU | PERF_SAMPLE_IDENTIFIER) + static void __perf_event_header__init_id(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event, u64 sample_type) { data->type = event->attr.sample_type; + data->sample_flags |= data->type & PERF_SAMPLE_ID_ALL; header->size += event->id_header_size; if (sample_type & PERF_SAMPLE_TID) { @@ -7554,6 +7563,11 @@ perf_callchain(struct perf_event *event, struct pt_regs *regs) return callchain ?: &__empty_callchain; } +static __always_inline u64 __cond_set(u64 flags, u64 s, u64 d) +{ + return d * !!(flags & s); +} + void perf_prepare_sample(struct perf_event_header *header, struct perf_sample_data *data, struct perf_event *event, @@ -7569,14 +7583,24 @@ void perf_prepare_sample(struct perf_event_header *header, header->misc |= perf_misc_flags(regs); /* - * Clear the sample flags that have already been done by the - * PMU driver. + * Add the sample flags that are dependent to others. And clear the + * sample flags that have already been done by the PMU driver. */ - filtered_sample_type = sample_type & ~data->sample_flags; + filtered_sample_type = sample_type; + filtered_sample_type |= __cond_set(sample_type, PERF_SAMPLE_CODE_PAGE_SIZE, + PERF_SAMPLE_IP); + filtered_sample_type |= __cond_set(sample_type, PERF_SAMPLE_DATA_PAGE_SIZE | + PERF_SAMPLE_PHYS_ADDR, PERF_SAMPLE_ADDR); + filtered_sample_type |= __cond_set(sample_type, PERF_SAMPLE_STACK_USER, + PERF_SAMPLE_REGS_USER); + filtered_sample_type &= ~data->sample_flags; + __perf_event_header__init_id(header, data, event, filtered_sample_type); - if (sample_type & (PERF_SAMPLE_IP | PERF_SAMPLE_CODE_PAGE_SIZE)) + if (filtered_sample_type & PERF_SAMPLE_IP) { data->ip = perf_instruction_pointer(regs); + data->sample_flags |= PERF_SAMPLE_IP; + } if (filtered_sample_type & PERF_SAMPLE_CALLCHAIN) perf_sample_save_callchain(data, event, regs); @@ -7593,10 +7617,15 @@ void perf_prepare_sample(struct perf_event_header *header, data->sample_flags |= PERF_SAMPLE_BRANCH_STACK; } - if (sample_type & (PERF_SAMPLE_REGS_USER | PERF_SAMPLE_STACK_USER)) + if (filtered_sample_type & PERF_SAMPLE_REGS_USER) perf_sample_regs_user(&data->regs_user, regs); - if (sample_type & PERF_SAMPLE_REGS_USER) { + /* + * It cannot use the filtered_sample_type here as REGS_USER can be set + * by STACK_USER (using __cond_set() above) and we don't want to update + * the dyn_size if it's not requested by users. + */ + if ((sample_type & ~data->sample_flags) & PERF_SAMPLE_REGS_USER) { /* regs dump ABI info */ int size = sizeof(u64); @@ -7606,9 +7635,10 @@ void perf_prepare_sample(struct perf_event_header *header, } data->dyn_size += size; + data->sample_flags |= PERF_SAMPLE_REGS_USER; } - if (sample_type & PERF_SAMPLE_STACK_USER) { + if (filtered_sample_type & PERF_SAMPLE_STACK_USER) { /* * Either we need PERF_SAMPLE_STACK_USER bit to be always * processed as the last one or have additional check added @@ -7631,23 +7661,30 @@ void perf_prepare_sample(struct perf_event_header *header, data->stack_user_size = stack_size; data->dyn_size += size; + data->sample_flags |= PERF_SAMPLE_STACK_USER; } - if (filtered_sample_type & PERF_SAMPLE_WEIGHT_TYPE) + if (filtered_sample_type & PERF_SAMPLE_WEIGHT_TYPE) { data->weight.full = 0; + data->sample_flags |= PERF_SAMPLE_WEIGHT_TYPE; + } - if (filtered_sample_type & PERF_SAMPLE_DATA_SRC) + if (filtered_sample_type & PERF_SAMPLE_DATA_SRC) { data->data_src.val = PERF_MEM_NA; + data->sample_flags |= PERF_SAMPLE_DATA_SRC; + } - if (filtered_sample_type & PERF_SAMPLE_TRANSACTION) + if (filtered_sample_type & PERF_SAMPLE_TRANSACTION) { data->txn = 0; + data->sample_flags |= PERF_SAMPLE_TRANSACTION; + } - if (sample_type & (PERF_SAMPLE_ADDR | PERF_SAMPLE_PHYS_ADDR | PERF_SAMPLE_DATA_PAGE_SIZE)) { - if (filtered_sample_type & PERF_SAMPLE_ADDR) - data->addr = 0; + if (filtered_sample_type & PERF_SAMPLE_ADDR) { + data->addr = 0; + data->sample_flags |= PERF_SAMPLE_ADDR; } - if (sample_type & PERF_SAMPLE_REGS_INTR) { + if (filtered_sample_type & PERF_SAMPLE_REGS_INTR) { /* regs dump ABI info */ int size = sizeof(u64); @@ -7660,19 +7697,22 @@ void perf_prepare_sample(struct perf_event_header *header, } data->dyn_size += size; + data->sample_flags |= PERF_SAMPLE_REGS_INTR; } - if (sample_type & PERF_SAMPLE_PHYS_ADDR && - filtered_sample_type & PERF_SAMPLE_PHYS_ADDR) + if (filtered_sample_type & PERF_SAMPLE_PHYS_ADDR) { data->phys_addr = perf_virt_to_phys(data->addr); + data->sample_flags |= PERF_SAMPLE_PHYS_ADDR; + } #ifdef CONFIG_CGROUP_PERF - if (sample_type & PERF_SAMPLE_CGROUP) { + if (filtered_sample_type & PERF_SAMPLE_CGROUP) { struct cgroup *cgrp; /* protected by RCU */ cgrp = task_css_check(current, perf_event_cgrp_id, 1)->cgroup; data->cgroup = cgroup_id(cgrp); + data->sample_flags |= PERF_SAMPLE_CGROUP; } #endif @@ -7681,13 +7721,17 @@ void perf_prepare_sample(struct perf_event_header *header, * require PERF_SAMPLE_ADDR, kernel implicitly retrieve the data->addr, * but the value will not dump to the userspace. */ - if (sample_type & PERF_SAMPLE_DATA_PAGE_SIZE) + if (filtered_sample_type & PERF_SAMPLE_DATA_PAGE_SIZE) { data->data_page_size = perf_get_page_size(data->addr); + data->sample_flags |= PERF_SAMPLE_DATA_PAGE_SIZE; + } - if (sample_type & PERF_SAMPLE_CODE_PAGE_SIZE) + if (filtered_sample_type & PERF_SAMPLE_CODE_PAGE_SIZE) { data->code_page_size = perf_get_page_size(data->ip); + data->sample_flags |= PERF_SAMPLE_CODE_PAGE_SIZE; + } - if (sample_type & PERF_SAMPLE_AUX) { + if (filtered_sample_type & PERF_SAMPLE_AUX) { u64 size; header->size += sizeof(u64); /* size */ @@ -7705,6 +7749,7 @@ void perf_prepare_sample(struct perf_event_header *header, WARN_ON_ONCE(size + header->size > U16_MAX); data->dyn_size += size + sizeof(u64); /* size above */ + data->sample_flags |= PERF_SAMPLE_AUX; } header->size += data->dyn_size;