From patchwork Tue Jan 9 22:35:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 186590 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:2411:b0:101:2151:f287 with SMTP id m17csp446904dyi; Tue, 9 Jan 2024 14:35:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IH5IkkjtNV+XTWirshE5e6X6eGoyB7mxIi4bJH3hBcpweY8XU4WJ1Mxf2mGq1n2queIknU8 X-Received: by 2002:a9d:7395:0:b0:6dd:ea10:b3e3 with SMTP id j21-20020a9d7395000000b006ddea10b3e3mr58316otk.41.1704839713147; Tue, 09 Jan 2024 14:35:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704839713; cv=none; d=google.com; s=arc-20160816; b=XVr3CIpZPwBQt9OJ7elJzojcZ0cmnxvnTsOxP/dW8kPawKdSy6ylGqmZhASMU4f42d EIW1G7a4CRoEJzeykC+R9q7X3wFSCtS8jUzJuaq2TXBXAR1yEUmSUymS2Qp60puQ3Wf7 7l198CIDVZ4R9vaPtv2mP6/4qybPUag4V4CjP0EblkUGfmld29VrR9fwYfP0u5Qfkrgv kFaL00nGBek0IeHayXiSJsz3HfYPLdfVBcBRCAaCspc81DPQ9n6+NHwW/EbWcEwcSofF xyl3QBg80LU4W64BeAMnCEy+sWYllUMudhOb9B0eOwKGG0oiTIwQxKKmC69dk0yPSQBp uBTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:message-id:subject:cc:to:from :date; bh=tOApRXVO6TG6BuxN/YRNOOQrY5hKmxOjDp0Ys9vjH9o=; fh=T5PMUC8J1xY+oXKI6cNB7KdCdwgfXWRD/MdTbgMMo0w=; b=WEUFbZW5KLTKy2ajwKrXn+UeNhpCwqJt90QKAKujuudyNOKJHr52zr6XmnF2QShYR/ abJJzwoTt7oJi270p7NzOHqvpd/5LDrbOPx6eBDaGtzeiGVimqp2w51h0scXWpBQV/qV cnv51eFgKnMpgvpqQC+Br/IvDO/Em6iud26b9vjSY12vd68krnH8olLrr7QcoJ88K/UV OoT7WrHuoIQq42u4kC+wzOnrrKe/bzX3RSqINZpbh7EKChBPZ+FfVkXzJ5TxxV47eEdX qmnfdnOxFAWZKJen94XW/UXKXjxNvmZnyZrXza3FaPYYYr0OAsel0zUTk1FJvEIozVTp Sh+Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-21480-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-21480-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id w19-20020a05622a135300b004299926b32dsi3079490qtk.432.2024.01.09.14.35.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jan 2024 14:35:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-21480-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-21480-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-21480-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id F224F1C24FE7 for ; Tue, 9 Jan 2024 22:35:06 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 40BCC3E47C; Tue, 9 Jan 2024 22:34:56 +0000 (UTC) Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B39C4364BD; Tue, 9 Jan 2024 22:34:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9DCC4C433C7; Tue, 9 Jan 2024 22:34:53 +0000 (UTC) Date: Tue, 9 Jan 2024 17:35:52 -0500 From: Steven Rostedt To: LKML , Linux Trace Kernel Cc: Masami Hiramatsu , Mathieu Desnoyers , Vincent Donnefort Subject: [PATCH v2] ring-buffer: Have mmapped ring buffer keep track of missed events Message-ID: <20240109173552.0b40d199@gandalf.local.home> X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787647001849171699 X-GMAIL-MSGID: 1787654006793473999 From: "Steven Rostedt (Google)" While testing libtracefs on the mmapped ring buffer, the test that checks if missed events are accounted for failed when using the mapped buffer. This is because the mapped page does not update the missed events that were dropped because the writer filled up the ring buffer before the reader could catch it. Add the missed events to the reader page/sub-buffer when the IOCTL is done and a new reader page is acquired. Note that all accesses to the reader_page via rb_page_commit() had to be switched to rb_page_size(), and rb_page_size() which was just a copy of rb_page_commit() but now it masks out the RB_MISSED bits. This is needed as the mapped reader page is still active in the ring buffer code and where it reads the commit field of the bpage for the size, it now must mask it otherwise the missed bits that are now set will corrupt the size returned. Signed-off-by: Steven Rostedt (Google) --- Changes since v1: https://lore.kernel.org/all/20240109205112.74225-5-rostedt@goodmis.org/ - Forgot to update the "real end" so that it has room to add the number of missed events. Without that update, it would just say "missed events" but not how many events were missed. kernel/trace/ring_buffer.c | 61 ++++++++++++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 6 deletions(-) diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index 07dae67424a9..b111f0694805 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ -312,6 +312,8 @@ static u64 rb_event_time_stamp(struct ring_buffer_event *event) /* Missed count stored at end */ #define RB_MISSED_STORED (1 << 30) +#define RB_MISSED_MASK (3 << 30) + struct buffer_data_page { u64 time_stamp; /* page time stamp */ local_t commit; /* write committed index */ @@ -2303,7 +2305,7 @@ rb_iter_head_event(struct ring_buffer_iter *iter) /* Size is determined by what has been committed */ static __always_inline unsigned rb_page_size(struct buffer_page *bpage) { - return rb_page_commit(bpage); + return rb_page_commit(bpage) & ~RB_MISSED_MASK; } static __always_inline unsigned @@ -2769,6 +2771,7 @@ static void rb_add_timestamp(struct ring_buffer_per_cpu *cpu_buffer, once++; pr_warn("Ring buffer clock went backwards: %llu -> %llu\n", info->before, info->ts); + dump_stack(); } } else rb_check_timestamp(cpu_buffer, info); @@ -3930,7 +3933,7 @@ static bool rb_per_cpu_empty(struct ring_buffer_per_cpu *cpu_buffer) return true; /* Reader should exhaust content in reader page */ - if (reader->read != rb_page_commit(reader)) + if (reader->read != rb_page_size(reader)) return false; /* @@ -4401,7 +4404,7 @@ int ring_buffer_iter_empty(struct ring_buffer_iter *iter) return ((iter->head_page == commit_page && iter->head >= commit) || (iter->head_page == reader && commit_page == head_page && head_page->read == commit && - iter->head == rb_page_commit(cpu_buffer->reader_page))); + iter->head == rb_page_size(cpu_buffer->reader_page))); } EXPORT_SYMBOL_GPL(ring_buffer_iter_empty); @@ -5745,7 +5748,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer, event = rb_reader_event(cpu_buffer); read = reader->read; - commit = rb_page_commit(reader); + commit = rb_page_size(reader); /* Check if any events were dropped */ missed_events = cpu_buffer->lost_events; @@ -5822,7 +5825,7 @@ int ring_buffer_read_page(struct trace_buffer *buffer, } else { /* update the entry counter */ cpu_buffer->read += rb_page_entries(reader); - cpu_buffer->read_bytes += rb_page_commit(reader); + cpu_buffer->read_bytes += rb_page_size(reader); /* swap the pages */ rb_init_page(bpage); @@ -6349,6 +6352,8 @@ struct page *ring_buffer_map_fault(struct trace_buffer *buffer, int cpu, int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu) { struct ring_buffer_per_cpu *cpu_buffer; + struct buffer_page *reader; + unsigned long missed_events; unsigned long reader_size; unsigned long flags; @@ -6374,9 +6379,53 @@ int ring_buffer_map_get_reader(struct trace_buffer *buffer, int cpu) goto out; } - if (WARN_ON(!rb_get_reader_page(cpu_buffer))) + reader = rb_get_reader_page(cpu_buffer); + if (WARN_ON(!reader)) goto out; + /* Check if any events were dropped */ + missed_events = cpu_buffer->lost_events; + + if (cpu_buffer->reader_page != cpu_buffer->commit_page) { + if (missed_events) { + struct buffer_data_page *bpage = reader->page; + unsigned int commit; + /* + * Use the real_end for the data size, + * This gives us a chance to store the lost events + * on the page. + */ + if (reader->real_end) + local_set(&bpage->commit, reader->real_end); + /* + * If there is room at the end of the page to save the + * missed events, then record it there. + */ + commit = rb_page_size(reader); + if (buffer->subbuf_size - commit >= sizeof(missed_events)) { + memcpy(&bpage->data[commit], &missed_events, + sizeof(missed_events)); + local_add(RB_MISSED_STORED, &bpage->commit); + commit += sizeof(missed_events); + } + local_add(RB_MISSED_EVENTS, &bpage->commit); + /* + * This page may be off to user land. Zero it out here. + */ + if (commit < buffer->subbuf_size) + memset(&bpage->data[commit], 0, + buffer->subbuf_size - commit); + } + } else { + /* + * There really shouldn't be any missed events if the commit + * is on the reader page. + */ + WARN_ON_ONCE(missed_events); + } + + cpu_buffer->lost_events = 0; + goto consume; out: raw_spin_unlock_irqrestore(&cpu_buffer->reader_lock, flags);