From patchwork Fri Dec 15 16:55:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 179413 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9418974dys; Fri, 15 Dec 2023 08:56:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IGd/FWATs6GvgpAQeM/qe0a+eGHfdQp6k1WySGYBCq983+xrNi4R5eCvhSg38iaz+hjlSKf X-Received: by 2002:a05:620a:1909:b0:77d:d53a:b54a with SMTP id bj9-20020a05620a190900b0077dd53ab54amr14823700qkb.40.1702659375565; Fri, 15 Dec 2023 08:56:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702659375; cv=none; d=google.com; s=arc-20160816; b=XivvRY3M99z20ekTKB8wDSysdFzVzbwJA1q+4Dr0Et7StWUbqP96O8s8d01L7+sL2b CvyNMKYF9jgzKu+fvXGcQX+uHbK+eHVMb33GYyXSFvvUfk6ZIqSoIuIWoVzRnF127Esg AWBtCam7BILQFLXPy3oVeBSTRODM3DeTm4f1NppAGgLLrw1C+QQuOrlWr3C13h8I6QtQ BKHu2hOQCJhIPq3xBEawOTDIGVQFWaHBA6JEyaP+LTP78mehQW0OLd+ZeRuFxDhP4AL6 INuIHFqMz6pJCvTiBWwP5jrEcOOIlUGITUil+Y1CV2Am84UZ52YYVYXWi6JQCxSIwMGv QVdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id; bh=XqGuKs0tfsSy1evrWKCa1kuqIIb1MHUr1y0iFPVqrC0=; fh=OI3eEy+AY0NH1j9zmiflrKRH8gD0OkiwAkNJqVJ3iV0=; b=taPpfnc/1DCKTqs5hpSkfoMIrTijVGZeuAMgU4e+OnbYnn2qyCXefE5gtC+feNoKiM qtxT5Y4mLqU+C+RPfHEUoHx4CKi1QbZf/ZNNtjc/YtZPLsopHOqXYjySA6jLrA9WMmXu iyX5EFK0eNDYD++HuDQ8FD8Z7pUlzvCZ1t5IqrD9/p8D4Ep3al917JnDlp1XSKx5Hn4m Cu7xpnRY9dFiQld3cDP0PfdwpsbHOS9+TfAOkJJotlonj5DFcXEd6jSYHzI/Jw5l7M4k zt0+8ZVwi7Sh+sAPwpeO71cHDV1+vYt9x+anmdx4hIP0prjqxZOi6K3X+knzswoMrgpE M3dg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-1339-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-1339-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id h6-20020a05620a400600b0077d5c1eb4absi18902263qko.308.2023.12.15.08.56.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 08:56:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-1339-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-1339-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-1339-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3DBEA1C23D2C for ; Fri, 15 Dec 2023 16:56:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C726541841; Fri, 15 Dec 2023 16:55:40 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1C753FB0A; Fri, 15 Dec 2023 16:55:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A785C433CA; Fri, 15 Dec 2023 16:55:38 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.97) (envelope-from ) id 1rEBUC-00000002vvd-1Ikg; Fri, 15 Dec 2023 11:56:28 -0500 Message-ID: <20231215165628.096822746@goodmis.org> User-Agent: quilt/0.67 Date: Fri, 15 Dec 2023 11:55:13 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Linus Torvalds Subject: [PATCH 1/2] ring-buffer: Replace rb_time_cmpxchg() with rb_time_cmp_and_update() References: <20231215165512.280088765@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785367757244542770 X-GMAIL-MSGID: 1785367757244542770 From: "Steven Rostedt (Google)" There's only one place that performs a 64-bit cmpxchg for the timestamp processing. The cmpxchg is only to set the write_stamp equal to the before_stamp, and if it doesn't get set, then the next event will simply be forced to add an absolute timestamp. Given that 64-bit cmpxchg is expensive on 32-bit, and the current workaround uses 3 consecutive 32-bit cmpxchg doesn't make it any faster. It's best to just not do the cmpxchg as a simple compare works for the accuracy of the timestamp. The only thing that will happen without the cmpxchg is the prepended absolute timestamp on the next event which is not that big of a deal as the path where this happens is seldom hit because it requires an interrupt to happen between a few lines of code that also writes an event into the same buffer. With this change, the 32-bit rb_time_t workaround can be removed. Signed-off-by: Steven Rostedt (Google) --- kernel/trace/ring_buffer.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index 1a26b3a1901f..c6842a4331a9 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ -762,6 +762,15 @@ static bool rb_time_cmpxchg(rb_time_t *t, u64 expect, u64 set) } #endif +/* + * Returns true if t == expect, and if possible will update with set, + * but t is not guaranteed to be updated even if this returns true + */ +static bool rb_time_cmp_and_update(rb_time_t *t, u64 expect, u64 set) +{ + return rb_time_cmpxchg(t, expect, set); +} + /* * Enable this to make sure that the event passed to * ring_buffer_event_time_stamp() is not committed and also @@ -3622,9 +3631,17 @@ __rb_reserve_next(struct ring_buffer_per_cpu *cpu_buffer, barrier(); /*E*/ if (write == (local_read(&tail_page->write) & RB_WRITE_MASK) && info->after < ts && - rb_time_cmpxchg(&cpu_buffer->write_stamp, - info->after, ts)) { - /* Nothing came after this event between C and E */ + rb_time_cmp_and_update(&cpu_buffer->write_stamp, + info->after, ts)) { + /* + * Nothing came after this event between C and E it is + * safe to use info->after for the delta. + * The above rb_time_cmp_and_update() may or may not + * have updated the write_stamp. If it did not then + * the next event will simply add an absolute timestamp + * as the write_stamp will be different than the + * before_stamp. + */ info->delta = ts - info->after; } else { /* From patchwork Fri Dec 15 16:55:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Rostedt X-Patchwork-Id: 179414 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9419028dys; Fri, 15 Dec 2023 08:56:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IFx8Y/vZ3nMZX7GGixaYT1aLvc8ucvVyC5huAkKuo9zwgvz8m6o2VMU3rfie4AgI7pz4EO7 X-Received: by 2002:a05:6122:458f:b0:4b2:c554:dfcc with SMTP id de15-20020a056122458f00b004b2c554dfccmr11732331vkb.27.1702659381055; Fri, 15 Dec 2023 08:56:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702659381; cv=none; d=google.com; s=arc-20160816; b=kZQEM/qr0GxLEbMqzo9Cul0RCy9M1ynNI5uRY16u+LmOnaa2sus2c0tvGGwkPstB+P QHvQ1GNKu/ZYs+uTJIiQ+aQ5epyS9pRCkBjYlQRERnWC9598VED2nT9XuOJXPhJGpy59 CQ0yit5RZyWnPqZ5I2p2LvuYxbN0kKPHx8lk174hDCg9Bde3q0JYR9pwUMnSgB6w8pRo TvY7mlMqU7v43lvJ0ufbqi9csEQXwb6rWosKGr6drv4qSmkNHAHdrVoHCQnvlLREQBVq rZtrF06B5wa6UrKllPLAu6EyUlp+zzSbKmdsqHKErGuNbtP/pxBiBgLloKXKPF2x8QUt hbnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id; bh=9yO5AyIf1Th1dofxWU0bOx04LSssJTe5hyHI2+FUgws=; fh=OI3eEy+AY0NH1j9zmiflrKRH8gD0OkiwAkNJqVJ3iV0=; b=KwT8q4214zR8pr6o56FonXh3pLDs5h8tgblEFjHZ0GaxCAsukvLvQ/bdjoUkn5BvR7 V2/8Vc3anbDZ1BXT96vZDMjH9tMAC2riocgIVZS7KRA5gxvuVB86er2GzjAEEVdUaaiX ClDzaZ7UVrLh439JjQdZd6P6HgEd+1lhpoQr+NKe7CocYGvlddgRT8KHGeO5GnWkeg3m DGtg2Apx/o/pIukm1EB4W5l/bStAzeoyKjQy29/9pj9h21MuADs7S9oEMg259svNpLLQ auq1Xs5cSEy0Gj77lMqY1qkrZ+dnkwSZ6Ukve2n/LMETWwlKf4W3PMnfXlYJSBfizX2i fkqA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-1340-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-1340-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id bk35-20020a05620a1a2300b007726499db83si19662457qkb.229.2023.12.15.08.56.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 08:56:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-1340-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel+bounces-1340-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-1340-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id B309D1C23D90 for ; Fri, 15 Dec 2023 16:56:20 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1881F41856; Fri, 15 Dec 2023 16:55:41 +0000 (UTC) X-Original-To: linux-kernel@vger.kernel.org Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 246263EA81; Fri, 15 Dec 2023 16:55:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A14B5C433CC; Fri, 15 Dec 2023 16:55:38 +0000 (UTC) Received: from rostedt by gandalf with local (Exim 4.97) (envelope-from ) id 1rEBUC-00000002vw7-2XUf; Fri, 15 Dec 2023 11:56:28 -0500 Message-ID: <20231215165628.387150742@goodmis.org> User-Agent: quilt/0.67 Date: Fri, 15 Dec 2023 11:55:14 -0500 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org Cc: Masami Hiramatsu , Mark Rutland , Mathieu Desnoyers , Andrew Morton , Linus Torvalds Subject: [PATCH 2/2] ring-buffer: Remove 32bit timestamp logic References: <20231215165512.280088765@goodmis.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785367763133118286 X-GMAIL-MSGID: 1785367763133118286 From: "Steven Rostedt (Google)" Each event has a 27 bit timestamp delta that is used to hold the delta from the last event. If the time between events is greater than 2^27, then a timestamp is added that holds a 59 bit absolute timestamp. Until a389d86f7fd09 ("ring-buffer: Have nested events still record running time stamp"), if an interrupt interrupted an event in progress, all the events delta would be zero to not deal with the races that need to be handled. The commit a389d86f7fd09 changed that to handle the races giving all events, even those that preempt other events, still have an accurate timestamp. To handle those races requires performing 64-bit cmpxchg on the timestamps. But doing 64-bit cmpxchg on 32-bit architectures is considered very slow. To try to deal with this the timestamp logic was broken into two and then three 32-bit cmpxchgs, with the thought that two (or three) 32-bit cmpxchgs are still faster than a single 64-bit cmpxchg on 32-bit architectures. Part of the problem with this is that I didn't have any 32-bit architectures to test on. After hitting several subtle bugs in this code, an effort was made to try and see if three 32-bit cmpxchgs are indeed faster than a single 64-bit. After a few people brushed off the dust of their old 32-bit machines, tests were done, and even though 32-bit cmpxchg was faster than a single 64-bit, it was in the order of 50% at best, not 300%. After some more refactoring of the code, 3 of the 4 64-bit cmpxchg were removed: https://lore.kernel.org/linux-trace-kernel/20231211114420.36dde01b@gandalf.local.home https://lore.kernel.org/linux-trace-kernel/20231214222921.193037a7@gandalf.local.home https://lore.kernel.org/linux-trace-kernel/20231215081810.1f4f38fe@rorschach.local.home The last remaining 64-bit cmpxchg is only in a slow path and to keep the next event from having to add an absolute timestamp. This is not worth the trouble of a slow cmpxchg. Only do the cmpxchg on 64-bit architectures and just have the 32-bit architectures insert an absolute timestamp for the next event that comes after the slow path. The slow path only happens if the event being recorded is interrupted between a few lines of code and that interrupt writes an event into the same buffer. This seldom happens and should not cause any issues by inserting an extra absolute timestamp when it does happen. Now the complex 32-bit workaround for 64-bit cmpxchg can be removed. Link: https://lore.kernel.org/all/20231213214632.15047c40@gandalf.local.home/ Signed-off-by: Steven Rostedt (Google) --- kernel/trace/ring_buffer.c | 230 +++---------------------------------- 1 file changed, 19 insertions(+), 211 deletions(-) diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c index c6842a4331a9..47f9eda99769 100644 --- a/kernel/trace/ring_buffer.c +++ b/kernel/trace/ring_buffer.c @@ -463,27 +463,9 @@ enum { RB_CTX_MAX }; -#if BITS_PER_LONG == 32 -#define RB_TIME_32 -#endif - -/* To test on 64 bit machines */ -//#define RB_TIME_32 - -#ifdef RB_TIME_32 - -struct rb_time_struct { - local_t cnt; - local_t top; - local_t bottom; - local_t msb; -}; -#else -#include struct rb_time_struct { local64_t time; }; -#endif typedef struct rb_time_struct rb_time_t; #define MAX_NEST 5 @@ -573,202 +555,35 @@ struct ring_buffer_iter { int missed_events; }; -#ifdef RB_TIME_32 - -/* - * On 32 bit machines, local64_t is very expensive. As the ring - * buffer doesn't need all the features of a true 64 bit atomic, - * on 32 bit, it uses these functions (64 still uses local64_t). - * - * For the ring buffer, 64 bit required operations for the time is - * the following: - * - * - Reads may fail if it interrupted a modification of the time stamp. - * It will succeed if it did not interrupt another write even if - * the read itself is interrupted by a write. - * It returns whether it was successful or not. - * - * - Writes always succeed and will overwrite other writes and writes - * that were done by events interrupting the current write. - * - * - A write followed by a read of the same time stamp will always succeed, - * but may not contain the same value. - * - * - A cmpxchg will fail if it interrupted another write or cmpxchg. - * Other than that, it acts like a normal cmpxchg. - * - * The 60 bit time stamp is broken up by 30 bits in a top and bottom half - * (bottom being the least significant 30 bits of the 60 bit time stamp). - * - * The two most significant bits of each half holds a 2 bit counter (0-3). - * Each update will increment this counter by one. - * When reading the top and bottom, if the two counter bits match then the - * top and bottom together make a valid 60 bit number. - */ -#define RB_TIME_SHIFT 30 -#define RB_TIME_VAL_MASK ((1 << RB_TIME_SHIFT) - 1) -#define RB_TIME_MSB_SHIFT 60 - -static inline int rb_time_cnt(unsigned long val) -{ - return (val >> RB_TIME_SHIFT) & 3; -} - -static inline u64 rb_time_val(unsigned long top, unsigned long bottom) -{ - u64 val; - - val = top & RB_TIME_VAL_MASK; - val <<= RB_TIME_SHIFT; - val |= bottom & RB_TIME_VAL_MASK; - - return val; -} - -static inline bool __rb_time_read(rb_time_t *t, u64 *ret, unsigned long *cnt) -{ - unsigned long top, bottom, msb; - unsigned long c; - - /* - * If the read is interrupted by a write, then the cnt will - * be different. Loop until both top and bottom have been read - * without interruption. - */ - do { - c = local_read(&t->cnt); - top = local_read(&t->top); - bottom = local_read(&t->bottom); - msb = local_read(&t->msb); - } while (c != local_read(&t->cnt)); - - *cnt = rb_time_cnt(top); - - /* If top, msb or bottom counts don't match, this interrupted a write */ - if (*cnt != rb_time_cnt(msb) || *cnt != rb_time_cnt(bottom)) - return false; - - /* The shift to msb will lose its cnt bits */ - *ret = rb_time_val(top, bottom) | ((u64)msb << RB_TIME_MSB_SHIFT); - return true; -} - -static bool rb_time_read(rb_time_t *t, u64 *ret) -{ - unsigned long cnt; - - return __rb_time_read(t, ret, &cnt); -} - -static inline unsigned long rb_time_val_cnt(unsigned long val, unsigned long cnt) -{ - return (val & RB_TIME_VAL_MASK) | ((cnt & 3) << RB_TIME_SHIFT); -} - -static inline void rb_time_split(u64 val, unsigned long *top, unsigned long *bottom, - unsigned long *msb) -{ - *top = (unsigned long)((val >> RB_TIME_SHIFT) & RB_TIME_VAL_MASK); - *bottom = (unsigned long)(val & RB_TIME_VAL_MASK); - *msb = (unsigned long)(val >> RB_TIME_MSB_SHIFT); -} - -static inline void rb_time_val_set(local_t *t, unsigned long val, unsigned long cnt) -{ - val = rb_time_val_cnt(val, cnt); - local_set(t, val); -} - -static void rb_time_set(rb_time_t *t, u64 val) -{ - unsigned long cnt, top, bottom, msb; - - rb_time_split(val, &top, &bottom, &msb); - - /* Writes always succeed with a valid number even if it gets interrupted. */ - do { - cnt = local_inc_return(&t->cnt); - rb_time_val_set(&t->top, top, cnt); - rb_time_val_set(&t->bottom, bottom, cnt); - rb_time_val_set(&t->msb, val >> RB_TIME_MSB_SHIFT, cnt); - } while (cnt != local_read(&t->cnt)); -} - -static inline bool -rb_time_read_cmpxchg(local_t *l, unsigned long expect, unsigned long set) -{ - return local_try_cmpxchg(l, &expect, set); -} - -static bool rb_time_cmpxchg(rb_time_t *t, u64 expect, u64 set) -{ - unsigned long cnt, top, bottom, msb; - unsigned long cnt2, top2, bottom2, msb2; - u64 val; - - /* Any interruptions in this function should cause a failure */ - cnt = local_read(&t->cnt); - - /* The cmpxchg always fails if it interrupted an update */ - if (!__rb_time_read(t, &val, &cnt2)) - return false; - - if (val != expect) - return false; - - if ((cnt & 3) != cnt2) - return false; - - cnt2 = cnt + 1; - - rb_time_split(val, &top, &bottom, &msb); - msb = rb_time_val_cnt(msb, cnt); - top = rb_time_val_cnt(top, cnt); - bottom = rb_time_val_cnt(bottom, cnt); - - rb_time_split(set, &top2, &bottom2, &msb2); - msb2 = rb_time_val_cnt(msb2, cnt); - top2 = rb_time_val_cnt(top2, cnt2); - bottom2 = rb_time_val_cnt(bottom2, cnt2); - - if (!rb_time_read_cmpxchg(&t->cnt, cnt, cnt2)) - return false; - if (!rb_time_read_cmpxchg(&t->msb, msb, msb2)) - return false; - if (!rb_time_read_cmpxchg(&t->top, top, top2)) - return false; - if (!rb_time_read_cmpxchg(&t->bottom, bottom, bottom2)) - return false; - return true; -} - -#else /* 64 bits */ - -/* local64_t always succeeds */ - -static inline bool rb_time_read(rb_time_t *t, u64 *ret) +static inline void rb_time_read(rb_time_t *t, u64 *ret) { *ret = local64_read(&t->time); - return true; } static void rb_time_set(rb_time_t *t, u64 val) { local64_set(&t->time, val); } -static bool rb_time_cmpxchg(rb_time_t *t, u64 expect, u64 set) -{ - return local64_try_cmpxchg(&t->time, &expect, set); -} +#if BITS_PER_LONG == 32 +#define RB_TIME_32 +#else +#include #endif +/* To test on 64 bit machines */ +//#define RB_TIME_32 + /* * Returns true if t == expect, and if possible will update with set, * but t is not guaranteed to be updated even if this returns true */ static bool rb_time_cmp_and_update(rb_time_t *t, u64 expect, u64 set) { - return rb_time_cmpxchg(t, expect, set); +#ifdef RB_TIME_32 + return expect == READ_ONCE(t->time); +#else + return local64_try_cmpxchg(&t->time, &expect, set); +#endif } /* @@ -876,10 +691,7 @@ u64 ring_buffer_event_time_stamp(struct trace_buffer *buffer, WARN_ONCE(1, "nest (%d) greater than max", nest); fail: - /* Can only fail on 32 bit */ - if (!rb_time_read(&cpu_buffer->write_stamp, &ts)) - /* Screw it, just read the current time */ - ts = rb_time_stamp(cpu_buffer->buffer); + rb_time_read(&cpu_buffer->write_stamp, &ts); return ts; } @@ -2876,7 +2688,7 @@ rb_check_timestamp(struct ring_buffer_per_cpu *cpu_buffer, (unsigned long long)info->ts, (unsigned long long)info->before, (unsigned long long)info->after, - (unsigned long long)(rb_time_read(&cpu_buffer->write_stamp, &write_stamp) ? write_stamp : 0), + (unsigned long long)({rb_time_read(&cpu_buffer->write_stamp, &write_stamp); write_stamp;}), sched_clock_stable() ? "" : "If you just came from a suspend/resume,\n" "please switch to the trace global clock:\n" @@ -3553,16 +3365,14 @@ __rb_reserve_next(struct ring_buffer_per_cpu *cpu_buffer, struct ring_buffer_event *event; struct buffer_page *tail_page; unsigned long tail, write, w; - bool a_ok; - bool b_ok; /* Don't let the compiler play games with cpu_buffer->tail_page */ tail_page = info->tail_page = READ_ONCE(cpu_buffer->tail_page); /*A*/ w = local_read(&tail_page->write) & RB_WRITE_MASK; barrier(); - b_ok = rb_time_read(&cpu_buffer->before_stamp, &info->before); - a_ok = rb_time_read(&cpu_buffer->write_stamp, &info->after); + rb_time_read(&cpu_buffer->before_stamp, &info->before); + rb_time_read(&cpu_buffer->write_stamp, &info->after); barrier(); info->ts = rb_time_stamp(cpu_buffer->buffer); @@ -3577,7 +3387,7 @@ __rb_reserve_next(struct ring_buffer_per_cpu *cpu_buffer, if (!w) { /* Use the sub-buffer timestamp */ info->delta = 0; - } else if (unlikely(!a_ok || !b_ok || info->before != info->after)) { + } else if (unlikely(info->before != info->after)) { info->add_timestamp |= RB_ADD_STAMP_FORCE | RB_ADD_STAMP_EXTEND; info->length += RB_LEN_TIME_EXTEND; } else { @@ -3624,9 +3434,7 @@ __rb_reserve_next(struct ring_buffer_per_cpu *cpu_buffer, } else { u64 ts; /* SLOW PATH - Interrupted between A and C */ - a_ok = rb_time_read(&cpu_buffer->write_stamp, &info->after); - /* Was interrupted before here, write_stamp must be valid */ - RB_WARN_ON(cpu_buffer, !a_ok); + rb_time_read(&cpu_buffer->write_stamp, &info->after); ts = rb_time_stamp(cpu_buffer->buffer); barrier(); /*E*/ if (write == (local_read(&tail_page->write) & RB_WRITE_MASK) &&