From patchwork Fri May 19 10:20:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 96365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1131077vqo; Fri, 19 May 2023 03:36:37 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5XSGjTghO8lTeYMouzVosCkqHI2g7TRTf4HreGt9Jf3/cgh4D8Li5kV8sLDLKRNA/ObgRJ X-Received: by 2002:a05:6a20:1006:b0:100:132e:c7c4 with SMTP id gs6-20020a056a20100600b00100132ec7c4mr1627752pzc.11.1684492597400; Fri, 19 May 2023 03:36:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684492597; cv=none; d=google.com; s=arc-20160816; b=1IS6dgeLvmZ0fwF//Grm/ZhvKT1mXZu1dIhz4msiqQVQR78RFHj6jyAPRvl15QwZxP I7NgXNKXoghwWaE3u/AVuQS0QflOcKonFdPwHYD7V3ysZhBybzwSoVKuwuzzaXTHRXNd xwORhQBDm79291k66MBlwcBF87IC1XTJRhIIrbhZjYcFjjuhal+qLva63ViE+ovFW7zy k7ySqf54SOmMteaO0IVxweJSwwrc96+CfUwF+eOlELX8fIx8vGC+ipeEU/XB7XhRkaJs JWL8zXKh7+Xeh1BoSZXXxyZwECukX11ml56WSKWvuF38QzOh0msXll37IRs6/queT/TO lFpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=9vqHKqr8qLvPAc6CZjuNpJuO7WKwN2GzdLEYPcbYwQc=; b=mo5djOVc3s6/L5yv04hup3JA49FOCYDktXiHD8OiV52JGae8eU0UUqqoll3SQD3FVp im88QyGXe41hsQiUbZw8RgVhU1Al9qHQtoBMisEiH5UsQwrt7mdUgztmaw5+Q46i4+Xm Err8QZChhA6qxzUa/ig9b8qysPjnucGEUERAKSGRwhgzmsjSxn0caFpbXUOqpMtevlh/ L1w5RM2r4uV+O81iTNc6FeNMd32As81tnHEFmpsU1ZqPDvBHil8OWrAXHdBQ5yQ6XmbE blfWfS1Cwb4/gru+Xv7gW3doqNTGldJEJnkdNlAlJ8AXzMg6QXtRfryTJk772acelH5e +MCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=U9rtCzfp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c4-20020a637244000000b005342409bdcasi3272643pgn.723.2023.05.19.03.36.24; Fri, 19 May 2023 03:36:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=U9rtCzfp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232085AbjESKeq (ORCPT + 99 others); Fri, 19 May 2023 06:34:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231982AbjESKeW (ORCPT ); Fri, 19 May 2023 06:34:22 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86D3B10CE; Fri, 19 May 2023 03:34:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=9vqHKqr8qLvPAc6CZjuNpJuO7WKwN2GzdLEYPcbYwQc=; b=U9rtCzfpSqdjhvWW5TY81J8GZh IYlZTFIX3AKFLrM8SUN+ShJMpaJYzC+L03QJfiw7hUfmJNbj4xhNr+ozknGdDXCNL+ISpphJ20T2c lCBD2Ip5Lg+StGepFx+6Utikm0Sy2o9SyjygXPiazxyIdCTXM4fQhF/YOKAIlJArKITFtryQZ4pPH 1DkGKvPyRMhmm6cqgJTMmtn2Oh/CaLM6zmb1oV09yZyi8kwH91mzd1bhmcv9aU9Xc2NTvQRzNlc+m hQXoQdO/h+HpVOSIWO24LXum0ZEJ8pKP8fBKJ3QLT8cyhVTVhM1hS1xCE+JIc6izMjtpjibuU/UNM fXT3BH1Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pzxPv-006UqU-TS; Fri, 19 May 2023 10:33:00 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D987C300AFA; Fri, 19 May 2023 12:32:55 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B3B68235EF082; Fri, 19 May 2023 12:32:55 +0200 (CEST) Message-ID: <20230519102715.233598176@infradead.org> User-Agent: quilt/0.66 Date: Fri, 19 May 2023 12:20:59 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [PATCH v2 01/13] seqlock/latch: Provide raw_read_seqcount_latch_retry() References: <20230519102058.581557770@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766318509659847080?= X-GMAIL-MSGID: =?utf-8?q?1766318509659847080?= The read side of seqcount_latch consists of: do { seq = raw_read_seqcount_latch(&latch->seq); ... } while (read_seqcount_latch_retry(&latch->seq, seq)); which is asymmetric in the raw_ department, and sure enough, read_seqcount_latch_retry() includes (explicit) instrumentation where raw_read_seqcount_latch() does not. This inconsistency becomes a problem when trying to use it from noinstr code. As such, fix it by renaming and re-implementing raw_read_seqcount_latch_retry() without the instrumentation. Specifically the instrumentation in question is kcsan_atomic_next(0) in do___read_seqcount_retry(). Loosing this annotation is not a problem because raw_read_seqcount_latch() does not pass through kcsan_atomic_next(KCSAN_SEQLOCK_REGION_MAX). Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Thomas Gleixner Reviewed-by: Petr Mladek --- include/linux/rbtree_latch.h | 2 +- include/linux/seqlock.h | 15 ++++++++------- kernel/printk/printk.c | 2 +- kernel/time/sched_clock.c | 2 +- kernel/time/timekeeping.c | 4 ++-- 5 files changed, 13 insertions(+), 12 deletions(-) --- a/include/linux/rbtree_latch.h +++ b/include/linux/rbtree_latch.h @@ -206,7 +206,7 @@ latch_tree_find(void *key, struct latch_ do { seq = raw_read_seqcount_latch(&root->seq); node = __lt_find(key, root, seq & 1, ops->comp); - } while (read_seqcount_latch_retry(&root->seq, seq)); + } while (raw_read_seqcount_latch_retry(&root->seq, seq)); return node; } --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -671,9 +671,9 @@ typedef struct { * * Return: sequence counter raw value. Use the lowest bit as an index for * picking which data copy to read. The full counter must then be checked - * with read_seqcount_latch_retry(). + * with raw_read_seqcount_latch_retry(). */ -static inline unsigned raw_read_seqcount_latch(const seqcount_latch_t *s) +static __always_inline unsigned raw_read_seqcount_latch(const seqcount_latch_t *s) { /* * Pairs with the first smp_wmb() in raw_write_seqcount_latch(). @@ -683,16 +683,17 @@ static inline unsigned raw_read_seqcount } /** - * read_seqcount_latch_retry() - end a seqcount_latch_t read section + * raw_read_seqcount_latch_retry() - end a seqcount_latch_t read section * @s: Pointer to seqcount_latch_t * @start: count, from raw_read_seqcount_latch() * * Return: true if a read section retry is required, else false */ -static inline int -read_seqcount_latch_retry(const seqcount_latch_t *s, unsigned start) +static __always_inline int +raw_read_seqcount_latch_retry(const seqcount_latch_t *s, unsigned start) { - return read_seqcount_retry(&s->seqcount, start); + smp_rmb(); + return unlikely(READ_ONCE(s->seqcount.sequence) != start); } /** @@ -752,7 +753,7 @@ read_seqcount_latch_retry(const seqcount * entry = data_query(latch->data[idx], ...); * * // This includes needed smp_rmb() - * } while (read_seqcount_latch_retry(&latch->seq, seq)); + * } while (raw_read_seqcount_latch_retry(&latch->seq, seq)); * * return entry; * } --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -597,7 +597,7 @@ static u64 latched_seq_read_nolock(struc seq = raw_read_seqcount_latch(&ls->latch); idx = seq & 0x1; val = ls->val[idx]; - } while (read_seqcount_latch_retry(&ls->latch, seq)); + } while (raw_read_seqcount_latch_retry(&ls->latch, seq)); return val; } --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -77,7 +77,7 @@ notrace struct clock_read_data *sched_cl notrace int sched_clock_read_retry(unsigned int seq) { - return read_seqcount_latch_retry(&cd.seq, seq); + return raw_read_seqcount_latch_retry(&cd.seq, seq); } unsigned long long notrace sched_clock(void) --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -450,7 +450,7 @@ static __always_inline u64 __ktime_get_f tkr = tkf->base + (seq & 0x01); now = ktime_to_ns(tkr->base); now += fast_tk_get_delta_ns(tkr); - } while (read_seqcount_latch_retry(&tkf->seq, seq)); + } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); return now; } @@ -566,7 +566,7 @@ static __always_inline u64 __ktime_get_r basem = ktime_to_ns(tkr->base); baser = ktime_to_ns(tkr->base_real); delta = fast_tk_get_delta_ns(tkr); - } while (read_seqcount_latch_retry(&tkf->seq, seq)); + } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); if (mono) *mono = basem + delta;