From patchwork Mon May 8 21:19:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91291 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2451347vqo; Mon, 8 May 2023 14:51:10 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6YjtwQIBgM/hH/gg/thgdwpW1wcIX50d5sAXVlYsUcl02FOmhpynG/Sa98nXpDxMTPZqpj X-Received: by 2002:a05:6a20:9f4d:b0:eb:a859:cace with SMTP id ml13-20020a056a209f4d00b000eba859cacemr12927839pzb.41.1683582670630; Mon, 08 May 2023 14:51:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683582670; cv=none; d=google.com; s=arc-20160816; b=ff+7/4rk0vC6vUY5073INMYHFfPNGRBoYiPqMzVGwqCKRIzy5aCdz89ersUGWj9jIK 3BuDVKF+kfuBQNf4E6iqbmm3TighqM614ha8fnc0lSWJlYfcshdhgn+oaEhbgUsKxtMi iX9jubxbFS0+l1mtLprR9BL6ltsUYSJNv9wZKCO5cgF2eTY1Bp3byxEyoHU+GUKzqDH4 nm+yOfSBe4MVb1DYSoeSK8JykDsqwMTUuvPw1QZMitlKCW4Vb0GlyYL+lRQ7HP3RhPys xVgfGM42nvLNtubz61nz/hEWKa2Q9HnN8F614zG9XrS9D+0TrmSUhFXqKQ+4Wr/pqdmj zWRw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=qJ542JOkywuQy44rpDjAFBKOiAWpMNGNJLpO6nBwf7o=; b=bnVmYwnQACNS8LmwJTQ8r7HB+wM2K8eW6+yvHpqQ/7H7e19Dv4utBvYJD9T986bfPe BILWDKds4NG1eltTGuZZyjgFE7biPs4Hf1Flxt6N5U1Y6+Frm6/rHKhqqOWGdT58U30a dZbBvt42DWiflNQfvBs/QOYBhwdVv208x90hA9WduliZvK+yuCCR0H4gJi+p22XUvbG9 6vwcIe/Pxx7JQY/2SyS6oZFQWus33cotDiO9Q2OB/AxCJ7UNNt1EnxMujwDc3AQXgbZO WbFjFOSyD8u+2VDf9gBst/cvinr+JEXnKe5+/AM/81WfdLRojyN0ViNjhLTfVDOetiyA 3maQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=GCaGOZkt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id rj14-20020a17090b3e8e00b002496b783923si11098340pjb.181.2023.05.08.14.50.55; Mon, 08 May 2023 14:51:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=GCaGOZkt; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234194AbjEHVgO (ORCPT + 99 others); Mon, 8 May 2023 17:36:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234126AbjEHVgH (ORCPT ); Mon, 8 May 2023 17:36:07 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C9C4793CF; Mon, 8 May 2023 14:35:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=qJ542JOkywuQy44rpDjAFBKOiAWpMNGNJLpO6nBwf7o=; b=GCaGOZktpNu5GOB/jf0mGRtijV mJjATlNWDd6SpBMT0GoDulcBW1hcIzOdTn5XhuUoWUS40W40EmvP/Ba+1cC5BlIeEHDbZMyU7r+Ts q4lxWlfhSq+/6Z4ZoBLEH5z07leAlubUAa6eCSlgEFGy+M1QlR4+cSigU7UpnV8K/b6pl5VZKiVcC PYdNZ+HURFsuY6dSKxl/mzaTAdmZKJRz4xV+ABE+v6sxMLoiA0OOsJd+/0ejC7U2Xpe+lqdLnjIrc MXNeUu36aaxP3qYNZLefq7wou8XtduYSxulv3eAYJqZMDaxELMQh+wpZ+hl5oh5atAB2nLfocRqRE iWjSG27A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pw8UY-00EW3c-6r; Mon, 08 May 2023 21:33:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 6903330612F; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2E3152482941A; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.448097252@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:52 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 1/9] seqlock/latch: Provide raw_read_seqcount_latch_retry() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765364382473538431?= X-GMAIL-MSGID: =?utf-8?q?1765364382473538431?= The read side of seqcount_latch consists of: do { seq = raw_read_seqcount_latch(&latch->seq); ... } while (read_seqcount_latch_retry(&latch->seq, seq)); which is asymmetric in the raw_ department, and sure enough, read_seqcount_latch_retry() includes (explicit) instrumentation where raw_read_seqcount_latch() does not. This inconsistency becomes a problem when trying to use it from noinstr code. As such, fix it by renaming and re-implementing raw_read_seqcount_latch_retry() without the instrumentation. Specifically the instrumentation in question is kcsan_atomic_next(0) in do___read_seqcount_retry(). Loosing this annotation is not a problem because raw_read_seqcount_latch() does not pass through kcsan_atomic_next(KCSAN_SEQLOCK_REGION_MAX). Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Thomas Gleixner Reviewed-by: Petr Mladek --- include/linux/rbtree_latch.h | 2 +- include/linux/seqlock.h | 15 ++++++++------- kernel/printk/printk.c | 2 +- kernel/time/sched_clock.c | 2 +- kernel/time/timekeeping.c | 4 ++-- 5 files changed, 13 insertions(+), 12 deletions(-) --- a/include/linux/rbtree_latch.h +++ b/include/linux/rbtree_latch.h @@ -206,7 +206,7 @@ latch_tree_find(void *key, struct latch_ do { seq = raw_read_seqcount_latch(&root->seq); node = __lt_find(key, root, seq & 1, ops->comp); - } while (read_seqcount_latch_retry(&root->seq, seq)); + } while (raw_read_seqcount_latch_retry(&root->seq, seq)); return node; } --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -671,9 +671,9 @@ typedef struct { * * Return: sequence counter raw value. Use the lowest bit as an index for * picking which data copy to read. The full counter must then be checked - * with read_seqcount_latch_retry(). + * with raw_read_seqcount_latch_retry(). */ -static inline unsigned raw_read_seqcount_latch(const seqcount_latch_t *s) +static __always_inline unsigned raw_read_seqcount_latch(const seqcount_latch_t *s) { /* * Pairs with the first smp_wmb() in raw_write_seqcount_latch(). @@ -683,16 +683,17 @@ static inline unsigned raw_read_seqcount } /** - * read_seqcount_latch_retry() - end a seqcount_latch_t read section + * raw_read_seqcount_latch_retry() - end a seqcount_latch_t read section * @s: Pointer to seqcount_latch_t * @start: count, from raw_read_seqcount_latch() * * Return: true if a read section retry is required, else false */ -static inline int -read_seqcount_latch_retry(const seqcount_latch_t *s, unsigned start) +static __always_inline int +raw_read_seqcount_latch_retry(const seqcount_latch_t *s, unsigned start) { - return read_seqcount_retry(&s->seqcount, start); + smp_rmb(); + return unlikely(READ_ONCE(s->seqcount.sequence) != start); } /** @@ -752,7 +753,7 @@ read_seqcount_latch_retry(const seqcount * entry = data_query(latch->data[idx], ...); * * // This includes needed smp_rmb() - * } while (read_seqcount_latch_retry(&latch->seq, seq)); + * } while (raw_read_seqcount_latch_retry(&latch->seq, seq)); * * return entry; * } --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -597,7 +597,7 @@ static u64 latched_seq_read_nolock(struc seq = raw_read_seqcount_latch(&ls->latch); idx = seq & 0x1; val = ls->val[idx]; - } while (read_seqcount_latch_retry(&ls->latch, seq)); + } while (raw_read_seqcount_latch_retry(&ls->latch, seq)); return val; } --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -77,7 +77,7 @@ notrace struct clock_read_data *sched_cl notrace int sched_clock_read_retry(unsigned int seq) { - return read_seqcount_latch_retry(&cd.seq, seq); + return raw_read_seqcount_latch_retry(&cd.seq, seq); } unsigned long long notrace sched_clock(void) --- a/kernel/time/timekeeping.c +++ b/kernel/time/timekeeping.c @@ -450,7 +450,7 @@ static __always_inline u64 __ktime_get_f tkr = tkf->base + (seq & 0x01); now = ktime_to_ns(tkr->base); now += fast_tk_get_delta_ns(tkr); - } while (read_seqcount_latch_retry(&tkf->seq, seq)); + } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); return now; } @@ -566,7 +566,7 @@ static __always_inline u64 __ktime_get_r basem = ktime_to_ns(tkr->base); baser = ktime_to_ns(tkr->base_real); delta = fast_tk_get_delta_ns(tkr); - } while (read_seqcount_latch_retry(&tkf->seq, seq)); + } while (raw_read_seqcount_latch_retry(&tkf->seq, seq)); if (mono) *mono = basem + delta; From patchwork Mon May 8 21:19:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91294 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2453846vqo; Mon, 8 May 2023 14:57:50 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6AtDDI1TR/Enof4aX7e8zQpylgSYI2zPXEdkYALJTqAejpf6q/+/crKRp7lTnaz+xRLM64 X-Received: by 2002:a05:6a20:3d93:b0:ef:acca:9e19 with SMTP id s19-20020a056a203d9300b000efacca9e19mr16456884pzi.14.1683583070410; Mon, 08 May 2023 14:57:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683583070; cv=none; d=google.com; s=arc-20160816; b=PiF6kuD4QslwLkM4wQbGQxFDRgCxeWt+FJf2XIrfOc526PdYCqp36HZMyD3BXsesEN Zlg4/B5B5JeOKwHk4kFMNE+xnCgXrtD01cYXRRB4Ob3u8fMqWPIYzqizcjgJINVSqJhk 3UWIGgjJpCG6CRq7+Qmn/zLe99Cc9QPNSxzRW61X8c9Dbhvymno7mzFJ7vvwbD+JaBIz RCN3RMlVig2MJtztT9IY9PrCL2gJx7l+J5t9OBDdZcq+J8wOCb2m3/mk/6E+TGYbqbcp Ywa5ygRhLJY5TF0mLyh3ROLyrPbQrriIsKlzJ5aTSHvgUL6t88+dnYm+MfqP3F1MV3Np 9Xwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=y/NZkFjA2o+5kqY9Uzuitbn0zBukaZRP/fG6yHo/Mzw=; b=VnKBzy8hgH4+O0rUDUS+ScVmRo9YCqW8RvohLpkBtoQANIQ15aWD04I2TJpvbw8Qw0 wW50Rs2fja8MYXpE9YMwfajrI/HU/wfF0QHXbDc5U0hFI9eePCdCOWe+lgxFF90oOW9h Z4RaC1wH6fNyIu2hHbTe9ILUGMyaBqG2B3S1BLP9M2lixi1hJG/fb2TJulz+uzrPNvxy /P6HSS6E/YANfUsKN8v0OvhbmUOQULQz6SHnSAufK/pBlssXowqfWGkCI45cnyW0XHh8 Fi9iVgCtRMW0FCsK/mo+nljngHhw0LpzSOAW49BQOJoMz8P0xbs4foJ5VGY4p+Lf5+QH HvIQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=DR37JdMJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v4-20020a1709029a0400b001a6d4eab490si8631006plp.63.2023.05.08.14.57.37; Mon, 08 May 2023 14:57:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=DR37JdMJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233958AbjEHVfF (ORCPT + 99 others); Mon, 8 May 2023 17:35:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229486AbjEHVe5 (ORCPT ); Mon, 8 May 2023 17:34:57 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44209DE; Mon, 8 May 2023 14:34:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=y/NZkFjA2o+5kqY9Uzuitbn0zBukaZRP/fG6yHo/Mzw=; b=DR37JdMJecYKE8vH9MPmtZRs5A XWITmzNs1sn56eCXZ1sFUJPLFtuVOFvWlWkl3ZFUyDy0UY2QbuQQStdtz6WpGq0WKb/CpD83ravJF cAhzD4bfbpVZkKvODjERzxd8VjoOdK6VLJdKZb4Gq7oPEknGH875+hTYt4dBj/EmOp5mJ+6UQMdLJ HlSO+eI8jz7OO0U5xrrXvNYj4hxiqSA+4cYFFydfnPMPPtjkdnmHNghYo8gfCor4l3FWRl9d9WwS3 BlPVYt3Z7qeJ2amRuFeQdKimGkygzER0E2GaTGbJjmTXqLL5SD0Q59vFCBFYZ1QrdQTlLpLfLyUS2 XmDPsofA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pw8UY-00EW3i-Pu; Mon, 08 May 2023 21:33:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 66F03302188; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 320C42482941C; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.515668092@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:53 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 2/9] time/sched_clock: Provide sched_clock_noinstr() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765364801897804928?= X-GMAIL-MSGID: =?utf-8?q?1765364801897804928?= With the intent to provide local_clock_noinstr(), a variant of local_clock() that's safe to be called from noinstr code (with the assumption that any such code will already be non-preemptible), prepare for things by providing a noinstr sched_clock_noinstr() function. Specifically, preempt_enable_*() calls out to schedule(), which upsets noinstr validation efforts. As such, pull out the preempt_{dis,en}able_notrace() requirements from the sched_clock_read() implementations by explicitly providing it in the sched_clock() function. This further requires said sched_clock_read() functions to be noinstr themselves, for ARCH_WANTS_NO_INSTR users. See the next few patches. Signed-off-by: Peter Zijlstra (Intel) --- kernel/time/sched_clock.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -64,7 +64,7 @@ static struct clock_data cd ____cachelin .actual_read_sched_clock = jiffy_sched_clock_read, }; -static inline u64 notrace cyc_to_ns(u64 cyc, u32 mult, u32 shift) +static __always_inline u64 cyc_to_ns(u64 cyc, u32 mult, u32 shift) { return (cyc * mult) >> shift; } @@ -80,23 +80,33 @@ notrace int sched_clock_read_retry(unsig return raw_read_seqcount_latch_retry(&cd.seq, seq); } -unsigned long long notrace sched_clock(void) +unsigned long long noinstr sched_clock_noinstr(void) { - u64 cyc, res; - unsigned int seq; struct clock_read_data *rd; + unsigned int seq; + u64 cyc, res; do { - rd = sched_clock_read_begin(&seq); + seq = raw_read_seqcount_latch(&cd.seq); + rd = cd.read_data + (seq & 1); cyc = (rd->read_sched_clock() - rd->epoch_cyc) & rd->sched_clock_mask; res = rd->epoch_ns + cyc_to_ns(cyc, rd->mult, rd->shift); - } while (sched_clock_read_retry(seq)); + } while (raw_read_seqcount_latch_retry(&cd.seq, seq)); return res; } +unsigned long long notrace sched_clock(void) +{ + unsigned long long ns; + preempt_disable_notrace(); + ns = sched_clock_noinstr(); + preempt_enable_notrace(); + return ns; +} + /* * Updating the data required to read the clock. * From patchwork Mon May 8 21:19:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91301 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2456474vqo; Mon, 8 May 2023 15:02:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6vYRKx/CL5SUxAolPA8gTwF108D2WPs7ACz9C0KS3Lv6eQrANQW0u9ZQlPe3/WyArUG+F3 X-Received: by 2002:a17:90b:190f:b0:24e:16ae:61ca with SMTP id mp15-20020a17090b190f00b0024e16ae61camr11697857pjb.34.1683583343936; Mon, 08 May 2023 15:02:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683583343; cv=none; d=google.com; s=arc-20160816; b=w0Yo5K6+uY5lUAxYGtKrjzsbtsL8EjIkqkWnGnsAIv4SNW2QkMdpfB40A8LXa4Ji8q kLpXmC8a9z87j8HHyRqb/pGHqJJT5+8efozmXU7Vjr2wu+cBwvcJDwOQdNLnGcCc9yhT X2omJiSNmu8p0TUUv9kVTvNyyGGusKK75jqp3XxzcyVki67/WA6HO09ejA9Iu0uyPraK JBP9mJIcLcVa192PgTzRTQFjsGPzeJE2uliMwHe+CVgQf86u//qGgf2ozhWsvs6OcJp0 BPKNuz8yZEMQCjB3bD5B5pH6jXN5AVCc2mufZI2exLIg3w0jF/h419JP2BGViWkYZFiG nu+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=bEROWOrEuHGtVGNiYWuINxVA4qSy8ulUpF6Cr1VtAOk=; b=iV8sgX6guv2YGKzbX+M0emy9+dSxOtMjh2E7D+eyoiTbFi+USPibtYn6R/Xp9QJJhE lP+5YSQr6daEfujU4CS1NQG6mQ1KW2H7wj+iG/xoc+Be6k4lbcgZ+RZBk3sWlVfJ/AuB X6d9hv2HCHqkpdYayY2hXOqvVoDPbw2mVPRYXHXuhaQz5nw2/+t2y4vlubJk0lg6X3LP d9i870wF8HU6lhzw3HWlv/O9gwoZbmhQ0F66kVKBlJc1Ije8vzeFSEGC+Ck7jaNm5sgd swF7ZE5BdeAkJr0Kiy2NBQZGETbqriI1zJQILIc/1B9+4mFv7V6RaPZ3/aJ4pbOHAThg MlWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Rw8MxGzM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q13-20020a17090aa00d00b002478d1e81c7si25284037pjp.174.2023.05.08.15.02.09; Mon, 08 May 2023 15:02:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Rw8MxGzM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233978AbjEHVfK (ORCPT + 99 others); Mon, 8 May 2023 17:35:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233238AbjEHVe7 (ORCPT ); Mon, 8 May 2023 17:34:59 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAF3C129; Mon, 8 May 2023 14:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=bEROWOrEuHGtVGNiYWuINxVA4qSy8ulUpF6Cr1VtAOk=; b=Rw8MxGzMcck/SmtNHPxJBTkUJ2 glIi2eVEv3SEs7dwl7ZznmvfLWL/+hcH6+HqoI/0OpRAKtLEwozOzJIP1ucdO77zKFOBYmn2HAeik Q9dVu9HEGaXeNwVh9YkItAwMOYPP5acMOrWwhH140v6BsV7TML3Gh6CG7Esu0OG6hyOvl9rB+1wb5 nA2pozVz0eUelrQmFssZEfx0yQZGfDWONRB5yt5jJnnXeSQ68+9CxDgZTz/0A9VMRaQiqImxK6CE9 BWcxxygJsv3c8ji0Cuzqxdn6Fvj3pVdTCleFQKnOhQuG3y4YayiE6ujN/Xz0m7h3YRbQcyTHwR5gI ptGi/ITg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pw8UZ-00EW3p-Dg; Mon, 08 May 2023 21:33:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 93D41306137; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 356B62482941D; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.583344579@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:54 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 3/9] arm64/io: Always inline all of __raw_{read,write}[bwlq]() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765365088263956009?= X-GMAIL-MSGID: =?utf-8?q?1765365088263956009?= The next patch will want to use __raw_readl() from a noinstr section and as such that needs to be marked __always_inline to avoid the compiler being a silly bugger. Turns out it already is, but its siblings are not. Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/io.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) --- a/arch/arm64/include/asm/io.h +++ b/arch/arm64/include/asm/io.h @@ -22,13 +22,13 @@ * Generic IO read/write. These perform native-endian accesses. */ #define __raw_writeb __raw_writeb -static inline void __raw_writeb(u8 val, volatile void __iomem *addr) +static __always_inline void __raw_writeb(u8 val, volatile void __iomem *addr) { asm volatile("strb %w0, [%1]" : : "rZ" (val), "r" (addr)); } #define __raw_writew __raw_writew -static inline void __raw_writew(u16 val, volatile void __iomem *addr) +static __always_inline void __raw_writew(u16 val, volatile void __iomem *addr) { asm volatile("strh %w0, [%1]" : : "rZ" (val), "r" (addr)); } @@ -40,13 +40,13 @@ static __always_inline void __raw_writel } #define __raw_writeq __raw_writeq -static inline void __raw_writeq(u64 val, volatile void __iomem *addr) +static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr) { asm volatile("str %x0, [%1]" : : "rZ" (val), "r" (addr)); } #define __raw_readb __raw_readb -static inline u8 __raw_readb(const volatile void __iomem *addr) +static __always_inline u8 __raw_readb(const volatile void __iomem *addr) { u8 val; asm volatile(ALTERNATIVE("ldrb %w0, [%1]", @@ -57,7 +57,7 @@ static inline u8 __raw_readb(const volat } #define __raw_readw __raw_readw -static inline u16 __raw_readw(const volatile void __iomem *addr) +static __always_inline u16 __raw_readw(const volatile void __iomem *addr) { u16 val; @@ -80,7 +80,7 @@ static __always_inline u32 __raw_readl(c } #define __raw_readq __raw_readq -static inline u64 __raw_readq(const volatile void __iomem *addr) +static __always_inline u64 __raw_readq(const volatile void __iomem *addr) { u64 val; asm volatile(ALTERNATIVE("ldr %0, [%1]", From patchwork Mon May 8 21:19:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91300 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2456406vqo; Mon, 8 May 2023 15:02:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4KoaSDwsGPz75zC2AM6Ckvl5aF/FTMLFtmTryoFtRaCJ3HtQQaGtCW9EW17MoKNSFqI0IX X-Received: by 2002:a05:6a00:a82:b0:646:e11c:f602 with SMTP id b2-20020a056a000a8200b00646e11cf602mr750894pfl.0.1683583338534; Mon, 08 May 2023 15:02:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683583338; cv=none; d=google.com; s=arc-20160816; b=RrXTA0y2WPAm5pQcrd8yFc34KaF2m1lb1+SqHP9jvqlFfLlmIfKoPnlUjTXRT3LxM/ a/dK3CcQluwyA8VxvPD4S/83kbXA7npOsol7+NHPnF0DveISUnumgLExbrKE+Hjjh+O5 /Obsf3MpPc1TvgX2MvNX00drAid7T1Vl9CI+71Rkkp7aKz1VzAGi+Vnrqg7agprwlHtR UtR1XETUI8pembTJNVTBvyx+QzHW+mDMpacKypzTcZUQ9xvjDJAbHg5nCM5e2KYLJB2y Oca0a6LJOnwJ40zKOwKv7Kr9LuHoQ/MPmIpaYHeRfSYvKild+/lnqsUxhCwtmue1gxrY l0Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=F/mEORQVn/+4FFZhPg8/YI2Izvfuw/wjd2XHaIHu63c=; b=xnmnhXkDpT6egnBwKLEg6GqosP7JbGYZ/3vzDC5AUou5KfV9J0V+TTHfX/fFzs1BHT eoSH46VO1eiUZPM5p8w0ysUB62JXwXlJ9ZiT6YZN+N5fgGhJZUC9ySfRTkz9tx0E3gh8 XlwN5tZ1a4cFzD24XDEAYpoOweNeGfZb9Uogh2lMlAhV58YrqwAd+lDVBtnlHNkNDjBD xFdl3pRRxlDwNifCCSxsBR759qB2G1SSmVppybaTao+HF0w92Eg5hUxo+oGDfica6tlm j97GMk4KDtSaxgxpu7j/vJG2f/gtZ+WVS1jqqVbMVu6gCcZTRm05ys0/TJGxxKaeXNWD m/hw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Pa03uAZP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x18-20020aa79ad2000000b006456144318asi825288pfp.302.2023.05.08.15.02.05; Mon, 08 May 2023 15:02:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Pa03uAZP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233989AbjEHVf5 (ORCPT + 99 others); Mon, 8 May 2023 17:35:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233883AbjEHVfx (ORCPT ); Mon, 8 May 2023 17:35:53 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42E5E9ECD; Mon, 8 May 2023 14:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=F/mEORQVn/+4FFZhPg8/YI2Izvfuw/wjd2XHaIHu63c=; b=Pa03uAZPkgmVJ1RZq0PF/mn8c1 HPEOlUYH9qYdyoyKnkKFpNHIQxWCxbxmPqtEcXtMCyC5MDbaWkfX0UrH0hkIb9CV8qOjWm6rgdprk niysJMBzcsXHUnMTk9nSR3egA9I3LQ1USG94wywxRwRJvHwmPKF6ayuB3A4oFDSj12zDQZmmUKHsH ttMq2ONLodkV45d9K7qsEUKCFt/NdTZ9JhtPbBg7T11iaTSKaVYre+oFuJfDMVLCbg7BB3nFk8wX6 hwrKf3Mf+048gPWaw5o1Eei5BCLQv2fvJHqreq6r/unM6mdvy4/TToL4svY6eeQDB7ymYDjRLs04F VZBXFB4g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pw8UY-00EW3b-6j; Mon, 08 May 2023 21:33:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 9F96C30613D; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 38F4D2482941E; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.650971590@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:55 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 4/9] arm64/arch_timer: Provide noinstr sched_clock_read() functions References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765365083117956780?= X-GMAIL-MSGID: =?utf-8?q?1765365083117956780?= With the intent to provide local_clock_noinstr(), a variant of local_clock() that's safe to be called from noinstr code (with the assumption that any such code will already be non-preemptible), prepare for things by providing a noinstr sched_clock_read() function. Specifically, preempt_enable_*() calls out to schedule(), which upsets noinstr validation efforts. Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/arch_timer.h | 8 ---- drivers/clocksource/arm_arch_timer.c | 60 ++++++++++++++++++++++++++--------- 2 files changed, 47 insertions(+), 21 deletions(-) --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -88,13 +88,7 @@ static inline notrace u64 arch_timer_rea #define arch_timer_reg_read_stable(reg) \ ({ \ - u64 _val; \ - \ - preempt_disable_notrace(); \ - _val = erratum_handler(read_ ## reg)(); \ - preempt_enable_notrace(); \ - \ - _val; \ + erratum_handler(read_ ## reg)(); \ }) /* --- a/drivers/clocksource/arm_arch_timer.c +++ b/drivers/clocksource/arm_arch_timer.c @@ -191,22 +191,40 @@ u32 arch_timer_reg_read(int access, enum return val; } -static notrace u64 arch_counter_get_cntpct_stable(void) +static noinstr u64 _arch_counter_get_cntpct_stable(void) { return __arch_counter_get_cntpct_stable(); } -static notrace u64 arch_counter_get_cntpct(void) +static notrace u64 arch_counter_get_cntpct_stable(void) +{ + u64 val; + preempt_disable_notrace(); + val = __arch_counter_get_cntpct_stable(); + preempt_enable_notrace(); + return val; +} + +static noinstr u64 arch_counter_get_cntpct(void) { return __arch_counter_get_cntpct(); } -static notrace u64 arch_counter_get_cntvct_stable(void) +static noinstr u64 _arch_counter_get_cntvct_stable(void) { return __arch_counter_get_cntvct_stable(); } -static notrace u64 arch_counter_get_cntvct(void) +static notrace u64 arch_counter_get_cntvct_stable(void) +{ + u64 val; + preempt_disable_notrace(); + val = __arch_counter_get_cntvct_stable(); + preempt_enable_notrace(); + return val; +} + +static noinstr u64 arch_counter_get_cntvct(void) { return __arch_counter_get_cntvct(); } @@ -753,14 +771,14 @@ static int arch_timer_set_next_event_phy return 0; } -static u64 arch_counter_get_cnt_mem(struct arch_timer *t, int offset_lo) +static noinstr u64 arch_counter_get_cnt_mem(struct arch_timer *t, int offset_lo) { u32 cnt_lo, cnt_hi, tmp_hi; do { - cnt_hi = readl_relaxed(t->base + offset_lo + 4); - cnt_lo = readl_relaxed(t->base + offset_lo); - tmp_hi = readl_relaxed(t->base + offset_lo + 4); + cnt_hi = __raw_readl(t->base + offset_lo + 4); + cnt_lo = __raw_readl(t->base + offset_lo); + tmp_hi = __raw_readl(t->base + offset_lo + 4); } while (cnt_hi != tmp_hi); return ((u64) cnt_hi << 32) | cnt_lo; @@ -1060,7 +1078,7 @@ bool arch_timer_evtstrm_available(void) return cpumask_test_cpu(raw_smp_processor_id(), &evtstrm_available); } -static u64 arch_counter_get_cntvct_mem(void) +static noinstr u64 arch_counter_get_cntvct_mem(void) { return arch_counter_get_cnt_mem(arch_timer_mem, CNTVCT_LO); } @@ -1074,6 +1092,13 @@ struct arch_timer_kvm_info *arch_timer_g static void __init arch_counter_register(unsigned type) { + /* + * Default to cp15 based access because arm64 uses this function for + * sched_clock() before DT is probed and the cp15 method is guaranteed + * to exist on arm64. arm doesn't use this before DT is probed so even + * if we don't have the cp15 accessors we won't have a problem. + */ + u64 (*scr)(void) = arch_counter_get_cntvct; u64 start_count; int width; @@ -1083,21 +1108,28 @@ static void __init arch_counter_register if ((IS_ENABLED(CONFIG_ARM64) && !is_hyp_mode_available()) || arch_timer_uses_ppi == ARCH_TIMER_VIRT_PPI) { - if (arch_timer_counter_has_wa()) + if (arch_timer_counter_has_wa()) { rd = arch_counter_get_cntvct_stable; - else + scr = _arch_counter_get_cntvct_stable; + } else { rd = arch_counter_get_cntvct; + scr = arch_counter_get_cntvct; + } } else { - if (arch_timer_counter_has_wa()) + if (arch_timer_counter_has_wa()) { rd = arch_counter_get_cntpct_stable; - else + scr = _arch_counter_get_cntpct_stable; + } else { rd = arch_counter_get_cntpct; + scr = arch_counter_get_cntpct; + } } arch_timer_read_counter = rd; clocksource_counter.vdso_clock_mode = vdso_default; } else { arch_timer_read_counter = arch_counter_get_cntvct_mem; + scr = arch_counter_get_cntvct_mem; } width = arch_counter_get_width(); @@ -1113,7 +1145,7 @@ static void __init arch_counter_register timecounter_init(&arch_timer_kvm_info.timecounter, &cyclecounter, start_count); - sched_clock_register(arch_timer_read_counter, width, arch_timer_rate); + sched_clock_register(scr, width, arch_timer_rate); } static void arch_timer_stop(struct clock_event_device *clk) From patchwork Mon May 8 21:19:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91289 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2446674vqo; Mon, 8 May 2023 14:40:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7VxdhiEsKAH+sEQfctA9/69I+8Jl7Iz9dQr7THgTya4aMQbV4t7CQdb6PnNtS1YuVMc+z2 X-Received: by 2002:a17:90a:31c6:b0:24d:ec21:b83c with SMTP id j6-20020a17090a31c600b0024dec21b83cmr11336456pjf.16.1683581999861; Mon, 08 May 2023 14:39:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683581999; cv=none; d=google.com; s=arc-20160816; b=DqSVCJT15QtX88Puq+ehkxNfkzfttGasU5HnpgfPUYW6iSmmMMbuaBbIxKFxPrwwSi N886y1IWlD0z+SFmUsoxmLqA4WfJLjlD/SStMQu2q6auszfc1NbDzADKhhAhX/LDtcQM Qc7i449zoaRFUYKYJ9q8+6Jy31U+r6i1rO3gn4UuaA58l+BIGykIyoEKQimDaie6/5X7 /sdlFT8HLAtqsB/GeJD5fJucCcRrzGLO9v3e004VKvRwjc3JuLLA+8DOjFVENAycYwAJ m+U68P06okPdyW4LIGiXW8F5wN+5SjQA2whtBiGpQPE4stL6lEKyWjRVgSnGgPEz0lGD U7MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=YB2UOIFIw2gwCNTVh1j30XHP2RZzZzmzkIRobxbOGl0=; b=RIGfMgSEF6Vl+qWpzvTg+nC3B8pXS9CAbXK/78lMiL+Y6HSb0ITXvwyHuJ+O68tZBY jH+jn8nY05zONPEbqciIhRXRoR0XaZr19L6e+NtuZB0mYc1M/HiE5OxzSRuZEGHaA49Y 7N7shTKdzTx+xgjDdqLGns6sN9mvruhPvky36+4p5E4CDjnlP4/Mn/TYiva4SCU/CsF+ lZ6Gix4JQ2qWjDtkgAsN7B07DhqQuCL6bfAXiZTbPxs93uYyVC3lmdctp5KEFOZcYGj4 xA7zZ6G+urwUDUu3GkIB8WKJAd023UFTyjijXVB6WkB0QAwRocXLgpN7AjYFxWyYoE/C w7Ow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=M2iOJBjK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u64-20020a17090a51c600b0024e2d70d28asi15705685pjh.140.2023.05.08.14.39.43; Mon, 08 May 2023 14:39:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=M2iOJBjK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229865AbjEHVhI (ORCPT + 99 others); Mon, 8 May 2023 17:37:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234248AbjEHVgt (ORCPT ); Mon, 8 May 2023 17:36:49 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0906746A1; Mon, 8 May 2023 14:36:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=YB2UOIFIw2gwCNTVh1j30XHP2RZzZzmzkIRobxbOGl0=; b=M2iOJBjKCa79Kv9g3tjWPVJtfy KhgiZrTa14XHFuWNlEPcmixkV04+wlzqZJejkStuJj9v7/L8doRfz+dSC1Hh4X+EV63Q7yEPMtx3l VvKERwRhkB0Pue0edXHPZLIi7dBrgCMV5plSt4V61wT3O4pNq41WMoOdff8iW+QCXNa4J6i4gdDkH 3Op4X0qcSAppyJHS8/G/TRWwuJCq8ntWBm8EmEJdXZH5sXXGJTSYslQP651uTLAomLG6te6RB3kEf fQGbSryGVWIyKlWQBoAiJhJ7ytQTG0orrgOpToKdgeO4JFZc/HJEIrEESutvAqjUogdgXVL1c75a3 ZUxEaSrA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pw8UZ-005J1E-1B; Mon, 08 May 2023 21:34:15 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id A98C630613F; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3EBFA24829429; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.718633831@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:56 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 5/9] loongarch: Provide noinstr sched_clock_read() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765363679020521680?= X-GMAIL-MSGID: =?utf-8?q?1765363679020521680?= With the intent to provide local_clock_noinstr(), a variant of local_clock() that's safe to be called from noinstr code (with the assumption that any such code will already be non-preemptible), prepare for things by providing a noinstr sched_clock_read() function. Specifically, preempt_enable_*() calls out to schedule(), which upsets noinstr validation efforts. Signed-off-by: Peter Zijlstra (Intel) --- arch/loongarch/include/asm/loongarch.h | 2 +- arch/loongarch/kernel/time.c | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) --- a/arch/loongarch/include/asm/loongarch.h +++ b/arch/loongarch/include/asm/loongarch.h @@ -1167,7 +1167,7 @@ static __always_inline void iocsr_write6 #ifndef __ASSEMBLY__ -static inline u64 drdtime(void) +static __always_inline u64 drdtime(void) { int rID = 0; u64 val = 0; --- a/arch/loongarch/kernel/time.c +++ b/arch/loongarch/kernel/time.c @@ -190,9 +190,9 @@ static u64 read_const_counter(struct clo return drdtime(); } -static u64 native_sched_clock(void) +static noinstr u64 sched_clock_read(void) { - return read_const_counter(NULL); + return drdtime(); } static struct clocksource clocksource_const = { @@ -211,7 +211,7 @@ int __init constant_clocksource_init(voi res = clocksource_register_hz(&clocksource_const, freq); - sched_clock_register(native_sched_clock, 64, freq); + sched_clock_register(sched_clock_read, 64, freq); pr_info("Constant clock source device register\n"); From patchwork Mon May 8 21:19:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2455390vqo; Mon, 8 May 2023 15:00:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5mSUuuOfJP9s7W3nTFMBCxk5OTM/z7cKXDWXeku68MdaePfdKGiBVCWKTifinBSnp7rVS1 X-Received: by 2002:a17:90b:3757:b0:247:afed:6d62 with SMTP id ne23-20020a17090b375700b00247afed6d62mr11580143pjb.46.1683583255983; Mon, 08 May 2023 15:00:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683583255; cv=none; d=google.com; s=arc-20160816; b=BsbKe17YQaig28leHvS74P3GiiM6DKpldjMxHsM6zR78L8MY56DfF/ZnRGv+25er61 PKvRLT4oAKUbRvpc+Di9TSv1SgtoVmcspkE+68d0QoefpStT9j8XlfC0zf11gt/UqLcY VvbCw4d0LlI/+4VXx/fZffwModsztuppG0pSxTsuOuXMCcA2AKV74crJpXyjSNOaecwy qvp9HHG+0b8gCNhAlDf00bcLvBYMhYvfKndDU8D6kjaFKRTa6EPq/nzaY3bsrZG1CHix MGCCEsRQD7vbUoKxoNNRGs+bpr771ngkNFc37TIM0sO6aiXZ/vg/wlzLFZ/YrpMBNLWg 0d2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=VYuNexE2e9xbspcsGvZenlIDjWfgw2N2vaQA4HRdjvE=; b=nie50eL5luVdyc1wNqOA/53lixUljYJfwjyG6tuAI2vd+T+8u6enwlOyiOcXAYlSQt CyDyjWktDIIGGuFTsLXio5LPY3VvQYoGV3QNop+KRd0jrcTn2ZGVTa/QyfgFA8XIL5lR x+X5si8+E3DKoY32yjoUsfaufdIBr+J0m4RjOOUAsDlccnRmzwScKTPirazMeIJkSw/M 77A/J1phzjiPmMEubZkH3yVvbEGrWPZpeSUyVcY3aYcpBuW9OlEZucB3XMftnDaVkqnq e5l1julxWQFhfIlWGKhV8sr3gktbv90K7MoyWMTsgHTG4mVgxIaooRT6hgRYUDntP8tN j8lg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=PHoZYCvv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f13-20020a63754d000000b0052c6f258152si9629825pgn.257.2023.05.08.15.00.42; Mon, 08 May 2023 15:00:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=PHoZYCvv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233354AbjEHVg4 (ORCPT + 99 others); Mon, 8 May 2023 17:36:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234181AbjEHVgj (ORCPT ); Mon, 8 May 2023 17:36:39 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A90C59ED0; Mon, 8 May 2023 14:36:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=VYuNexE2e9xbspcsGvZenlIDjWfgw2N2vaQA4HRdjvE=; b=PHoZYCvvmT1hG95taohLWS5MV8 r4yrd4PaEq7f4WAmF+OTmLvnxZVSfMakDad5wTPn12JXCXWk2GLwT8/dDR82lB8BOQgFZxdza3qHL vmxpyGPlEnNdGUt4eFmlC32sPr4sJFcHvhAD2uTDLW/dYr36FMZqntEuhrQa0hheIe5LqniryDfKw pwnQ8F7RbF1IuPFFlYQv1mEDAd6PbOe1gq/ETRL7d3PmsKqIw4GTCGhKk+RvisoI9lVQLhXJf7NVE XQyDGswjUQDEGh3rXK4Pkdl/fMoSt3pHhzKUVr0swTsjJO5WhI208u84das+E8Ia8JtL5ygbTPqG8 kTf/vQsQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pw8UY-005J1D-2b; Mon, 08 May 2023 21:34:15 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id B286C306141; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4178324829420; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.786238095@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:57 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 6/9] s390/time: Provide sched_clock_noinstr() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765364995991222921?= X-GMAIL-MSGID: =?utf-8?q?1765364995991222921?= With the intent to provide local_clock_noinstr(), a variant of local_clock() that's safe to be called from noinstr code (with the assumption that any such code will already be non-preemptible), prepare for things by providing a noinstr sched_clock_noinstr() function. Specifically, preempt_enable_*() calls out to schedule(), which upsets noinstr validation efforts. Signed-off-by: Peter Zijlstra (Intel) --- arch/s390/include/asm/timex.h | 13 +++++++++---- arch/s390/kernel/time.c | 11 ++++++++++- 2 files changed, 19 insertions(+), 5 deletions(-) --- a/arch/s390/include/asm/timex.h +++ b/arch/s390/include/asm/timex.h @@ -63,7 +63,7 @@ static inline int store_tod_clock_ext_cc return cc; } -static inline void store_tod_clock_ext(union tod_clock *tod) +static __always_inline void store_tod_clock_ext(union tod_clock *tod) { asm volatile("stcke %0" : "=Q" (*tod) : : "cc"); } @@ -177,7 +177,7 @@ static inline void local_tick_enable(uns typedef unsigned long cycles_t; -static inline unsigned long get_tod_clock(void) +static __always_inline unsigned long get_tod_clock(void) { union tod_clock clk; @@ -204,6 +204,11 @@ void init_cpu_timer(void); extern union tod_clock tod_clock_base; +static __always_inline unsigned long __get_tod_clock_monotonic(void) +{ + return get_tod_clock() - tod_clock_base.tod; +} + /** * get_clock_monotonic - returns current time in clock rate units * @@ -216,7 +221,7 @@ static inline unsigned long get_tod_cloc unsigned long tod; preempt_disable_notrace(); - tod = get_tod_clock() - tod_clock_base.tod; + tod = __get_tod_clock_monotonic(); preempt_enable_notrace(); return tod; } @@ -240,7 +245,7 @@ static inline unsigned long get_tod_cloc * -> ns = (th * 125) + ((tl * 125) >> 9); * */ -static inline unsigned long tod_to_ns(unsigned long todval) +static __always_inline unsigned long tod_to_ns(unsigned long todval) { return ((todval >> 9) * 125) + (((todval & 0x1ff) * 125) >> 9); } --- a/arch/s390/kernel/time.c +++ b/arch/s390/kernel/time.c @@ -102,12 +102,21 @@ void __init time_early_init(void) ((long) qui.old_leap * 4096000000L); } +unsigned long long noinstr sched_clock_noinstr(void) +{ + return tod_to_ns(__get_tod_clock_monotonic()); +} + /* * Scheduler clock - returns current time in nanosec units. */ unsigned long long notrace sched_clock(void) { - return tod_to_ns(get_tod_clock_monotonic()); + unsigned long long ns; + preempt_disable_notrace(); + ns = tod_to_ns(get_tod_clock_monotonic()); + preempt_enable_notrace(); + return ns; } NOKPROBE_SYMBOL(sched_clock); From patchwork Mon May 8 21:19:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91292 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2452690vqo; Mon, 8 May 2023 14:54:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ59zTQ4jJOAxQ6bAW/aTVAb/lipSDD/NrahjvN5nEB+xtiM8aV02dOyeLTUKj61YWvma9D1 X-Received: by 2002:a17:902:db01:b0:1a6:fe25:4138 with SMTP id m1-20020a170902db0100b001a6fe254138mr13942556plx.59.1683582880652; Mon, 08 May 2023 14:54:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683582880; cv=none; d=google.com; s=arc-20160816; b=QMNglLcEjEG8TkNNRBa/FOg3b0h1G99++Dj2dJyQvK4Ie2a2/4G4h2Jq5R8ZXoLxw/ aHY1UZckwYOgoE2RzTjsGE23AbxwAu3hLM5wBBIDs+L/T6MuAUVFmZvVt8zHD2RuY0xm toOZf3j3PBhLQnIh15OeIeNNzAYQxc2G0S6PQEsysCBdL8MJ0l75Y8QTvsEmFXAj5HzD kXZXZjBfzgjujuaiM1dk207JIRcaBYCQNHHGDBEMYJlHvGOtCNZcnXaVIfRXwABSSvDx X8VDvhay5LJ4A4miwajLj9SabkyLF5tRXLjEXLaiNTZ9bTuRDD0tc9/dG+85kwk8OHEk VgYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=33CfYuIrVYZl2aHKgRGfmqvRmjmw9yx5a1cynu5QEgI=; b=X8xxIg7+pi5C/zU3NgtQlXda1Kq2wjufF78NjLmuunTSvKhQg7wtDtsKell1olNATn +hVDUwDi63BhkNdPRvLtblv7QvkRh5VKR13KduwBcvvOnF6+yuecgnmltuRaGnociQHS 0rRWnNSvq1XMogml4irxr1uG/z4KRSTFSq8J7XNDO1QqKqrCbaIu4NrTV+ZckVvq7OS7 rHvJxBVxyqa7f99SsoN3O9niEboZVE1c8lXQ8NdKIMWnF9/EjVRlS5kp1vT0DF+wNdsX 0WWyqeW1wDK9mMOA39Xlk2uyh7mDDdtbDdsRFFKsICMNGHXYjInBjVzquLHSqCfEa5QS eZtQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="U9/HqRh7"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id iw20-20020a170903045400b001a63b8e071esi8750412plb.28.2023.05.08.14.54.28; Mon, 08 May 2023 14:54:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="U9/HqRh7"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233577AbjEHVhB (ORCPT + 99 others); Mon, 8 May 2023 17:37:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234197AbjEHVgj (ORCPT ); Mon, 8 May 2023 17:36:39 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54CC893DD; Mon, 8 May 2023 14:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=33CfYuIrVYZl2aHKgRGfmqvRmjmw9yx5a1cynu5QEgI=; b=U9/HqRh7NJpCcAxxxSP9FV7Eie MEdu0Gh7Nzrq0ve2MN/sx6wfa0QOt4/gbjQrkl78uzjuzEIZsWWix+2VhgAgc1XjOprqwXwMF01cF bLy0VwWAuIYQH6zrX1IwaTGgccbW97pHN72DaYLsAS1P93TRGkzZyg7AICjrNkpnkvVR+karqqZ9z EOU9eRirRrG0WkvMJIp7RSZF/d/ELAc57IFffCvMubgAdeMB3uH+EakU1wrGrNQX4lQKtAPJhAKl1 8G3WeOtMnc7DecO1M2xS9WheTs8e4srdZTcoiTuoXxr1WrBE8VJ8Os1qKOwN6/vHmgPXq28TUTDZ0 zfjhfmvQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pw8UY-005J1B-0f; Mon, 08 May 2023 21:34:15 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id B9CCD306144; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 48C572482942B; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.853677542@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:58 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 7/9] x86/tsc: Provide sched_clock_noinstr() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765364602546501214?= X-GMAIL-MSGID: =?utf-8?q?1765364602546501214?= With the intent to provide local_clock_noinstr(), a variant of local_clock() that's safe to be called from noinstr code (with the assumption that any such code will already be non-preemptible), prepare for things by providing a noinstr sched_clock_noinstr() function. Specifically, preempt_enable_*() calls out to schedule(), which upsets noinstr validation efforts. vmlinux.o: warning: objtool: native_sched_clock+0x96: call to preempt_schedule_notrace_thunk() leaves .noinstr.text section vmlinux.o: warning: objtool: kvm_clock_read+0x22: call to preempt_schedule_notrace_thunk() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/kernel/kvmclock.c | 4 +-- arch/x86/kernel/tsc.c | 38 ++++++++++++++++++++++++++++--------- arch/x86/xen/time.c | 3 -- drivers/clocksource/hyperv_timer.c | 4 +-- include/clocksource/hyperv_timer.h | 4 +-- 5 files changed, 36 insertions(+), 17 deletions(-) --- a/arch/x86/kernel/kvmclock.c +++ b/arch/x86/kernel/kvmclock.c @@ -71,7 +71,7 @@ static int kvm_set_wallclock(const struc return -ENODEV; } -static noinstr u64 kvm_clock_read(void) +static u64 kvm_clock_read(void) { u64 ret; @@ -88,7 +88,7 @@ static u64 kvm_clock_get_cycles(struct c static noinstr u64 kvm_sched_clock_read(void) { - return kvm_clock_read() - kvm_sched_clock_offset; + return pvclock_clocksource_read_nowd(this_cpu_pvti()) - kvm_sched_clock_offset; } static inline void kvm_sched_clock_init(bool stable) --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -69,12 +69,10 @@ static int __init tsc_early_khz_setup(ch } early_param("tsc_early_khz", tsc_early_khz_setup); -__always_inline void cyc2ns_read_begin(struct cyc2ns_data *data) +__always_inline void __cyc2ns_read(struct cyc2ns_data *data) { int seq, idx; - preempt_disable_notrace(); - do { seq = this_cpu_read(cyc2ns.seq.seqcount.sequence); idx = seq & 1; @@ -86,6 +84,12 @@ __always_inline void cyc2ns_read_begin(s } while (unlikely(seq != this_cpu_read(cyc2ns.seq.seqcount.sequence))); } +__always_inline void cyc2ns_read_begin(struct cyc2ns_data *data) +{ + preempt_disable_notrace(); + __cyc2ns_read(data); +} + __always_inline void cyc2ns_read_end(void) { preempt_enable_notrace(); @@ -115,18 +119,25 @@ __always_inline void cyc2ns_read_end(voi * -johnstul@us.ibm.com "math is hard, lets go shopping!" */ -static __always_inline unsigned long long cycles_2_ns(unsigned long long cyc) +static __always_inline unsigned long long __cycles_2_ns(unsigned long long cyc) { struct cyc2ns_data data; unsigned long long ns; - cyc2ns_read_begin(&data); + __cyc2ns_read(&data); ns = data.cyc2ns_offset; ns += mul_u64_u32_shr(cyc, data.cyc2ns_mul, data.cyc2ns_shift); - cyc2ns_read_end(); + return ns; +} +static __always_inline unsigned long long cycles_2_ns(unsigned long long cyc) +{ + unsigned long long ns; + preempt_disable_notrace(); + ns = __cycles_2_ns(cyc); + preempt_enable_notrace(); return ns; } @@ -223,7 +234,7 @@ noinstr u64 native_sched_clock(void) u64 tsc_now = rdtsc(); /* return the value in ns */ - return cycles_2_ns(tsc_now); + return __cycles_2_ns(tsc_now); } /* @@ -250,7 +261,7 @@ u64 native_sched_clock_from_tsc(u64 tsc) /* We need to define a real function for sched_clock, to override the weak default version */ #ifdef CONFIG_PARAVIRT -noinstr u64 sched_clock(void) +noinstr u64 sched_clock_noinstr(void) { return paravirt_sched_clock(); } @@ -260,11 +271,20 @@ bool using_native_sched_clock(void) return static_call_query(pv_sched_clock) == native_sched_clock; } #else -u64 sched_clock(void) __attribute__((alias("native_sched_clock"))); +u64 sched_clock_noinstr(void) __attribute__((alias("native_sched_clock"))); bool using_native_sched_clock(void) { return true; } #endif +notrace u64 sched_clock(void) +{ + u64 now; + preempt_disable_notrace(); + now = sched_clock_noinstr(); + preempt_enable_notrace(); + return now; +} + int check_tsc_unstable(void) { return tsc_unstable; --- a/arch/x86/xen/time.c +++ b/arch/x86/xen/time.c @@ -66,11 +66,10 @@ static noinstr u64 xen_sched_clock(void) struct pvclock_vcpu_time_info *src; u64 ret; - preempt_disable_notrace(); src = &__this_cpu_read(xen_vcpu)->time; ret = pvclock_clocksource_read_nowd(src); ret -= xen_sched_clock_offset; - preempt_enable_notrace(); + return ret; } --- a/drivers/clocksource/hyperv_timer.c +++ b/drivers/clocksource/hyperv_timer.c @@ -408,9 +408,9 @@ static u64 notrace read_hv_clock_tsc_cs( return read_hv_clock_tsc(); } -static u64 notrace read_hv_sched_clock_tsc(void) +static u64 noinstr read_hv_sched_clock_tsc(void) { - return (read_hv_clock_tsc() - hv_sched_clock_offset) * + return (hv_read_tsc_page(hv_get_tsc_page()) - hv_sched_clock_offset) * (NSEC_PER_SEC / HV_CLOCK_HZ); } --- a/include/clocksource/hyperv_timer.h +++ b/include/clocksource/hyperv_timer.h @@ -38,7 +38,7 @@ extern void hv_remap_tsc_clocksource(voi extern unsigned long hv_get_tsc_pfn(void); extern struct ms_hyperv_tsc_page *hv_get_tsc_page(void); -static inline notrace u64 +static __always_inline notrace u64 hv_read_tsc_page_tsc(const struct ms_hyperv_tsc_page *tsc_pg, u64 *cur_tsc) { u64 scale, offset; @@ -85,7 +85,7 @@ hv_read_tsc_page_tsc(const struct ms_hyp return mul_u64_u64_shr(*cur_tsc, scale, 64) + offset; } -static inline notrace u64 +static __always_inline notrace u64 hv_read_tsc_page(const struct ms_hyperv_tsc_page *tsc_pg) { u64 cur_tsc; From patchwork Mon May 8 21:19:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91287 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2445619vqo; Mon, 8 May 2023 14:37:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5FMtuNIjWbAbxOFVPi3rL9YJdUcEx+bvdmPSkYwSR/5HRngQLafdXcxPLSJ6/JVNRkwDlI X-Received: by 2002:a17:902:d48d:b0:1ac:66c4:6071 with SMTP id c13-20020a170902d48d00b001ac66c46071mr9185436plg.57.1683581854707; Mon, 08 May 2023 14:37:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683581854; cv=none; d=google.com; s=arc-20160816; b=BtSPrEjVyTqsrD0Ch4hCkw/OISWK7Pt0VzQ/pUFjaG4TXQbzL3iwYhLOu7pMQT+z3s QUJ19d2v1KoGmv+STY1/VzyqjvsPLa1u9CKwjVQYdIgXdmlQ8luJG7Bm7jVXJ5CMvPmD x81gfon150W9mDNpDLg3Dy3wr/t7k1AENykmHKZwzTI8ZkRVybcWTfhQg2AeaQzqDYHP Bh9LqVKJ4dWTPxh2iWVpvtDK8v9ic/C4JWDo1GUGS+fbGX/2NRwxg0ffdFC/3I8XMOzI +FwrTSIgQtj48QTmY4FbbmSW/lWRf2XLbTYpgQJHvQRCKBQqkgI79khWQaAuXOnzVnW2 Onmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=Er0HJF5+HBoF3muAv/w3BVVkES/scqceLlbLLVy6xkA=; b=EN/9Bdxl9XbgTCA5T1/nSsi31mIKU0gF1d4j+wCdkvDeJ+IoavsPTeD2FIiM9ohDLA h0VyQgWEtWI0W6Ed0D4Gl0hqaYPvfGrpFw4kNruZ3kvcyLMQ36+yXNEUAmPoZmAhZtjA xR7bBVbLnd/op0i3jpsZkenWkJhUd7HAESUVO/JYp9ksBLZxHfPIW1u/dByp3EXUZATz XNPBKttf7Jwv6/1KG8QdszY0XoKFkly0gcHd8Ara48VTKiAdF8+z+BklBCRGJ+2HaCO/ NRqbv56cd7qWIz90MoEFyhVXHvabjXtA6NGJrJx1SPxIhlxvGebyncvWdl+l4u0mTys/ X98w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=AHrbBohx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m7-20020a170902db0700b001ac7ea55768si2938239plx.499.2023.05.08.14.37.21; Mon, 08 May 2023 14:37:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=AHrbBohx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233953AbjEHVfz (ORCPT + 99 others); Mon, 8 May 2023 17:35:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233787AbjEHVfv (ORCPT ); Mon, 8 May 2023 17:35:51 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8A2F8A71; Mon, 8 May 2023 14:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Er0HJF5+HBoF3muAv/w3BVVkES/scqceLlbLLVy6xkA=; b=AHrbBohxk9L7amjAySd/QLMFLx yN6fwaHqY0dSmAu2KjyhrpDAmMtsXE6E9JnwKZQax/WoFSCs39hsHw6U0DKLpxl6GmqYXZxMKqtEl 4vW4iyFqQHzFw2NS21TBa4VRbWTk/dzekZerer8CmS483RFpKYuOP+2SfYY0JTPJYF70f15XtGmSf OOX1ka8ZG2DYbzd/nZfhPDZriXKwFll1cVMYedMw4WnusF68aV3bGnDP4Y8Ub7+iA+Kl5kd50OrmO lmX1CXDbNf/fFyCIIg2K1q+I8RU2Ss+U/Z/V65UrFQNep8sbZ5IiWX5ZHqeK6LOcYJ0HDTyRjVUuz FYyDIqrw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pw8UY-00EW3j-QC; Mon, 08 May 2023 21:33:58 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id C29BD306148; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4B64F2482942A; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.922038236@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:19:59 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 8/9] sched/clock: Provide local_clock_noinstr() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765363526452035858?= X-GMAIL-MSGID: =?utf-8?q?1765363526452035858?= Now that all ARCH_WANTS_NO_INSTR architectures (arm64, loongarch, s390, x86) provide sched_clock_noinstr(), use this to provide local_clock_noinstr(). This local_clock_noinstr() will be safe to use from noinstr code with the assumption that any such noinstr code is non-preemptible (it had better be, entry code will have IRQs disabled while __cpuidle must have preemption disabled). Specifically, preempt_enable_notrace(), a common part of many a sched_clock() implementation calls out to schedule() -- even though, per the above, it will never trigger -- which frustrates noinstr validation. vmlinux.o: warning: objtool: local_clock+0xb5: call to preempt_schedule_notrace_thunk() leaves .noinstr.text section Signed-off-by: Peter Zijlstra (Intel) --- include/linux/sched/clock.h | 17 ++++++++++++++++- kernel/sched/clock.c | 19 +++++++++++++------ 2 files changed, 29 insertions(+), 7 deletions(-) --- a/include/linux/sched/clock.h +++ b/include/linux/sched/clock.h @@ -12,7 +12,16 @@ * * Please use one of the three interfaces below. */ -extern unsigned long long notrace sched_clock(void); +extern u64 sched_clock(void); + +#if defined(CONFIG_ARCH_WANTS_NO_INSTR) || defined(CONFIG_GENERIC_SCHED_CLOCK) +extern u64 sched_clock_noinstr(void); +#else +__always_inline u64 sched_clock_noinstr(void) +{ + return sched_clock(); +} +#endif /* * See the comment in kernel/sched/clock.c @@ -45,6 +54,11 @@ static inline u64 cpu_clock(int cpu) return sched_clock(); } +static __always_inline u64 local_clock_noinstr(void) +{ + return sched_clock_noinstr(); +} + static __always_inline u64 local_clock(void) { return sched_clock(); @@ -79,6 +93,7 @@ static inline u64 cpu_clock(int cpu) return sched_clock_cpu(cpu); } +extern u64 local_clock_noinstr(void); extern u64 local_clock(void); #endif --- a/kernel/sched/clock.c +++ b/kernel/sched/clock.c @@ -266,7 +266,7 @@ static __always_inline u64 sched_clock_l s64 delta; again: - now = sched_clock(); + now = sched_clock_noinstr(); delta = now - scd->tick_raw; if (unlikely(delta < 0)) delta = 0; @@ -293,22 +293,29 @@ static __always_inline u64 sched_clock_l return clock; } -noinstr u64 local_clock(void) +noinstr u64 local_clock_noinstr(void) { u64 clock; if (static_branch_likely(&__sched_clock_stable)) - return sched_clock() + __sched_clock_offset; + return sched_clock_noinstr() + __sched_clock_offset; if (!static_branch_likely(&sched_clock_running)) - return sched_clock(); + return sched_clock_noinstr(); - preempt_disable_notrace(); clock = sched_clock_local(this_scd()); - preempt_enable_notrace(); return clock; } + +u64 local_clock(void) +{ + u64 now; + preempt_disable_notrace(); + now = local_clock_noinstr(); + preempt_enable_notrace(); + return now; +} EXPORT_SYMBOL_GPL(local_clock); static notrace u64 sched_clock_remote(struct sched_clock_data *scd) From patchwork Mon May 8 21:20:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 91288 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2445946vqo; Mon, 8 May 2023 14:38:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6w1iBFVJygjQJH8giExG0IBT0cnCX66olEd1t7DXSwwbv0SML2nzpYI25xF3f9Dnwd6Gpj X-Received: by 2002:a05:6a20:1581:b0:f9:1f3e:cccb with SMTP id h1-20020a056a20158100b000f91f3ecccbmr15879759pzj.10.1683581894537; Mon, 08 May 2023 14:38:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683581894; cv=none; d=google.com; s=arc-20160816; b=w2kjcutgRD+W1vXH1kuLzsryi407ZtoJjQWozbHp19jOvmBEiub0lbo8cXmNxMgWSB 8Oe7Q0rU8BGmBZ3o3T/AmwnxuL/m1xciDifmU6lIhnEalw+5E8B0NKWB4WcPGf/GMR9n 82ad8u5pJJ8yPcuItK+3DrMQkLlWYRu3RUewrhutubctp9tWs7xuRoZ7OrAEN30ohyyB XolFT6w/myih5p8m5fq6jim6/e6Y6D9DjD0Wy2YQCQvEPTdtu1UGe135hpHekcRIFy80 ZRKQuZOCOPi5nB0mD3EYsjT08cRjbwQHs7Jqw8inJJpgP8i45jb4jM+kNSYnVf9JitIm NCDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=vb5RHA5WAD/i+pcNIxyZaXw8sGymL5Wslp2oL9HHf5o=; b=bPMJrn56F4863Oi6EeObpyPhgNeqKxDhm3ouVe/W/IGSTiZxtUL6+FdD5NM91g7n9V DTu0iboYsigpKRD9VndmQFEY9aMfoYO7pncCUu/S3O+3RTVHeRkEGLmKLkQ75njoFIhI WCepr7iAm+0XMxK+53kyqKJlvaNdwPQx9BREEiF5aIgAR3N28TBtI1/lKbtzQOYR16xs R18XDIJ4E8mPHoY5wCiYOAZlzbyBAc8tg390l9DyO8hj0qLay6fcW5qNNdRu1YDgBETi uathXTRqe2QiyEAwL1m6GmTahTpa6dKW2lZSwPocmZmau4KVs+Duq/XcJTfOpETu04DN r7uw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=tCSw7wyz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v16-20020a63d550000000b0052c8d4d0d1esi9043905pgi.364.2023.05.08.14.37.59; Mon, 08 May 2023 14:38:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=tCSw7wyz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234089AbjEHVgD (ORCPT + 99 others); Mon, 8 May 2023 17:36:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233588AbjEHVfy (ORCPT ); Mon, 8 May 2023 17:35:54 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 738AD9ED1; Mon, 8 May 2023 14:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vb5RHA5WAD/i+pcNIxyZaXw8sGymL5Wslp2oL9HHf5o=; b=tCSw7wyzM5jCUBmoiGImrzrVMf H/ZkDYGbPCkHyTxMrtpf+U6dHzUyVsPW7AEfG0XssOJSV4kg3e8pz7ZJKIt2zgel1Tg3scu0q87KX XOGfQLNrVl701X/bZl65glhBPOZfF/4zUxmiv7nVkkSPkyGMxinwX/GHPApF9GoBvs9bYw6r7ewxn X4tTNuQUxRZNTcX8fh50RBEUA8q/DMH4OvtBRsskPQPXuLsLNENKNssUXCF141sZwuMRTOp+ricCh X8jdVIBKJsrzV1GFPWKS5U8xiSZcmMDh8vZXIac4hlaS2z4/Hlb/evVbvzVzGW+FVjdzYIKs0Vb/s ffwGxzrA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pw8UZ-00EW3n-C8; Mon, 08 May 2023 21:33:59 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id CA95230614B; Mon, 8 May 2023 23:33:48 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 52A012482942D; Mon, 8 May 2023 23:33:48 +0200 (CEST) Message-ID: <20230508213147.990013706@infradead.org> User-Agent: quilt/0.66 Date: Mon, 08 May 2023 23:20:00 +0200 From: Peter Zijlstra To: bigeasy@linutronix.de Cc: mark.rutland@arm.com, maz@kernel.org, catalin.marinas@arm.com, will@kernel.org, chenhuacai@kernel.org, kernel@xen0n.name, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, pbonzini@redhat.com, wanpengli@tencent.com, vkuznets@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, jgross@suse.com, boris.ostrovsky@oracle.com, daniel.lezcano@linaro.org, kys@microsoft.com, haiyangz@microsoft.com, wei.liu@kernel.org, decui@microsoft.com, rafael@kernel.org, peterz@infradead.org, longman@redhat.com, boqun.feng@gmail.com, pmladek@suse.com, senozhatsky@chromium.org, rostedt@goodmis.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, jstultz@google.com, sboyd@kernel.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linux-s390@vger.kernel.org, kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, linux-pm@vger.kernel.org Subject: [RFC][PATCH 9/9] cpuidle: Use local_clock_noinstr() References: <20230508211951.901961964@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765363568560347381?= X-GMAIL-MSGID: =?utf-8?q?1765363568560347381?= With the introduction of local_clock_noinstr(), local_clock() itself is no longer marked noinstr, use the correct function. Signed-off-by: Peter Zijlstra (Intel) --- drivers/cpuidle/cpuidle.c | 8 ++++---- drivers/cpuidle/poll_state.c | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -145,7 +145,7 @@ static noinstr void enter_s2idle_proper( instrumentation_begin(); - time_start = ns_to_ktime(local_clock()); + time_start = ns_to_ktime(local_clock_noinstr()); tick_freeze(); /* @@ -169,7 +169,7 @@ static noinstr void enter_s2idle_proper( tick_unfreeze(); start_critical_timings(); - time_end = ns_to_ktime(local_clock()); + time_end = ns_to_ktime(local_clock_noinstr()); dev->states_usage[index].s2idle_time += ktime_us_delta(time_end, time_start); dev->states_usage[index].s2idle_usage++; @@ -243,7 +243,7 @@ noinstr int cpuidle_enter_state(struct c sched_idle_set_state(target_state); trace_cpu_idle(index, dev->cpu); - time_start = ns_to_ktime(local_clock()); + time_start = ns_to_ktime(local_clock_noinstr()); stop_critical_timings(); if (!(target_state->flags & CPUIDLE_FLAG_RCU_IDLE)) { @@ -276,7 +276,7 @@ noinstr int cpuidle_enter_state(struct c start_critical_timings(); sched_clock_idle_wakeup_event(); - time_end = ns_to_ktime(local_clock()); + time_end = ns_to_ktime(local_clock_noinstr()); trace_cpu_idle(PWR_EVENT_EXIT, dev->cpu); /* The cpu is no longer idle or about to enter idle. */ --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -15,7 +15,7 @@ static int __cpuidle poll_idle(struct cp { u64 time_start; - time_start = local_clock(); + time_start = local_clock_noinstr(); dev->poll_time_limit = false; @@ -32,7 +32,7 @@ static int __cpuidle poll_idle(struct cp continue; loop_count = 0; - if (local_clock() - time_start > limit) { + if (local_clock_noinstr() - time_start > limit) { dev->poll_time_limit = true; break; }