From patchwork Thu Apr 6 06:44:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Jia X-Patchwork-Id: 80086 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp824921vqo; Thu, 6 Apr 2023 00:02:27 -0700 (PDT) X-Google-Smtp-Source: AKy350b6VrfsfELIVhD6rPo2r3gTC+0fKTZYvaZ8RBBhWUzgzevmnICc9QERafuCU0DscwOEf8Si X-Received: by 2002:a05:6a20:4a14:b0:e3:8710:6846 with SMTP id fr20-20020a056a204a1400b000e387106846mr2059864pzb.10.1680764547302; Thu, 06 Apr 2023 00:02:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680764547; cv=none; d=google.com; s=arc-20160816; b=ZkTKP4hdEpOzxbVEGsGPh8nq9lNJOBcVXlMfozu2F5AF2p1y9RmNJNH3B9qWANkNI6 zS+hY0YsR8CNCXX+AlWyDHRYD1PAuvk2G65N3wieHrcmuHUxkYA3zUj3yO74ZcZkWMKe Yu2iApfy61gsnHYwzQ7Ly0xCS3tYsvnhf1r6Nzw4Uw4P8pM3OhyNkpyKNO/1ZznXUDjB bAyHiFX/n+Ol63LeoVt4k7IjhB0fhe1hxeeWwgs2aVNQspxqDEr6zk6k95y0S9JVgh8m pno9m66RizVQZKEevz3QHW+ZpVZbHbm4zklC6VUjmROCjJLpRLsIM1tOLKWAu687ckRn xP+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=SSjUHXD/LhQIUTlrHWLn8aPqnhEOO0fsMY11nZErJaU=; b=rXjkP7diGiDVb7ED8E5pVCMxwYbk5yrmv7/ZWYVASl9FQpqCZB/CE5L/mFZeCq89cM bBtBaAXPKI82HEwHw3UXLJuknqdYCh7HrfjpmmrrfJiese3xVKETMz4tDzlMVx+XGNih Ivb9/DpYxj+f+6IVyroROJgemS6VmCJ96y8g4hA3yY1acD9Z8zLvVX7oE+hIMHF07D0l 62qDQPuvFHizirAbcwNPm8/Z0E9BWkWC3ICZXVtq4dokShv6XzYk5d4vnq0w54qqwgnJ AeNBIm1QqGxdh6Jz2YzgM5AHyiZ6N33s0Mp/EWxblEhqL1FS6cHDvRLiIS07g0jZG6cl W6Qg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Eg7Yqnn4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c197-20020a621cce000000b005dde05007b8si765808pfc.18.2023.04.06.00.02.15; Thu, 06 Apr 2023 00:02:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=Eg7Yqnn4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235596AbjDFGo4 (ORCPT + 99 others); Thu, 6 Apr 2023 02:44:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235582AbjDFGoy (ORCPT ); Thu, 6 Apr 2023 02:44:54 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6B8E9010 for ; Wed, 5 Apr 2023 23:44:26 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id o2so36684419plg.4 for ; Wed, 05 Apr 2023 23:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1680763466; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=SSjUHXD/LhQIUTlrHWLn8aPqnhEOO0fsMY11nZErJaU=; b=Eg7Yqnn4G9oMXexxUoM/skvv1nV3exncgdctUmXMUhrI98ZIk16yls+gARhO8JvqkA ZSAmHo05g48CW/7rTVKO792cIZUUUOfO7Q9A4lUU3lvUthG07h357LhZ/eIA0IphcBRz OzNxB5ZGNaYHVw4I5IdOSDQ6fVn1c8k9kOJFKuC0EjayXBcFhFBIq3G1oW6hpKgsvJWj Sy9SAXr3AoBlO0KeWNgeg09b4E038fLpbQZvOYz5qRewqLb42gDIC2Bw7Y/jcoeBMFyN c3XSQhqDl1qOdnfndb6R/k/mH5H+cRTBZ3bx7bLG+uXcbCmOW173w+QOg6/V8BFRW317 rf+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680763466; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SSjUHXD/LhQIUTlrHWLn8aPqnhEOO0fsMY11nZErJaU=; b=42VEe6ysllf7gX439o5Eej8Y9aU1rK9qCzSuZO8WWmRnEx/TMI/z0tngxYfT9wgRSQ kWsnSuXspL4vOkmJcSMF96BNT8dulVz18xFTVCDDPOdCOjMlkmtFjW5OzJjpVT1/CABN 64/jfuUpVr30/zDbtY/D9sgYOwvyMwToqwNO+nI4jvtGFwI8KwGJ8ktinNi8O5bLnENd 67PhK2t6RQLiUAWjMfJTI/6X6+ZtfXwpq5MPxYqBXwlYouYIJheSU1lh0cBoT08jH3Yb jTegTXoYcaMddIiqHM1WAHCqRFZTW+xB9+pVk6WhkF6necjonWki307bnHLylUXZSz5Q fkag== X-Gm-Message-State: AAQBX9drUUwfKtsnEbxd7ZrwbA9P7kUZdhYwjMrqn8NL1xg08QtJV5M5 18Sr6YEXBE74NwAX971TltjLiA== X-Received: by 2002:a17:90b:3ec4:b0:237:5a3c:e86c with SMTP id rm4-20020a17090b3ec400b002375a3ce86cmr10170522pjb.24.1680763466351; Wed, 05 Apr 2023 23:44:26 -0700 (PDT) Received: from C02G87K0MD6R.bytedance.net ([139.177.225.244]) by smtp.gmail.com with ESMTPSA id p13-20020a17090a348d00b002348d711ebbsm496882pjb.16.2023.04.05.23.44.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Apr 2023 23:44:25 -0700 (PDT) From: Hao Jia To: mingo@redhat.com, peterz@infradead.org, mingo@kernel.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mgorman@techsingularity.net Cc: linux-kernel@vger.kernel.org, Hao Jia , Phil Auld Subject: [PATCH v4] sched/core: Adapt WARN_DOUBLE_CLOCK machinery for core-sched Date: Thu, 6 Apr 2023 14:44:15 +0800 Message-Id: <20230406064415.17110-1-jiahao.os@bytedance.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762409365981807903?= X-GMAIL-MSGID: =?utf-8?q?1762409365981807903?= When sched_core_enabled(), we sometimes need to call update_rq_clock() to update the rq clock of sibling CPUs on the same core, before that we need to clear RQCF_UPDATED of rq->clock_update_flags to avoid the WARN_DOUBLE_CLOCK warning. Because at this time the rq->clock_update_flags of sibling CPUs may be RQCF_UPDATED. If sched_core_enabled(), we will get a core-wide rq->lock, so at this point we can safely clear RQCF_UPDATED of rq->clock_update_flags of all CPUs on this core to avoid the WARN_DOUBLE_CLOCK warning. We sometimes use rq_pin_lock() and raw_spin_rq_lock() separately, For example newidle_balance() and _double_lock_balance(). We will temporarily give up core-wide rq->lock, and then use raw_spin_rq_lock() to reacquire core-wide rq->lock without rq_pin_lock(), so We can not clear RQCF_UPDATED of rq->clock_update_flags of other cpus on the same core in rq_pin_lock(). Steps to reproduce: 1. Enable CONFIG_SCHED_DEBUG and CONFIG_SCHED_CORE when compiling the kernel 2. echo 1 > /sys/kernel/debug/clear_warn_once echo "WARN_DOUBLE_CLOCK" > /sys/kernel/debug/sched/features 3. Run the linux/tools/testing/selftests/sched/cs_prctl_test test Signed-off-by: Hao Jia Reviewed-by: Phil Auld --- v3->v4: - Replace "core wide" with "core-wide" everywhere. - Add "Reviewed-by: Phil Auld ". [v3] https://lore.kernel.org/all/20230330035827.16937-1-jiahao.os@bytedance.com v2->v3: - Modify the function name to sched_core_clear_rqcf_updated, and add function comments. - Modify commit information. [v2] https://lore.kernel.org/all/20230215073927.97802-1-jiahao.os@bytedance.com v1->v2: - Adapt WARN_DOUBLE_CLOCK machinery for core-sched instead of clearing WARN_DOUBLE_CLOCK warning one by one. - Modify commit information [v1] https://lore.kernel.org/all/20221206070550.31763-1-jiahao.os@bytedance.com kernel/sched/core.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 0d18c3969f90..c6e2c79152ef 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -429,11 +429,32 @@ void sched_core_put(void) schedule_work(&_work); } +/* + * Now, we have obtained a core-wide rq->lock, then we need to clear + * RQCF_UPDATED of rq->clock_update_flags of the sibiling CPU + * on this core to avoid the WARN_DOUBLE_CLOCK warning. + */ +static inline void sched_core_clear_rqcf_updated(struct rq *rq) +{ +#ifdef CONFIG_SCHED_DEBUG + const struct cpumask *smt_mask; + int i; + + if (rq->core_enabled) { + smt_mask = cpu_smt_mask(rq->cpu); + for_each_cpu(i, smt_mask) { + if (rq->cpu != i) + cpu_rq(i)->clock_update_flags &= (RQCF_REQ_SKIP|RQCF_ACT_SKIP); + } + } +#endif +} #else /* !CONFIG_SCHED_CORE */ static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { } static inline void sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { } +static inline void sched_core_clear_rqcf_updated(struct rq *rq) { } #endif /* CONFIG_SCHED_CORE */ @@ -548,6 +569,7 @@ void raw_spin_rq_lock_nested(struct rq *rq, int subclass) if (likely(lock == __rq_lockp(rq))) { /* preempt_count *MUST* be > 1 */ preempt_enable_no_resched(); + sched_core_clear_rqcf_updated(rq); return; } raw_spin_unlock(lock);