From patchwork Tue Apr 11 21:41:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 82202 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2890081vqo; Tue, 11 Apr 2023 15:17:20 -0700 (PDT) X-Google-Smtp-Source: AKy350YaFmMNQP6BWSsMT5AtBh8nc+aP5DkqM8yQz9nAbK6oJr9iPYVUUG3SPNq1tRItIgQqTCH8 X-Received: by 2002:a05:6a20:4b14:b0:eb:83f1:80a6 with SMTP id fp20-20020a056a204b1400b000eb83f180a6mr954488pzb.47.1681251440465; Tue, 11 Apr 2023 15:17:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681251440; cv=none; d=google.com; s=arc-20160816; b=xS/d+DIonoyln85n8dy7debOQNSfG+7xjWWO3WDHklkbrU/MdS3lflp+SentjRrYxn TxVxz1TSnYnAT3049YNbpQFOsslMjs2RhE+NWzYlWGztvvlmqSVK7gaXvRYQ959/Fxbc /4WuROSBCxkSVtEO0+PBVbZBkxhU+4j1TBJRUGpllSGEnKXg0qsafH2iqti16i6oIuV6 8lyj8FZdaRWZzUj2q6GPOsOi8R3akhbKUdnUeZq0GqaQ0tBK49UZ9H+hDDSANjuB7BSo QPIOPAdyiEuoOGxGl4kqvc3U4IQTwayz/0I15iwMOl8qB1loeN6lyqoTbGrRTqQmZWJX Ri/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Ma3iXrQ50xTEGrsaOcipg59Es3UPahzNcnP/kOkak28=; b=Ngq/ZbOgXZIAGLIKL6iK5vYEYBIZxFWIJ/hJ1170YQykgACVlSTcgAF53lLHBqyr6C wIrL4NXL+gvzELmdJCwCUMBQpZd4cS/99yozOU0AKTm9OmpV0aHE1dEOhkAaO0Wux/Mc /DwCTK6T5US1WoWxm2Zaea7+vTHcptYxc72RevwZtzrYbrdYRPm8JUigwPxaZZRMECsI uLIuvOawpP5beywUCilkDudSU6bPcwJp15Z/2IskOLv/J9pBe0xb9doIkXZ8Uu0PuF1g 7iy3uArehQIwqSfYuZNrpOMuBAAyUcFLVmDZaG4LGrx2dQRbpysnxZH5rhCoMn7CAQqk cMyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=NFYhG2g6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f15-20020a63510f000000b0050c0e269719si13642814pgb.513.2023.04.11.15.17.00; Tue, 11 Apr 2023 15:17:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=NFYhG2g6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229820AbjDKVl2 (ORCPT + 99 others); Tue, 11 Apr 2023 17:41:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjDKVlZ (ORCPT ); Tue, 11 Apr 2023 17:41:25 -0400 Received: from smtpout.efficios.com (unknown [IPv6:2607:5300:203:b2ee::31e5]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 463ED449E for ; Tue, 11 Apr 2023 14:41:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1681249281; bh=2YcjaEdF3Fedq8b1d6ChcarV16QJyDOvjmjefjNrqM0=; h=From:To:Cc:Subject:Date:From; b=NFYhG2g6DP3uKegdxJTT0uDL6yUqQe4sbLtp+QYT0170JR1iingBXp51Yu431L4gU yHV0HNnjqP2Av7jO9/6DuX6DAybKw5rqEPuoWnr/G5kfR1jHCzy7HSm5zFcPGNGiS7 Gp8ZBah1lE1fcdjDiWHrY5gfe2J6J/LzeIWOApxUl/+fhdqOR8mt70F2OhZeIAI+Td 20xqmEgAipMrlODn0sacElkQh+E7Z9fdgCF8e7KkiuyIFr1tLsQ/hwgQ+hm4EeLYl1 etuiYtQvidivmSvtfDZRXQaoS5u5iwIJhArmowey8p9va0fmkDW6kDxIUt2iCnzA3f DAopk9XaYyhKA== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4Pwzmx1MPFztwD; Tue, 11 Apr 2023 17:41:21 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , Aaron Lu , Olivier Dion , michael.christie@oracle.com Subject: [RFC PATCH] sched: Rate limit migrations Date: Tue, 11 Apr 2023 17:41:16 -0400 Message-Id: <20230411214116.361016-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-1.3 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RDNS_NONE,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762919910403950195?= X-GMAIL-MSGID: =?utf-8?q?1762919910403950195?= This WIP patch rate-limits migrations to 32 migrations per 10ms window for each task. The specific migration count and window size can be changed with the following defines in kernel/sched/sched.h: - SCHED_MIGRATION_WINDOW_NS - SCHED_MIGRATION_LIMIT Testing is welcome, especially to see if it helps with Aaron's migration-heavy workload wrt rseq concurrency id performance regression. Link: https://lore.kernel.org/lkml/20230327080502.GA570847@ziqianlu-desk2/ Signed-off-by: Mathieu Desnoyers Cc: Aaron Lu Cc: Peter Zijlstra Cc: Olivier Dion Cc: michael.christie@oracle.com --- include/linux/sched.h | 9 +++++++++ kernel/fork.c | 3 +++ kernel/sched/core.c | 42 ++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 7 +++++++ 4 files changed, 57 insertions(+), 4 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 48d48b2c73a5..bfd5e268900c 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1316,6 +1316,15 @@ struct task_struct { int last_mm_cid; /* Most recent cid in mm */ int mm_cid_active; /* Whether cid bitmap is active */ #endif + /* + * Keep track of last migration time to compare sched_clock + * locally from a single CPU perspective. + */ + u64 last_migration_time; + /* Time slice used in current migration window. */ + u64 migration_window_time_slice; + /* Number of migrations in current migration window. */ + u32 migration_count; struct tlbflush_unmap_batch tlb_ubc; diff --git a/kernel/fork.c b/kernel/fork.c index 3832bea713c4..791792a218f6 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1061,6 +1061,9 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->last_mm_cid = -1; tsk->mm_cid_active = 0; #endif + tsk->last_migration_time = 0; + tsk->migration_window_time_slice = 0; + tsk->migration_count = 0; return tsk; free_stack: diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 2aac6f14f21c..a530727b11f3 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2087,6 +2087,7 @@ void activate_task(struct rq *rq, struct task_struct *p, int flags) if (task_on_rq_migrating(p)) { flags |= ENQUEUE_MIGRATED; sched_mm_cid_migrate_to(rq, p); + p->last_migration_time = sched_clock(); } enqueue_task(rq, p, flags); @@ -3547,17 +3548,47 @@ static int select_fallback_rq(int cpu, struct task_struct *p) return dest_cpu; } +static inline +bool migration_allowed(struct task_struct *p, u64 current_time) +{ + u64 delta = current_time - p->last_migration_time; + + if (delta + p->migration_window_time_slice > SCHED_MIGRATION_WINDOW_NS || + p->migration_count < SCHED_MIGRATION_LIMIT) + return true; + return false; +} + +static inline +void migration_add_delta_to_slice(struct task_struct *p, u64 current_time) +{ + u64 delta = current_time - p->last_migration_time; + + if (delta + p->migration_window_time_slice > SCHED_MIGRATION_WINDOW_NS) { + /* Reset the migration window if it has ended. */ + p->migration_window_time_slice = 0; + p->migration_count = 0; + return; + } + p->migration_window_time_slice += delta; + p->migration_count++; +} + /* * The caller (fork, wakeup) owns p->pi_lock, ->cpus_ptr is stable. */ static inline -int select_task_rq(struct task_struct *p, int cpu, int wake_flags) +int select_task_rq(struct task_struct *p, int prev_cpu, int wake_flags) { + u64 current_time = sched_clock(); + int cpu = prev_cpu; + lockdep_assert_held(&p->pi_lock); - if (p->nr_cpus_allowed > 1 && !is_migration_disabled(p)) - cpu = p->sched_class->select_task_rq(p, cpu, wake_flags); - else + if (p->nr_cpus_allowed > 1 && !is_migration_disabled(p)) { + if (migration_allowed(p, current_time)) + cpu = p->sched_class->select_task_rq(p, cpu, wake_flags); + } else cpu = cpumask_any(p->cpus_ptr); /* @@ -3573,6 +3604,9 @@ int select_task_rq(struct task_struct *p, int cpu, int wake_flags) if (unlikely(!is_cpu_allowed(p, cpu))) cpu = select_fallback_rq(task_cpu(p), p); + if (prev_cpu != cpu) + migration_add_delta_to_slice(p, current_time); + return cpu; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 64220134fb45..e52cc38f10fc 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -104,6 +104,13 @@ struct cpuidle_state; #define TASK_ON_RQ_QUEUED 1 #define TASK_ON_RQ_MIGRATING 2 +/* + * A task can be migrated at most SCHED_MIGRATION_WINDOW_LIMIT times per + * sched-migration window. + */ +#define SCHED_MIGRATION_WINDOW_NS (10ULL * 1000000) /* 10 ms */ +#define SCHED_MIGRATION_LIMIT 32 + extern __read_mostly int scheduler_running; extern unsigned long calc_load_update;