From patchwork Thu Nov 10 17:50:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Guittot X-Patchwork-Id: 18252 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp286544wru; Thu, 10 Nov 2022 09:53:02 -0800 (PST) X-Google-Smtp-Source: AMsMyM5vtJALjP3A90dy+nzx12yju8upObnSJEmK79/4WFY8BBMhrIxkRR1DZKA7nZfzGRL0D3oR X-Received: by 2002:a17:90a:7808:b0:214:1a8a:a40a with SMTP id w8-20020a17090a780800b002141a8aa40amr49295426pjk.137.1668102782118; Thu, 10 Nov 2022 09:53:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668102782; cv=none; d=google.com; s=arc-20160816; b=MfRMOzJwmtV4JebkeorX2ev3Ky4MP+foEoU1rvoGYYFncbjnOqt7GrbFYtmjA/C/3n yQD15KiB8m7ByjzXyzl4z6OfUn8xKYZ92GKGceQpTGA0T88LjqDvbgGzk1GKCmkw8+fh A+2oolfjo5IPGCE9yhIFsipvZythbTlwNSyr/vPixNwFpiUO/faWgBifUGbu6ZX6j871 LLVu2esXJKbJ/fVAMYqW1bIM4F2P9N24Y7imjSuKudmY4+EwHwbDBatNmcSCeDaa39P3 ANpf5Y1HIEF1CzxeB3jTbLlhOjgE3YbsNeExETr24naIgbS2NBFa5Ps60jyzMLIovmqv XMng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from:dkim-signature; bh=l+29IGkaCXOYpZn2jHPEOLz7ZMwnGQwxoqUEqxvOsgk=; b=aEjvH2ID794ZdOiNqIgexRn0pbCkz2xcVFljtJJdy0IiEE/TKMQj+AOCAuMUcCBwl/ vyVxOKquXdYOONQXrczczX/f5wsw+vN2amlSLv5LJ9tJi+vklVwtfJ6i4n3RhZ2kr5N/ XmkIyaH4Z26ln76V6mXMymd/nf774iwqv1bzJGg+pCeGkq96QzqQM0+vdBmqN6OrNQg0 AjRkdoI/F3EGn1WiiqPvB8BpAXwfaSfkG4CIQhPbc7RIKFAkJ+vjg7Eq3pSeIIC2+8AI vn2KqXiL/jqK9YDNV7uQsSaSHuWweBV4lW7Mq/VKULG6TuaDDpx4h6+UqKuJCPV7KIv/ 0I+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IQZHxebG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l7-20020a17090a850700b0020d2286c30csi153145pjn.134.2022.11.10.09.52.48; Thu, 10 Nov 2022 09:53:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=IQZHxebG; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231443AbiKJRuu (ORCPT + 99 others); Thu, 10 Nov 2022 12:50:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231390AbiKJRuj (ORCPT ); Thu, 10 Nov 2022 12:50:39 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 304693F05A for ; Thu, 10 Nov 2022 09:50:28 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id h133-20020a1c218b000000b003cf4d389c41so3997095wmh.3 for ; Thu, 10 Nov 2022 09:50:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=l+29IGkaCXOYpZn2jHPEOLz7ZMwnGQwxoqUEqxvOsgk=; b=IQZHxebG+swqyRL1GrlaPg7XGoqXLFverqCbZRvsHp4AuYjtfCvMaTiyKhK6BySHNc XNx1PdKShf0d9gGAMcc4EtNUvRN04c1skxzQsSrXICz6n9lapx6ZLP/q09VsEMgQLzfq Dk+yVLBzHJEHaR1ZlkG48BWay/dFVlR5NcDx/fQkqrT4asTtvhLSXTlb9tFyBu7H8kw/ gUzGba38uMlvi7RZODngbj5RWr6Mlk+2Su5F3aht1t2BWXfdhXuyLNgxnaocQIdd+8DC yTmryx2kXVKcJh6byEOTnZKbZUDCICzP8xKOmvHl6g05DzSEVbRKEEVU0JUogqIWfraN NMyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l+29IGkaCXOYpZn2jHPEOLz7ZMwnGQwxoqUEqxvOsgk=; b=YZ7J5VTz3tX8zkunRa77fD784DLMBTD2hc3+f0uxnsROj9H7CNcyfhDh9vaKk4JJSJ boePrBhIEOaIJJtQDcTbbfrIMhLs29AbiL7Nx0vdo3I7pEuzn+wwHGh0UYgWCpOUPips M55iSTqDZTnKmAMvngy+Xty7HeuevOu01eXI9RfE//iF4oAgfKDufMp96E6gH/rTAHWA t9WEo2CJ4ZIhJwA6OisTk1ll9/QzYSTdPKvarkBlPqjKJmzM+DjHBOk2CZcLXQqVqHUl YsTUn8xlxq8acMjfkD166/u3qLuKlP10goykuf54JgVxx3gBNfVB/TsEDqxincQSfhiE +/bQ== X-Gm-Message-State: ACrzQf0ZNtGXOtHnLFHIvA+9dnqMbOhyRjdI9vxJQcyJKwAuIELyMkAw IS2jpPQ9bAgHIuKNJHxTKd+axA== X-Received: by 2002:a05:600c:3546:b0:3cf:a18e:12e5 with SMTP id i6-20020a05600c354600b003cfa18e12e5mr18865370wmq.112.1668102626631; Thu, 10 Nov 2022 09:50:26 -0800 (PST) Received: from localhost.localdomain ([2a01:e0a:f:6020:342a:468f:562a:9cc4]) by smtp.gmail.com with ESMTPSA id f6-20020a05600c4e8600b003cfc02ab8basm6514677wmq.33.2022.11.10.09.50.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 09:50:24 -0800 (PST) From: Vincent Guittot To: mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org, parth@linux.ibm.com Cc: qyousef@layalina.io, chris.hyser@oracle.com, patrick.bellasi@matbug.net, David.Laight@aculab.com, pjt@google.com, pavel@ucw.cz, tj@kernel.org, qperret@google.com, tim.c.chen@linux.intel.com, joshdon@google.com, timj@gnu.org, kprateek.nayak@amd.com, yu.c.chen@intel.com, youssefesmat@chromium.org, joel@joelfernandes.org, Vincent Guittot Subject: [PATCH v8 5/9] sched/fair: Take into account latency priority at wakeup Date: Thu, 10 Nov 2022 18:50:05 +0100 Message-Id: <20221110175009.18458-6-vincent.guittot@linaro.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20221110175009.18458-1-vincent.guittot@linaro.org> References: <20221110175009.18458-1-vincent.guittot@linaro.org> X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,PDS_BTC_ID,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749132543157477692?= X-GMAIL-MSGID: =?utf-8?q?1749132543157477692?= Take into account the latency priority of a thread when deciding to preempt the current running thread. We don't want to provide more CPU bandwidth to a thread but reorder the scheduling to run latency sensitive task first whenever possible. As long as a thread didn't use its bandwidth, it will be able to preempt the current thread. At the opposite, a thread with a low latency priority will preempt current thread at wakeup only to keep fair CPU bandwidth sharing. Otherwise it will wait for the tick to get its sched slice. curr vruntime | sysctl_sched_wakeup_granularity <--> ----------------------------------|----|-----------------------|--------------- | |<---------------------> | . sysctl_sched_latency | . default/current latency entity | . | . 1111111111111111111111111111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1- se preempts curr at wakeup ------>|<- se doesn't preempt curr ----------------- | . | . | . low latency entity | . ---------------------->| % of sysctl_sched_latency | 1111111111111111111111111111111111111111111111111111111111|0000|-1-1-1-1-1-1-1- preempt ------------------------------------------------->|<- do not preempt -- | . | . | . high latency entity | . |<-----------------------|----. | % of sysctl_sched_latency . 111111111|0000|-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1-1 preempt->|<- se doesn't preempt curr ------------------------------------------ Tests results of nice latency impact on heavy load like hackbench: hackbench -l (2560 / group) -g group group latency 0 latency 19 1 1.378(+/- 1%) 1.337(+/- 1%) + 3% 4 1.393(+/- 3%) 1.312(+/- 3%) + 6% 8 1.308(+/- 2%) 1.279(+/- 1%) + 2% 16 1.347(+/- 1%) 1.317(+/- 1%) + 2% hackbench -p -l (2560 / group) -g group group 1 1.836(+/- 17%) 1.148(+/- 5%) +37% 4 1.586(+/- 6%) 1.109(+/- 8%) +30% 8 1.209(+/- 4%) 0.780(+/- 4%) +35% 16 0.805(+/- 5%) 0.728(+/- 4%) +10% By deacreasing the latency prio, we reduce the number of preemption at wakeup and help hackbench making progress. Test results of nice latency impact on short live load like cyclictest while competing with heavy load like hackbench: hackbench -l 10000 -g $group & cyclictest --policy other -D 5 -q -n latency 0 latency -20 group min avg max min avg max 0 16 19 29 17 18 29 1 43 299 7359 63 84 3422 4 56 449 14806 45 83 284 8 63 820 51123 63 83 283 16 64 1326 70684 41 157 26852 group = 0 means that hackbench is not running. The avg is significantly improved with nice latency -20 especially with large number of groups but min and max remain quite similar. If we add the histogram parameter to get details of latency, we have : hackbench -l 10000 -g 16 & cyclictest --policy other -D 5 -q -n -H 20000 --histfile data.txt latency 0 latency -20 Min Latencies: 64 62 Avg Latencies: 1170 107 Max Latencies: 88069 10417 50% latencies: 122 86 75% latencies: 614 91 85% latencies: 961 94 90% latencies: 1225 97 95% latencies: 6120 102 99% latencies: 18328 159 With percentile details, we see the benefit of nice latency -20 as only 1% of the latencies are above 159us whereas the default latency has got 15% around ~1ms or above and 5% over the 6ms. Signed-off-by: Vincent Guittot --- include/linux/sched.h | 4 ++- init/init_task.c | 2 +- kernel/sched/core.c | 38 +++++++++++++++++++++---- kernel/sched/debug.c | 2 +- kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 12 ++++++++ 6 files changed, 109 insertions(+), 15 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 6805f378a9c3..a74cad08e91e 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -567,6 +567,8 @@ struct sched_entity { /* cached value of my_q->h_nr_running */ unsigned long runnable_weight; #endif + /* preemption offset in ns */ + long latency_offset; #ifdef CONFIG_SMP /* @@ -783,7 +785,7 @@ struct task_struct { int static_prio; int normal_prio; unsigned int rt_priority; - int latency_nice; + int latency_prio; struct sched_entity se; struct sched_rt_entity rt; diff --git a/init/init_task.c b/init/init_task.c index 7dd71dd2d261..b8ddf403bc62 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -78,7 +78,7 @@ struct task_struct init_task .prio = MAX_PRIO - 20, .static_prio = MAX_PRIO - 20, .normal_prio = MAX_PRIO - 20, - .latency_nice = DEFAULT_LATENCY_NICE, + .latency_prio = NICE_WIDTH - 20, .policy = SCHED_NORMAL, .cpus_ptr = &init_task.cpus_mask, .user_cpus_ptr = NULL, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index b2accc9da4fe..caf54e54a74f 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1284,6 +1284,16 @@ static void set_load_weight(struct task_struct *p, bool update_load) } } +static void set_latency_offset(struct task_struct *p) +{ + long weight = sched_latency_to_weight[p->latency_prio]; + s64 offset; + + offset = weight * get_sched_latency(false); + offset = div_s64(offset, NICE_LATENCY_WEIGHT_MAX); + p->se.latency_offset = (long)offset; +} + #ifdef CONFIG_UCLAMP_TASK /* * Serializes updates of utilization clamp values @@ -4559,7 +4569,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p) p->prio = p->normal_prio = p->static_prio; set_load_weight(p, false); - p->latency_nice = DEFAULT_LATENCY_NICE; + p->latency_prio = NICE_TO_LATENCY(0); + set_latency_offset(p); + /* * We don't need the reset flag anymore after the fork. It has * fulfilled its duty: @@ -7324,8 +7336,10 @@ static void __setscheduler_params(struct task_struct *p, static void __setscheduler_latency(struct task_struct *p, const struct sched_attr *attr) { - if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) - p->latency_nice = attr->sched_latency_nice; + if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE) { + p->latency_prio = NICE_TO_LATENCY(attr->sched_latency_nice); + set_latency_offset(p); + } } /* @@ -7510,7 +7524,7 @@ static int __sched_setscheduler(struct task_struct *p, if (attr->sched_flags & SCHED_FLAG_UTIL_CLAMP) goto change; if (attr->sched_flags & SCHED_FLAG_LATENCY_NICE && - attr->sched_latency_nice != p->latency_nice) + attr->sched_latency_nice != LATENCY_TO_NICE(p->latency_prio)) goto change; p->sched_reset_on_fork = reset_on_fork; @@ -8051,7 +8065,7 @@ SYSCALL_DEFINE4(sched_getattr, pid_t, pid, struct sched_attr __user *, uattr, get_params(p, &kattr); kattr.sched_flags &= SCHED_FLAG_ALL; - kattr.sched_latency_nice = p->latency_nice; + kattr.sched_latency_nice = LATENCY_TO_NICE(p->latency_prio); #ifdef CONFIG_UCLAMP_TASK /* @@ -11204,6 +11218,20 @@ const u32 sched_prio_to_wmult[40] = { /* 15 */ 119304647, 148102320, 186737708, 238609294, 286331153, }; +/* + * latency weight for wakeup preemption + */ +const int sched_latency_to_weight[40] = { + /* -20 */ -1024, -973, -922, -870, -819, + /* -15 */ -768, -717, -666, -614, -563, + /* -10 */ -512, -461, -410, -358, -307, + /* -5 */ -256, -205, -154, -102, -51, + /* 0 */ 0, 51, 102, 154, 205, + /* 5 */ 256, 307, 358, 410, 461, + /* 10 */ 512, 563, 614, 666, 717, + /* 15 */ 768, 819, 870, 922, 973, +}; + void call_trace_sched_update_nr_running(struct rq *rq, int count) { trace_sched_update_nr_running_tp(rq, count); diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c index a3f7876217a6..06aaa0c81d4b 100644 --- a/kernel/sched/debug.c +++ b/kernel/sched/debug.c @@ -1042,7 +1042,7 @@ void proc_sched_show_task(struct task_struct *p, struct pid_namespace *ns, #endif P(policy); P(prio); - P(latency_nice); + P(latency_prio); if (task_has_dl_policy(p)) { P(dl.runtime); P(dl.deadline); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index eb04c83112a0..4299d5108dc7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4558,6 +4558,8 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags) update_idle_cfs_rq_clock_pelt(cfs_rq); } +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se); + /* * Preempt the current task with a newly woken task if needed: */ @@ -4566,7 +4568,7 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) { unsigned long ideal_runtime, delta_exec; struct sched_entity *se; - s64 delta; + s64 delta, offset; ideal_runtime = sched_slice(cfs_rq, curr); delta_exec = curr->sum_exec_runtime - curr->prev_sum_exec_runtime; @@ -4591,10 +4593,12 @@ check_preempt_tick(struct cfs_rq *cfs_rq, struct sched_entity *curr) se = __pick_first_entity(cfs_rq); delta = curr->vruntime - se->vruntime; - if (delta < 0) + offset = wakeup_latency_gran(curr, se); + if (delta < offset) return; - if (delta > ideal_runtime) + if ((delta > ideal_runtime) || + (delta > get_latency_max())) resched_curr(rq_of(cfs_rq)); } @@ -5716,6 +5720,35 @@ static int sched_idle_cpu(int cpu) } #endif +static void set_next_buddy(struct sched_entity *se); + +static void check_preempt_from_others(struct cfs_rq *cfs, struct sched_entity *se) +{ + struct sched_entity *next; + + if (se->latency_offset >= 0) + return; + + if (cfs->nr_running <= 1) + return; + /* + * When waking from another class, we don't need to check to preempt at + * wakeup and don't set next buddy as a candidate for being picked in + * priority. + * In case of simultaneous wakeup when current is another class, the + * latency sensitive tasks lost opportunity to preempt non sensitive + * tasks which woke up simultaneously. + */ + + if (cfs->next) + next = cfs->next; + else + next = __pick_first_entity(cfs); + + if (next && wakeup_preempt_entity(next, se) == 1) + set_next_buddy(se); +} + /* * The enqueue_task method is called before nr_running is * increased. Here we update the fair scheduling stats and @@ -5802,14 +5835,15 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags) if (!task_new) update_overutilized_status(rq); + if (rq->curr->sched_class != &fair_sched_class) + check_preempt_from_others(cfs_rq_of(&p->se), &p->se); + enqueue_throttle: assert_list_leaf_cfs_rq(rq); hrtick_update(rq); } -static void set_next_buddy(struct sched_entity *se); - /* * The dequeue_task method is called before nr_running is * decreased. We remove the task from the rbtree and @@ -7128,6 +7162,23 @@ balance_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) } #endif /* CONFIG_SMP */ +static long wakeup_latency_gran(struct sched_entity *curr, struct sched_entity *se) +{ + long latency_offset = se->latency_offset; + + /* + * A negative latency offset means that the sched_entity has latency + * requirement that needs to be evaluated versus other entity. + * Otherwise, use the latency weight to evaluate how much scheduling + * delay is acceptable by se. + */ + if ((latency_offset < 0) || (curr->latency_offset < 0)) + latency_offset -= curr->latency_offset; + latency_offset = min_t(long, latency_offset, get_latency_max()); + + return latency_offset; +} + static unsigned long wakeup_gran(struct sched_entity *se) { unsigned long gran = sysctl_sched_wakeup_granularity; @@ -7166,11 +7217,12 @@ static int wakeup_preempt_entity(struct sched_entity *curr, struct sched_entity *se) { s64 gran, vdiff = curr->vruntime - se->vruntime; + s64 offset = wakeup_latency_gran(curr, se); - if (vdiff <= 0) + if (vdiff < offset) return -1; - gran = wakeup_gran(se); + gran = offset + wakeup_gran(se); /* * At wake up, the vruntime of a task is capped to not be older than diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 4bf9d7777f99..99f10b4dc230 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -142,6 +142,17 @@ extern int sched_rr_timeslice; * Default tasks should be treated as a task with latency_nice = 0. */ #define DEFAULT_LATENCY_NICE 0 +#define DEFAULT_LATENCY_PRIO (DEFAULT_LATENCY_NICE + LATENCY_NICE_WIDTH/2) + +/* + * Convert user-nice values [ -20 ... 0 ... 19 ] + * to static latency [ 0..39 ], + * and back. + */ +#define NICE_TO_LATENCY(nice) ((nice) + DEFAULT_LATENCY_PRIO) +#define LATENCY_TO_NICE(prio) ((prio) - DEFAULT_LATENCY_PRIO) +#define NICE_LATENCY_SHIFT (SCHED_FIXEDPOINT_SHIFT) +#define NICE_LATENCY_WEIGHT_MAX (1L << NICE_LATENCY_SHIFT) /* * Increase resolution of nice-level calculations for 64-bit architectures. @@ -2116,6 +2127,7 @@ static_assert(WF_TTWU == SD_BALANCE_WAKE); extern const int sched_prio_to_weight[40]; extern const u32 sched_prio_to_wmult[40]; +extern const int sched_latency_to_weight[40]; /* * {de,en}queue flags: