From patchwork Wed Oct 19 12:28:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abel Wu X-Patchwork-Id: 5399 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp311516wrs; Wed, 19 Oct 2022 06:00:36 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6M0bIkyOjkexda5oRfIOQ+iCrwFCbUX31Dmau0JJ5dmKPjnqKVSD3+c3ABJ1b+5jcF8HbK X-Received: by 2002:a17:902:e841:b0:180:49a2:8e6c with SMTP id t1-20020a170902e84100b0018049a28e6cmr8275385plg.143.1666184425342; Wed, 19 Oct 2022 06:00:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666184425; cv=none; d=google.com; s=arc-20160816; b=XWFBdAeUbVdecNEc4N1+apHIZJoCbNkbPR3BfEUKZFryWtTM9+v31LPGZ2SSQ9ADai 073k+/p8qFViSK8VNBdHkHAIBqH4mmzSW8sNvkNsOIWzQbiYnr263Mk34cVAtze5Hwkp 6z8cUTZ8BRtPUlwTG4CQn47ru+cakWO/nzhtFRQv9vS1JyZwCLUmkZuqAisjU+gLeE89 melmHeLyWHAthlESbrqtm9lh00NUVf/aj0bsQOH1ZgShGURtuIDGRI4/cxkwNBiYGnNP 2WIhZsieMc1vZH3CCm9JPBaarcYQnN1SBLA3buuFvGE91B/yiiodXn36r/NVfzKx15Sx W9sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=eRCTWKYjx2wpPZZk04Ley0VgHSoVGSlzkE3jRQHvIBA=; b=JPnr5B9RvYump3ekgpucQ+EaRZTF/LmaXh8rMn4GAXufpV2FVOSmyZZ/5zUHfoZaVV BzvVecBF1XE72Tn2bhoiOwmbFA0rL/OG4OLd6rAG8XuVla7exbPloD+qgurheSFwNwQa l1kwtny4GGkh6FzAd7TgFibCo0yY2+GVZKNnNlHFL4xpg1Wooin7JN95Y53R1WM077w1 JnsrDGXwVY/L0YhPNfKFTu3rRTTQrSU99w4B8lh0pgO7FqWamFFAy3JX+o9C/hND0d76 F4apRGxh3KSwfYonB54gHnqFQFR2jnR/R7nLpAb3XNKsUDP3xfjA2//9F4sXwxC+oLz5 96gg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="J/Z/9XTm"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i62-20020a638741000000b00434e171b9aasi18263070pge.573.2022.10.19.06.00.11; Wed, 19 Oct 2022 06:00:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="J/Z/9XTm"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233337AbiJSMro (ORCPT + 99 others); Wed, 19 Oct 2022 08:47:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232498AbiJSMrW (ORCPT ); Wed, 19 Oct 2022 08:47:22 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C26D812977F for ; Wed, 19 Oct 2022 05:30:23 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id l4so17089789plb.8 for ; Wed, 19 Oct 2022 05:30:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eRCTWKYjx2wpPZZk04Ley0VgHSoVGSlzkE3jRQHvIBA=; b=J/Z/9XTmxWcYsEXXcMX3wHLvo5mA8In7iIOAmCzDG/x2Eq5v5dGj9fIlE3CaWG1Izp RrvUMWXRPQpdzTa3bTTWAQTJsy9E86kONlaiNrAuQD4qfHAYUzSBzWtziP6quryaM6eF txli+FDXkF1IO689AUO+4O/Q2ZF5P5anE8bwp9ksrPOb2F6fDz3yrkFBBVUMtEXrUOqa Dbqpi1HKpYTvJGCnsBC+mahMgDf+Zr9uWLK/W3iC7E3KC0u151mZAIuQjJPxWHq4Yni0 WbnUNA1USzFbWIzl8iHJjlgHC9h9493cw1yZ8qH3WNq2YCFQi0ijMggMWwhmDBNXGyp1 nCoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eRCTWKYjx2wpPZZk04Ley0VgHSoVGSlzkE3jRQHvIBA=; b=O2/P4VEse9kSx5neG6CWkYF8R+5wmKnnFg/O1O3YTLBH7JO8IT1ei/fB0CgPYJA0N1 Y7/zQ1CdCnQCnrVkFOmzNH49n2Df+Tn+pBQVSV3rMpCLtbk0ZnVbplk+BmOuIKrIGiAo 8JVX+5JsuXsLrorhUmYyp3yTA7mKzICwVokWWr1WuOrugxruXwZYbJWT2QIy+k9/mPoG Xzkipk7lUdAbgY7lkcgXai/K9yXRjMm3I9bs1fEfbMnjlsrnybRvdoDI7r+u28DvoZ5Z 2EM3nIjKHb44iAEQxJqvCPfsMZZuOx1ZWzuGxoiVpVpTHakh3bsULyMjyTv1+AxdsQeD JMSA== X-Gm-Message-State: ACrzQf1vQYKiTFrGSJmMTU88w0xXTpnlyJsYvkJB2PqkfEcsXQd0fEWs tLP3encYFUOjvG7P9tYJcgE+YQ== X-Received: by 2002:a17:90a:74c2:b0:20a:f63c:2464 with SMTP id p2-20020a17090a74c200b0020af63c2464mr9724981pjl.224.1666182576122; Wed, 19 Oct 2022 05:29:36 -0700 (PDT) Received: from C02DV8HUMD6R.bytedance.net ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id c21-20020a63da15000000b00439c6a4e1ccsm9881825pgh.62.2022.10.19.05.29.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Oct 2022 05:29:35 -0700 (PDT) From: Abel Wu To: Peter Zijlstra , Ingo Molnar , Mel Gorman , Vincent Guittot , Dietmar Eggemann , Valentin Schneider Cc: Josh Don , Chen Yu , Tim Chen , K Prateek Nayak , "Gautham R . Shenoy" , Aubrey Li , Qais Yousef , Juri Lelli , Rik van Riel , Yicong Yang , Barry Song <21cnbao@gmail.com>, linux-kernel@vger.kernel.org, Abel Wu Subject: [PATCH v6 1/4] sched/fair: Skip core update if task pending Date: Wed, 19 Oct 2022 20:28:56 +0800 Message-Id: <20221019122859.18399-2-wuyun.abel@bytedance.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019122859.18399-1-wuyun.abel@bytedance.com> References: <20221019122859.18399-1-wuyun.abel@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747120999761308491?= X-GMAIL-MSGID: =?utf-8?q?1747120999761308491?= The function __update_idle_core() considers this cpu is idle so only checks its siblings to decide whether the resident core is idle or not and update has_idle_cores hint if necessary. But the problem is that this cpu might not be idle at that moment any more, resulting in the hint being misleading. It's not proper to make this check everywhere in the idle path, but checking just before core updating can make the has_idle_core hint more reliable with negligible cost. Signed-off-by: Abel Wu Reviewed-by: Tim Chen --- kernel/sched/fair.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5ffec4370602..e7f82fa92c5b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6294,6 +6294,9 @@ void __update_idle_core(struct rq *rq) int core = cpu_of(rq); int cpu; + if (rq->ttwu_pending) + return; + rcu_read_lock(); if (test_idle_cores(core)) goto unlock; From patchwork Wed Oct 19 12:28:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abel Wu X-Patchwork-Id: 5405 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp312181wrs; Wed, 19 Oct 2022 06:01:28 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6i0hRJYUhZmSRWkdBXvVeM0LySAGC6z9Ig7WwwY1rPuQVOCGDO83kOskXDk+4i6YTEw271 X-Received: by 2002:a17:90b:3891:b0:20d:59e3:1f4f with SMTP id mu17-20020a17090b389100b0020d59e31f4fmr9588453pjb.21.1666184487864; Wed, 19 Oct 2022 06:01:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666184487; cv=none; d=google.com; s=arc-20160816; b=efXTclVqjQx9StkDytvWGCOL2oOQi3mk6H6e6po4pw5FQMv98y4/nAC0HgWYSJQJWL AnrX0NPM8mGBAWVRWmSzPvjTiiXv3YfRccvdr3uLKnGWqGDvJj++s9BerahE7BZtsUEP LPEnD0FxUnm9Kw/q78ydtfNbdh7+LUBEVFBQFWc1+RG+1k85d6uxuJ7sqYg0n1GuaduA nvPwC/7hwtQAkCEZdF35z/ApEEvdOiNIrqP+BNfvB5EbVSFn/C4Ygo79vTw+f3CNHgjY ltWAM0O1wzVFNfyNYB3BOndMw1oCGFswG5vxAd/HzeG1gCRQv7tXcb7ptatuNdDbMYdy Byaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OSeVek4DWAXpR4R90owtFBiMLKu023iutKikpjmU0co=; b=lQ6JcXwB8wCI3u6x7v2FIzVC19oGR6OyZPabvesLoY0zpliTU8T/Ti4bstedV+0hfz nKO5aUOxcYFqtW/nh5tbkfWtDXk7lebCfVFPFbRKajCDrPX5nV6+8t2mZS7r8qzTwFli +bJ6UPTFraahr270lolGhcK3KYUol9MpKg1BWUAuno0QkMYW+eZg+iIEVmXwX4bx6TCi el/gN2dQ8ATfGrnnMBL9Nal99nx8j1B/1c6x41kDosysY/7ugsiLqylo1z/0MWCEseuB LHv6A5M4xFcxF1HV/FWyKphER++uZq3kBQ5XHPuJV1x6xU4QZGzDHGODJHi87UnmSk33 hzXw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=3H54g9C3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k12-20020a170902ba8c00b00179f9521d08si16115180pls.149.2022.10.19.06.00.54; Wed, 19 Oct 2022 06:01:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=3H54g9C3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231298AbiJSMsR (ORCPT + 99 others); Wed, 19 Oct 2022 08:48:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233373AbiJSMri (ORCPT ); Wed, 19 Oct 2022 08:47:38 -0400 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A365191D40 for ; Wed, 19 Oct 2022 05:30:39 -0700 (PDT) Received: by mail-pl1-x634.google.com with SMTP id c24so17085891pls.9 for ; Wed, 19 Oct 2022 05:30:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OSeVek4DWAXpR4R90owtFBiMLKu023iutKikpjmU0co=; b=3H54g9C37A6stfEGM0FnJt1/rtt6dcPlNNRW1AVTeKPZ94o4gIE5RB45VtI9EQHmAy B+/sOk9RlFHKETZx2kWSA3hdU1cuLsa7kK79UlEDJ3sa+hPObyMIll4sUcHM3EMN42+v 1C5zszRoNCaekZgIimEr1Zv6z1xBa/aEnqxlhr8Ytu3t0zbyq9WIHqEgs19tX2xbO+U8 lT3QhnZSeow8f81cfv925yFiz4JcxOntqsPEYT+cNHgggo72A81lqJEKRN1s1eW8Elj/ CAeECSunQhYtzqHqLTqkt0tFVyPnWgtRhSqQZTzXBs94d3lJNcgmr9ZKeOI9IJiSoo21 j/Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OSeVek4DWAXpR4R90owtFBiMLKu023iutKikpjmU0co=; b=BSTaOqeWEERdM1A5FmW7rno1hNR4Y+jpkw/0yuEeoEtBqAd6bqdAtht3f06rSAppxT Hq1+FwxDindH5DGRDR2RYez2MnK9BONlzGML398wSIs+0cF36hX1ACV6WDYTVP48VdOP orYIgb2y+LHRA6uL9jLfuEhKWQp+MMV8qq74f9MIc1ZtTqoJ4f3IPEuLAwf+R96gD0CI +aQlvsfn79emGPHi73VXw2lQbOjEzrCjeI//4eukunzvc0/EDFY77JzMIm5npoyHBsx0 ny0RNayHA4D/Q5dBBE10d74g79rkmdljvU+M28rwTC69rW+MMa6WFMYG9wAJajKQuq0B 79tQ== X-Gm-Message-State: ACrzQf0Kc4x2a1LhsX1kWfi+MBmLUq3/12WJVvnhu6jizGB+FbVbePVX si7+Qfb7WR+iFgmfN/aSNDpEoA== X-Received: by 2002:a17:90a:db0f:b0:20a:7294:638d with SMTP id g15-20020a17090adb0f00b0020a7294638dmr43710507pjv.228.1666182582643; Wed, 19 Oct 2022 05:29:42 -0700 (PDT) Received: from C02DV8HUMD6R.bytedance.net ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id c21-20020a63da15000000b00439c6a4e1ccsm9881825pgh.62.2022.10.19.05.29.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Oct 2022 05:29:42 -0700 (PDT) From: Abel Wu To: Peter Zijlstra , Ingo Molnar , Mel Gorman , Vincent Guittot , Dietmar Eggemann , Valentin Schneider Cc: Josh Don , Chen Yu , Tim Chen , K Prateek Nayak , "Gautham R . Shenoy" , Aubrey Li , Qais Yousef , Juri Lelli , Rik van Riel , Yicong Yang , Barry Song <21cnbao@gmail.com>, linux-kernel@vger.kernel.org, Abel Wu Subject: [PATCH v6 2/4] sched/fair: Ignore SIS_UTIL when has_idle_core Date: Wed, 19 Oct 2022 20:28:57 +0800 Message-Id: <20221019122859.18399-3-wuyun.abel@bytedance.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019122859.18399-1-wuyun.abel@bytedance.com> References: <20221019122859.18399-1-wuyun.abel@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747121065307621086?= X-GMAIL-MSGID: =?utf-8?q?1747121065307621086?= When SIS_UTIL is enabled, SIS domain scan will be skipped if the LLC is overloaded even the has_idle_core hint is true. Since idle load balancing is triggered at tick boundary, the idle cores can stay cold for the whole tick period wasting time meanwhile some of other cpus might be overloaded. Give it a chance to scan for idle cores if the hint implies a worthy effort. Benchmark ========= All of the benchmarks are done inside a normal cpu cgroup in a clean environment with cpu turbo disabled, and test machines are: A) A dual socket machine modeled Intel Xeon(R) Platinum 8260 with SNC disabled, so there are 2 NUMA nodes each of which has 24C/48T. Each NUMA shares an LLC. B) A dual socket machine modeled AMD EPYC 7Y83 64-Core Processor with NPS1 enabled, so there are 2 NUMA nodes each of which has 64C/128T. Each NUMA node contains several LLCs sized of 16 cpus. Based on tip sched/core fb04563d1cae (v5.19.0). Results ======= hackbench-process-pipes (A) vanilla patched Amean 1 0.2767 ( 0.00%) 0.2540 ( 8.19%) Amean 4 0.6080 ( 0.00%) 0.6220 ( -2.30%) Amean 7 0.7923 ( 0.00%) 0.8020 ( -1.22%) Amean 12 1.3917 ( 0.00%) 1.1823 ( 15.04%) Amean 21 3.6747 ( 0.00%) 2.7717 ( 24.57%) Amean 30 6.7070 ( 0.00%) 5.1200 * 23.66%* Amean 48 9.3537 ( 0.00%) 8.5890 * 8.18%* Amean 79 11.6627 ( 0.00%) 11.2580 ( 3.47%) Amean 110 13.4473 ( 0.00%) 13.1283 ( 2.37%) Amean 141 16.4747 ( 0.00%) 15.5967 * 5.33%* Amean 172 19.0000 ( 0.00%) 18.1153 * 4.66%* Amean 203 21.4200 ( 0.00%) 21.1340 ( 1.34%) Amean 234 24.2250 ( 0.00%) 23.8227 ( 1.66%) Amean 265 27.2400 ( 0.00%) 26.8293 ( 1.51%) Amean 296 30.6937 ( 0.00%) 29.5800 * 3.63%* (B) Amean 1 0.3543 ( 0.00%) 0.3650 ( -3.01%) Amean 4 0.4623 ( 0.00%) 0.4837 ( -4.61%) Amean 7 0.5117 ( 0.00%) 0.4997 ( 2.35%) Amean 12 0.5707 ( 0.00%) 0.5863 ( -2.75%) Amean 21 0.9717 ( 0.00%) 0.8930 * 8.10%* Amean 30 1.4423 ( 0.00%) 1.2530 ( 13.13%) Amean 48 2.3520 ( 0.00%) 1.9743 * 16.06%* Amean 79 5.7193 ( 0.00%) 3.4933 * 38.92%* Amean 110 6.9893 ( 0.00%) 5.5963 * 19.93%* Amean 141 9.1103 ( 0.00%) 7.6550 ( 15.97%) Amean 172 10.2490 ( 0.00%) 8.8323 * 13.82%* Amean 203 11.3727 ( 0.00%) 10.8683 ( 4.43%) Amean 234 12.7627 ( 0.00%) 11.8683 ( 7.01%) Amean 265 13.8947 ( 0.00%) 13.4717 ( 3.04%) Amean 296 14.1093 ( 0.00%) 13.8130 ( 2.10%) The results can approximately divided into 3 sections: - busy, e.g. <12 groups on A and <21 groups on B - overloaded, e.g. 12~48 groups on A and 21~172 groups on B - saturated, the rest part For the busy part the result is neutral with slight wins or loss. It is probably because there are still idle cpus not hard to be find so the effort we paid for locating an idle core will bring limited benefit which can be negated by the cost of full scan easily. While for the overloaded but not saturated part, great improvement can be seen due to exploiting the cpu resources by more actively kicking idle cores working. But once all cpus are totally saturated, scanning for idle cores doesn't help much. One concern of the full scan is that the cost gets bigger in larger LLCs, but the test result seems still positive. One possible reason is due to the low SIS success rate (<2%), so the paid effort will indeed trade for efficiency. tbench4 Throughput (A) vanilla patched Hmean 1 275.61 ( 0.00%) 280.53 * 1.78%* Hmean 2 541.28 ( 0.00%) 561.94 * 3.82%* Hmean 4 1102.62 ( 0.00%) 1109.14 * 0.59%* Hmean 8 2149.58 ( 0.00%) 2229.39 * 3.71%* Hmean 16 4305.40 ( 0.00%) 4383.06 * 1.80%* Hmean 32 7088.36 ( 0.00%) 7124.14 * 0.50%* Hmean 64 8609.16 ( 0.00%) 8815.41 * 2.40%* Hmean 128 19304.92 ( 0.00%) 19519.35 * 1.11%* Hmean 256 19147.04 ( 0.00%) 19392.24 * 1.28%* Hmean 384 18970.86 ( 0.00%) 19201.07 * 1.21%* (B) Hmean 1 519.62 ( 0.00%) 515.98 * -0.70%* Hmean 2 1042.92 ( 0.00%) 1031.54 * -1.09%* Hmean 4 1959.10 ( 0.00%) 1953.44 * -0.29%* Hmean 8 3842.82 ( 0.00%) 3622.52 * -5.73%* Hmean 16 6768.50 ( 0.00%) 6545.82 * -3.29%* Hmean 32 12589.50 ( 0.00%) 13697.73 * 8.80%* Hmean 64 24797.23 ( 0.00%) 25589.59 * 3.20%* Hmean 128 38036.66 ( 0.00%) 35667.64 * -6.23%* Hmean 256 65069.93 ( 0.00%) 65215.85 * 0.22%* Hmean 512 61147.99 ( 0.00%) 66035.57 * 7.99%* Hmean 1024 48542.73 ( 0.00%) 53391.64 * 9.99%* The tbench4 test has a ~40% success rate on used target, prev or recent cpus, and ~45% of total success rate. And the core scan is also not very frequent, so the benefit this patch brings is limited while still some gains can be seen. netperf The netperf has an almost 100% success rate on used target, prev or recent cpus, so the domain scan is generally not performed and not affected by this patch. Conclusion ========== Taking full scan for idle cores is generally good for making better use of the cpu resources. Signed-off-by: Abel Wu Reviewed-by: Tim Chen Tested-by: Chen Yu --- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e7f82fa92c5b..7b668e16812e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6436,7 +6436,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool time = cpu_clock(this); } - if (sched_feat(SIS_UTIL)) { + if (sched_feat(SIS_UTIL) && !has_idle_core) { sd_share = rcu_dereference(per_cpu(sd_llc_shared, target)); if (sd_share) { /* because !--nr is the condition to stop scan */ From patchwork Wed Oct 19 12:28:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abel Wu X-Patchwork-Id: 5397 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp311285wrs; Wed, 19 Oct 2022 06:00:14 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4e4TdAUprLqfXoJNR+XnvPHgobUMDhlLQNveS4QBbzBYHNolt5hoHfxOVAh0u7enI9GrSi X-Received: by 2002:a17:902:dac4:b0:186:2df3:c84 with SMTP id q4-20020a170902dac400b001862df30c84mr8316777plx.116.1666184413505; Wed, 19 Oct 2022 06:00:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666184413; cv=none; d=google.com; s=arc-20160816; b=HdFyL+0P9EXicbbZG3o8dtqpdSSacLDHay6mywx6DSyX6dPcYb1BiKXbHSuI5N1UGd qhDLMZ8THsZu8PeBWJNIMA9ZamEZu+3o347iszMg6H/2zz/ZVrnY4N+d11F+RsNM98+L fhEaeCrlEtoexHnEDCPfnj7nPkysdUyFWYdt94L8D0s9gIStthcA2A+9HgIedbzHpYn9 dKlgmA+8xtfpd2uD9xO7wdcQ+ipcQmr9eSJTOCkJd4qWScY3t6VoRiFVZBlRAWmEJ2At WZLQ9E/rUWb7SZMlDumpq6zUYxOzNJslytF5VN0pfw/NvZHq4GScdWUa0Hv+cbgRb98F gafw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=b833A5Qsan0f1SmZzQdaOcnrkCwR8FuKT3y3L+uSymQ=; b=hd8Xpa6CBXZIrC7NiS9994eCizFJ0ocr53Ndc2NctNqsDmqZlOx9KnF2lLCdfNgLHS sh5Zk+8Fh+Rh5uysZ00nQMXL9ERpw8X7Sdc+rFYdnujaUWJTqY4Wbfeiwwr23c2mqQBN TLiU/kqWqY2okca9sTk+GJzZTGTSDWU/OX+cJdPI6bGzLSTIrwF/juZZxVqkByxx5e8a XWE9HACdMISHjsX3KGioJvR6itiiNYx9gvVs6xHB/r8lPC1PaGMBHNC2kymTr7agfGq3 8J++9RIqrGfQK6NfunZ/ioNUOtptMpLd0EIk49SrWr/FrXyBl/7ytDwf5+44x71gg545 wLow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=WElDJ3m+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h8-20020a170902f54800b0017693dbd8cdsi20100957plf.551.2022.10.19.05.59.59; Wed, 19 Oct 2022 06:00:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=WElDJ3m+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230262AbiJSMsg (ORCPT + 99 others); Wed, 19 Oct 2022 08:48:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233446AbiJSMr6 (ORCPT ); Wed, 19 Oct 2022 08:47:58 -0400 Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 970C77677 for ; Wed, 19 Oct 2022 05:30:47 -0700 (PDT) Received: by mail-pl1-x636.google.com with SMTP id l1so17068667pld.13 for ; Wed, 19 Oct 2022 05:30:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b833A5Qsan0f1SmZzQdaOcnrkCwR8FuKT3y3L+uSymQ=; b=WElDJ3m+gbu/8yW1wMY5/NQQpGk5O6qWPJ7IXza5ac6pMQSh+lfcyWuSS1azUMNNbI iEEA9LVKPTDZbtNMQQjhKqwSngFIk5ZFmvU6YnyAoPyTqkL6hbac1H/XY4u7SFKxNS3d 8dGtuILcsMW4ONrvg6ZrQVWBh+GEeeM/ZdsZfy7DuEulns/ueDeN4u946vhXxU+APxMQ H9ryMcesDUP56D7vwavxgTzr7tsXKCjLI9YUxEoHZ3JUdPGMJEfQX7+6o5JbWd+5wQeG 3jZoTU/xMDseD+DmCaUSv7ikidruz6TuJ2ncwfpzk/AiLUkRI0j66vgqAKp89l7FUalE QNow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b833A5Qsan0f1SmZzQdaOcnrkCwR8FuKT3y3L+uSymQ=; b=4w+33OJHqMNPTc8ySslCXc44n26iBUxlmpkVlBUdRznNRqulCejeXdH0Ts0YWWT0ft TvTg5S9J4+7shAvHmuJSdNYnAdBi+WOdt5oiLCmERRVCJxuRxnhsg9yTO05/aIrRN5We P5JRFYI3pNczWp9kH5dt7ieVExq3kP3NnaOVPgS5T68D417GZTHhX6fsDDzB31ftMjTE 8csMSAFOb+HHYXvKu+jd3jSEYWitfkTAAhNgoh1OC5Mm/xmc3tv8AoLGUkYx0jfsl6FM eezK8SPYVkwyzavBFSifDQRj2Snp2JYhAq+TCCAinDELaAyTrVp/27MlQ0kQDQsoVyZN ZLEQ== X-Gm-Message-State: ACrzQf20rA/eYLrI/gQJ4FH/CmmHedbcaBfMV+szP7782M5GsWqQsPYr 9V2OW3xqqiKoiYQsG7cKiKGBiw== X-Received: by 2002:a17:902:e848:b0:180:c732:1e52 with SMTP id t8-20020a170902e84800b00180c7321e52mr8378443plg.83.1666182589286; Wed, 19 Oct 2022 05:29:49 -0700 (PDT) Received: from C02DV8HUMD6R.bytedance.net ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id c21-20020a63da15000000b00439c6a4e1ccsm9881825pgh.62.2022.10.19.05.29.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Oct 2022 05:29:49 -0700 (PDT) From: Abel Wu To: Peter Zijlstra , Ingo Molnar , Mel Gorman , Vincent Guittot , Dietmar Eggemann , Valentin Schneider Cc: Josh Don , Chen Yu , Tim Chen , K Prateek Nayak , "Gautham R . Shenoy" , Aubrey Li , Qais Yousef , Juri Lelli , Rik van Riel , Yicong Yang , Barry Song <21cnbao@gmail.com>, linux-kernel@vger.kernel.org, Abel Wu Subject: [PATCH v6 3/4] sched/fair: Introduce SIS_CORE Date: Wed, 19 Oct 2022 20:28:58 +0800 Message-Id: <20221019122859.18399-4-wuyun.abel@bytedance.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019122859.18399-1-wuyun.abel@bytedance.com> References: <20221019122859.18399-1-wuyun.abel@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747120987573147788?= X-GMAIL-MSGID: =?utf-8?q?1747120987573147788?= The wakeup fastpath for fair tasks, select_idle_sibling() aka SIS, plays an important role in maximizing the usage of cpu resources and can greatly affect overall performance of the system. The SIS tries to find an idle cpu inside the targeting LLC to place the woken-up task. The cache hot cpus are preferred, but if none of them is idle, a domain scan is fired to check the other cpus of that LLC. It currently scans in linear fashion, and it works well under light pressure due to lots of idle cpus can be available. But things change. The LLC is getting bigger in modern and future machines, and players like cloud service providers are continuously trying to use all kinds of resources more efficiently to reduce TCO. In either case, locating an idle cpu is no longer as easy as before. So the linear scan doesn't fit well in such cases. There are already features like SIS_{UTIL,PROP} exist to deal with the scalability issue by limiting the scan depth, and it would be better if we can improve the way how it scans as well. And this is exactly what the SIS_CORE is born for. When SIS_CORE is enabled, a cpumask containing idle cpus for each LLC is maintained and stored in each LLC shared domain. The idle cpus are recorded at CORE granule, so in theory only one idle cpu of a CORE can be set to the mask. The ideas behind this: - Recording idle cpus is to narrow down the SIS scan, so we can avoid touching the runqueues that must be in a busy state, as we all know that the runqueues are one of the most hot data in the system. And because all the possibly idle cpus are in the mask, the hint of has_idle_core still works. - The rule of CORE granule update helps spreading load out to different cores trying to make better use of cpu capacity. A major concern is the accuracy of the idle cpumask. A cpu present in the mask might not be idle any more, which is called the false positive cpu. Such cpus will negate lots of benefit this feature brings. The strategy against the false positives will be introduced in next patch. Another concern is the overhead of accessing the LLC-shared cpumask, which can be more severe in large LLCs. But a perf stat on cache miss rate during hackbench doesn't show obvious difference. This patch records idle cpus when they goes idle at CORE granule for each LLC, and the cpumask is in LLC shared domain. The false positive cpus are cleared when SIS domain scan fails. Signed-off-by: Abel Wu --- include/linux/sched/topology.h | 15 ++++++++++ kernel/sched/fair.c | 51 +++++++++++++++++++++++++++++++--- kernel/sched/features.h | 7 +++++ kernel/sched/topology.c | 8 +++++- 4 files changed, 76 insertions(+), 5 deletions(-) diff --git a/include/linux/sched/topology.h b/include/linux/sched/topology.h index 816df6cc444e..ac2162f33ada 100644 --- a/include/linux/sched/topology.h +++ b/include/linux/sched/topology.h @@ -82,6 +82,16 @@ struct sched_domain_shared { atomic_t nr_busy_cpus; int has_idle_cores; int nr_idle_scan; + + /* + * Used by sched feature SIS_CORE to record idle cpus at core + * granule to improve efficiency of SIS domain scan. + * + * NOTE: this field is variable length. (Allocated dynamically + * by attaching extra space to the end of the structure, + * depending on how many CPUs the kernel has booted up with) + */ + unsigned long icpus[]; }; struct sched_domain { @@ -167,6 +177,11 @@ static inline struct cpumask *sched_domain_span(struct sched_domain *sd) return to_cpumask(sd->span); } +static inline struct cpumask *sched_domain_icpus(struct sched_domain *sd) +{ + return to_cpumask(sd->shared->icpus); +} + extern void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new[], struct sched_domain_attr *dattr_new); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 7b668e16812e..3aa699e9d4af 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6282,6 +6282,33 @@ static inline bool test_idle_cores(int cpu) return false; } +/* + * To honor the rule of CORE granule update, set this cpu to the LLC idle + * cpumask only if there is no cpu of this core showed up in the cpumask. + */ +static void update_idle_cpu(int cpu) +{ + struct sched_domain_shared *sds; + + if (!sched_feat(SIS_CORE)) + return; + + sds = rcu_dereference(per_cpu(sd_llc_shared, cpu)); + if (sds) { + struct cpumask *icpus = to_cpumask(sds->icpus); + + /* + * This is racy against clearing in select_idle_cpu(), + * and can lead to idle cpus miss the chance to be set to + * the idle cpumask, thus the idle cpus are temporarily + * out of reach in SIS domain scan. But it should be rare + * and we still have ILB to kick them working. + */ + if (!cpumask_intersects(cpu_smt_mask(cpu), icpus)) + cpumask_set_cpu(cpu, icpus); + } +} + /* * Scans the local SMT mask to see if the entire core is idle, and records this * information in sd_llc_shared->has_idle_cores. @@ -6298,6 +6325,7 @@ void __update_idle_core(struct rq *rq) return; rcu_read_lock(); + update_idle_cpu(core); if (test_idle_cores(core)) goto unlock; @@ -6343,7 +6371,13 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu if (idle) return core; - cpumask_andnot(cpus, cpus, cpu_smt_mask(core)); + /* + * It is unlikely that more than one cpu of a core show up + * in the @cpus if SIS_CORE enabled. + */ + if (!sched_feat(SIS_CORE)) + cpumask_andnot(cpus, cpus, cpu_smt_mask(core)); + return -1; } @@ -6394,7 +6428,7 @@ static inline int select_idle_smt(struct task_struct *p, int target) */ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool has_idle_core, int target) { - struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask); + struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask), *icpus = NULL; int i, cpu, idle_cpu = -1, nr = INT_MAX; struct sched_domain_shared *sd_share; struct rq *this_rq = this_rq(); @@ -6402,8 +6436,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool struct sched_domain *this_sd = NULL; u64 time = 0; - cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); - if (sched_feat(SIS_PROP) && !has_idle_core) { u64 avg_cost, avg_idle, span_avg; unsigned long now = jiffies; @@ -6447,6 +6479,11 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool } } + if (sched_feat(SIS_CORE) && sched_smt_active()) + icpus = sched_domain_icpus(sd); + + cpumask_and(cpus, icpus ? icpus : sched_domain_span(sd), p->cpus_ptr); + for_each_cpu_wrap(cpu, cpus, target + 1) { if (has_idle_core) { i = select_idle_core(p, cpu, cpus, &idle_cpu); @@ -6465,6 +6502,12 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool if (has_idle_core) set_idle_cores(target, false); + if (icpus && idle_cpu == -1) { + /* Reset the idle cpu mask if a full scan fails */ + if (nr > 0) + cpumask_clear(icpus); + } + if (sched_feat(SIS_PROP) && this_sd && !has_idle_core) { time = cpu_clock(this) - time; diff --git a/kernel/sched/features.h b/kernel/sched/features.h index ee7f23c76bd3..bf3cae94caa6 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -63,6 +63,13 @@ SCHED_FEAT(TTWU_QUEUE, true) SCHED_FEAT(SIS_PROP, false) SCHED_FEAT(SIS_UTIL, true) +/* + * Record idle cpus at core granule for each LLC to improve efficiency of + * SIS domain scan. Combine with the above features of limiting scan depth + * to better deal with the scalability issue. + */ +SCHED_FEAT(SIS_CORE, true) + /* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 8739c2a5a54e..a2bb0091c10d 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -1641,6 +1641,12 @@ sd_init(struct sched_domain_topology_level *tl, sd->shared = *per_cpu_ptr(sdd->sds, sd_id); atomic_inc(&sd->shared->ref); atomic_set(&sd->shared->nr_busy_cpus, sd_weight); + + /* + * This will temporarily break the rule of CORE granule, + * but will be fixed after SIS scan failures. + */ + cpumask_copy(sched_domain_icpus(sd), sd_span); } sd->private = sdd; @@ -2106,7 +2112,7 @@ static int __sdt_alloc(const struct cpumask *cpu_map) *per_cpu_ptr(sdd->sd, j) = sd; - sds = kzalloc_node(sizeof(struct sched_domain_shared), + sds = kzalloc_node(sizeof(struct sched_domain_shared) + cpumask_size(), GFP_KERNEL, cpu_to_node(j)); if (!sds) return -ENOMEM; From patchwork Wed Oct 19 12:28:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Abel Wu X-Patchwork-Id: 5408 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp312921wrs; Wed, 19 Oct 2022 06:02:27 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6g4fztQsSFlYsD1rXubfCdy5/z73vDoM/wSGkZAvAYzwllnpAhWPcW3QDLnfttCgV2sRb2 X-Received: by 2002:a63:171b:0:b0:438:e47d:e79b with SMTP id x27-20020a63171b000000b00438e47de79bmr7016607pgl.542.1666184546777; Wed, 19 Oct 2022 06:02:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666184546; cv=none; d=google.com; s=arc-20160816; b=yIZNB2MD1YNlTt080yZMvoCHAtcWiZ5mcV+YDM6CGqHSS/xHwWa2xgT3KjUiCM2dv1 stN9WsN+9z8G4zahYJ/WbtlBuT4dDJes8RUjGH+8v1uAiBpzl8mnniaPvb559lGXZa4L l8jpf6mVVz7dtbC6PhE6dRFzAtQoj5IAJmln/+Wyj7J9OOqNd+A8bEfjwXn4YxU3WxBR y8PGnbHpN+KT0fpuT/veoNyAXX4Z76VZioIwzobaTtybWUwO7YeIMnEMYt+RvvNJ2mPB 59HfvG5f+C4gg2J5laP/4T4+7kMABXok6fm7k8bw+2Z2CqYDFRT7W+h+wI9IsNuzKR1H fBMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=bZgcfNdayfFXkcge4kHlidh3EZUZ+6CNq8c/La+Qf7c=; b=UgAbAdKlzcbKjDqpWYEQlC+6kiLWF9lyO3voOXGKV39W5NGVZjvQWRyF47iHrSdGS4 F7H8HheZn+7Ih0Bh5wGUohnUBxZ8G2tvbCETP+WvMLklL1PXdA23zpKFkgMZEJxyW1JZ UB3V2S9+RrFruvl6Oowo1QnEvMtOhykS/myDJ+Ym/waphV5ISn4AQnkdnjxi0XsrtDd4 q1F+yeYlcoIgdM3y9lTWb2pggktINH86+eIlZKNTbDrejWb6A7gONliAMZjaqh74pQL+ 1bV9GO4+u81U13T4O2WaST4F/ahRHHIZVxbww7UnysKI/VSRtrTDsadqfOIwOHPg+7A/ H1xQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=EN2G7yth; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s1-20020a62e701000000b0056196446366si16493689pfh.226.2022.10.19.06.01.56; Wed, 19 Oct 2022 06:02:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=EN2G7yth; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231747AbiJSMs2 (ORCPT + 99 others); Wed, 19 Oct 2022 08:48:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56166 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233226AbiJSMrn (ORCPT ); Wed, 19 Oct 2022 08:47:43 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AD8B2D753 for ; Wed, 19 Oct 2022 05:30:49 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id b5so16113262pgb.6 for ; Wed, 19 Oct 2022 05:30:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bZgcfNdayfFXkcge4kHlidh3EZUZ+6CNq8c/La+Qf7c=; b=EN2G7ythNaQoql5lJ6kdcazc6m+WelLAL7ze6ZExNo9PAdeoHo6G9r7AGUNMf/TKoz yy1LhFnh2OJ+NSjy3CbBdG9LhQoFybqSCj8IDBQ2svmkqof1AaCCFtKHdBlEos2KI1cn ylLga1qosHEmiJQK1ErdOD3SzOPBmFbTHYZJT8s4wV8ay7HaEapkKF7NQHT+gwcVW47U +iDL5aQEqY1OJpuvU3MSpZVR+heP46DOzmPQtvpsynXkB82BIR1+bR3NUs6ar35O+4AE qpoN8lX86nLDlbu1MjEY+gDIR9AfASAAbZ/OzfqGvlDfydPSpbLfzrUqJN6oL5D+dsxG kEhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bZgcfNdayfFXkcge4kHlidh3EZUZ+6CNq8c/La+Qf7c=; b=gPH/EJGA/rWwEDzGiSRMkn6Ijup+eU8+rPIXh/h+5AC9ImBzOTMfLxNceUjy0+ckco NDBjQJ/C40AOQsCXwaS9544R0v6OsjWsAJhovLQVhOABEz1SM6wNcc9b1lonUNQmjtdK N+DjIjkAKojWsmWi1VcrTk6iQqjKS6fwQxSrANEP61r1MUupHhAJf+XjJ2GkdbkVKUdC lPvGbYBylxP8oFR3myChoUa4GUeN42BmrXO2aLkwMbYNPjzPIDo8TI4uqYApbhnEjQIA AXd0rYwswqEcalG9i/Tozt6W5IdjVWWLR1a3U72nvRTcNnDir8jFEbGL1gVR+puHv/Ad 2oQw== X-Gm-Message-State: ACrzQf2xLsWyq1e/8L5VVb+CYutFw3fWuMM9tmna0VAzzysFC9Kj/WVD BkaZLQozZmwAlEwRRC6l42x+1Q== X-Received: by 2002:a63:da4f:0:b0:43f:6af:74ed with SMTP id l15-20020a63da4f000000b0043f06af74edmr7142827pgj.290.1666182595892; Wed, 19 Oct 2022 05:29:55 -0700 (PDT) Received: from C02DV8HUMD6R.bytedance.net ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id c21-20020a63da15000000b00439c6a4e1ccsm9881825pgh.62.2022.10.19.05.29.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 19 Oct 2022 05:29:55 -0700 (PDT) From: Abel Wu To: Peter Zijlstra , Ingo Molnar , Mel Gorman , Vincent Guittot , Dietmar Eggemann , Valentin Schneider Cc: Josh Don , Chen Yu , Tim Chen , K Prateek Nayak , "Gautham R . Shenoy" , Aubrey Li , Qais Yousef , Juri Lelli , Rik van Riel , Yicong Yang , Barry Song <21cnbao@gmail.com>, linux-kernel@vger.kernel.org, Abel Wu Subject: [PATCH v6 4/4] sched/fair: Deal with SIS scan failures Date: Wed, 19 Oct 2022 20:28:59 +0800 Message-Id: <20221019122859.18399-5-wuyun.abel@bytedance.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221019122859.18399-1-wuyun.abel@bytedance.com> References: <20221019122859.18399-1-wuyun.abel@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747121127289200256?= X-GMAIL-MSGID: =?utf-8?q?1747121127289200256?= When SIS_CORE is active, idle cpus are recorded into a cpumask at CORE granule, and the SIS domain scan tries to locate an idle cpu in this cpumask. So the quality of the cpumask matters. There are generally two types of cpus that need to be dealt with: - False positive: the cpus that are present in the idle cpumask but is actually busy. This can be caused by task enqueuing to these cpus without removing from the cpumask immediately. To solve this, the cpumask needs to be shrinked if poor quality of the mask is observed. - True negative: the cpus that are idle but not show up in the idle cpumask. This is due to the rule of CORE granule update that these idle cpus will be ignored when their SMT siblings are already present in the idle cpumask. According to the nature of the two type cpus and some heuristics, the strategies against SIS scan failures are as follows: - It can be predicted that if a scan fails, the next scan from the same cpu will probably fail too. So mark these cpus fail-prone. A scan from fail-prone cpu is the time to shrink the cpumask. - All true negative cpus are the SMT siblings of the false positive cpus, and are taken for granted to be treated as the fallbacks of the false positive cpus. So a fail-prone scan should also check the SMT siblings to see if any true negative cpu exists. The number of false positive cpus being removed during one scan is not constrained, but it will be implicitly constrained by the load of the LLC. If LLC is under heavy pressure, both the weight of idle cpumask and the scan depth are reduced, so does the number of cpus removed. To sum up, this patch marks the cpus fail-prone if scans from them failed last time, so next time scanning from them will also check their SMT siblings to see if any true negative cpus available. The false positive cpus will be removed during fail-prone scans if its belonging core is fully busy. Benchmark ========= All of the benchmarks are done inside a normal cpu cgroup in a clean environment with cpu turbo disabled, and test machines are: A) A dual socket machine modeled Intel Xeon(R) Platinum 8260 with SNC disabled, so there are 2 NUMA nodes each of which has 24C/48T. Each NUMA shares an LLC. B) A dual socket machine modeled AMD EPYC 7Y83 64-Core Processor with NPS1 enabled, so there are 2 NUMA nodes each of which has 64C/128T. Each NUMA node contains several LLCs sized of 16 cpus. The 'baseline' is based on tip sched/core fb04563d1cae (v5.19.0) with the first 2 patches of this series applied. While 'sis_core' includes the whole patchset. Results ======= hackbench-process-pipes (A) baseline sis_core Amean 1 0.2540 ( 0.00%) 0.2533 ( 0.26%) Amean 4 0.6220 ( 0.00%) 0.5993 ( 3.64%) Amean 7 0.8020 ( 0.00%) 0.7663 * 4.45%* Amean 12 1.1823 ( 0.00%) 1.1037 * 6.65%* Amean 21 2.7717 ( 0.00%) 2.2203 * 19.89%* Amean 30 5.1200 ( 0.00%) 3.7133 * 27.47%* Amean 48 8.5890 ( 0.00%) 7.0863 * 17.50%* Amean 79 11.2580 ( 0.00%) 10.3717 * 7.87%* Amean 110 13.1283 ( 0.00%) 12.4133 * 5.45%* Amean 141 15.5967 ( 0.00%) 14.4883 * 7.11%* Amean 172 18.1153 ( 0.00%) 17.2557 * 4.75%* Amean 203 21.1340 ( 0.00%) 20.2807 ( 4.04%) Amean 234 23.8227 ( 0.00%) 22.8510 ( 4.08%) Amean 265 26.8293 ( 0.00%) 25.7367 * 4.07%* Amean 296 29.5800 ( 0.00%) 28.7847 ( 2.69%) (B) Amean 1 0.3650 ( 0.00%) 0.3510 ( 3.84%) Amean 4 0.4837 ( 0.00%) 0.4753 ( 1.72%) Amean 7 0.4997 ( 0.00%) 0.5073 ( -1.53%) Amean 12 0.5863 ( 0.00%) 0.5807 ( 0.97%) Amean 21 0.8930 ( 0.00%) 0.8953 ( -0.26%) Amean 30 1.2530 ( 0.00%) 1.2633 ( -0.82%) Amean 48 1.9743 ( 0.00%) 1.9023 ( 3.65%) Amean 79 3.4933 ( 0.00%) 3.2820 ( 6.05%) Amean 110 5.5963 ( 0.00%) 5.3923 ( 3.65%) Amean 141 7.6550 ( 0.00%) 6.8633 ( 10.34%) Amean 172 8.8323 ( 0.00%) 8.2973 * 6.06%* Amean 203 10.8683 ( 0.00%) 9.5170 * 12.43%* Amean 234 11.8683 ( 0.00%) 10.6217 ( 10.50%) Amean 265 13.4717 ( 0.00%) 11.9357 * 11.40%* Amean 296 13.8130 ( 0.00%) 12.7430 * 7.75%* The results on machine A can approximately divided into 3 sections: - busy, e.g. <21 groups - overloaded, e.g. 21~48 groups - saturated, the rest part The two cases of 296 groups in B and 110 groups in A have same amount of tasks per cpu. So does 30 groups in B and 12 groups in A. So the sections on A also apply to B, except that B only has the first two. This implies that this feature seems to have consistant behaviour on LLCs with different sizes. For the busy part the result is neutral with slight wins or loss. It's not hard to find an idle cpu in such case, so SIS_CORE doesn't outperform the linear scanner, considering the cpumask is maintained at a cost which will negate the slight benefit. Once load increases, SIS_CORE helps improving the throughput quite a lot by squeezing out the hidden capacity of the cpus. And even under extreme load pressure when the cpu capacity is almost fully utilized, there is still some can be exploited. Although it is unlikely to be such loaded in the real world, the long running workloads like training jobs can also keep cpus busy and can benefit from this feature a lot. netperf (A-udp) baseline sis_core Hmean send-64 214.34 ( 0.00%) 210.79 * -1.65%* Hmean send-128 427.90 ( 0.00%) 417.96 * -2.32%* Hmean send-256 839.65 ( 0.00%) 823.78 * -1.89%* Hmean send-1024 3207.45 ( 0.00%) 3167.96 * -1.23%* Hmean send-2048 6097.24 ( 0.00%) 6089.01 ( -0.13%) Hmean send-3312 9350.83 ( 0.00%) 9299.09 ( -0.55%) Hmean send-4096 11368.25 ( 0.00%) 11186.44 * -1.60%* Hmean send-8192 18273.21 ( 0.00%) 18103.81 ( -0.93%) Hmean send-16384 28207.81 ( 0.00%) 28259.82 ( 0.18%) (B-udp) Hmean send-64 249.97 ( 0.00%) 256.99 * 2.81%* Hmean send-128 500.68 ( 0.00%) 514.44 * 2.75%* Hmean send-256 991.59 ( 0.00%) 1017.38 * 2.60%* Hmean send-1024 3913.02 ( 0.00%) 3982.68 * 1.78%* Hmean send-2048 7627.99 ( 0.00%) 7590.30 ( -0.49%) Hmean send-3312 11907.07 ( 0.00%) 12114.03 * 1.74%* Hmean send-4096 14300.09 ( 0.00%) 14753.34 * 3.17%* Hmean send-8192 24576.21 ( 0.00%) 25431.42 * 3.48%* Hmean send-16384 42105.89 ( 0.00%) 41813.30 ( -0.69%) (A-tcp) Hmean 64 1191.91 ( 0.00%) 1220.47 * 2.40%* Hmean 128 2318.60 ( 0.00%) 2354.56 ( 1.55%) Hmean 256 4267.41 ( 0.00%) 4226.72 ( -0.95%) Hmean 1024 13190.66 ( 0.00%) 13065.91 ( -0.95%) Hmean 2048 20466.22 ( 0.00%) 20704.66 ( 1.17%) Hmean 3312 24363.57 ( 0.00%) 24613.99 * 1.03%* Hmean 4096 26144.44 ( 0.00%) 26204.24 ( 0.23%) Hmean 8192 30387.77 ( 0.00%) 30703.65 * 1.04%* Hmean 16384 34942.71 ( 0.00%) 34205.44 * -2.11%* (B-tcp) Hmean 64 1971.18 ( 0.00%) 2120.61 * 7.58%* Hmean 128 3752.96 ( 0.00%) 3995.68 * 6.47%* Hmean 256 6861.58 ( 0.00%) 7342.93 * 7.02%* Hmean 1024 21966.06 ( 0.00%) 23725.30 * 8.01%* Hmean 2048 33949.66 ( 0.00%) 35620.67 * 4.92%* Hmean 3312 40681.75 ( 0.00%) 41543.26 * 2.12%* Hmean 4096 44309.70 ( 0.00%) 45390.03 * 2.44%* Hmean 8192 50909.35 ( 0.00%) 52157.16 * 2.45%* Hmean 16384 57198.37 ( 0.00%) 57686.96 ( 0.85%) The netperf has an almost 100% success rate on used target, prev or recent cpus, so the domain scan is generally not performed. This test is to see how much overhead of maintaining the idle cpumask and the result is neutral, which sounds like the overhead is acceptable. tbench4 Throughput (A) baseline sis_core Hmean 1 280.53 ( 0.00%) 289.44 * 3.17%* Hmean 2 561.94 ( 0.00%) 571.46 * 1.69%* Hmean 4 1109.14 ( 0.00%) 1129.88 * 1.87%* Hmean 8 2229.39 ( 0.00%) 2266.52 * 1.67%* Hmean 16 4383.06 ( 0.00%) 4473.48 * 2.06%* Hmean 32 7124.14 ( 0.00%) 7223.83 * 1.40%* Hmean 64 8815.41 ( 0.00%) 8770.21 * -0.51%* Hmean 128 19519.35 ( 0.00%) 20337.24 * 4.19%* Hmean 256 19392.24 ( 0.00%) 20052.98 * 3.41%* Hmean 384 19201.07 ( 0.00%) 19563.63 * 1.89%* (B) Hmean 1 515.98 ( 0.00%) 499.91 * -3.12%* Hmean 2 1031.54 ( 0.00%) 1044.38 * 1.24%* Hmean 4 1953.44 ( 0.00%) 1959.30 * 0.30%* Hmean 8 3622.52 ( 0.00%) 3773.08 * 4.16%* Hmean 16 6545.82 ( 0.00%) 6814.46 * 4.10%* Hmean 32 13697.73 ( 0.00%) 13078.74 * -4.52%* Hmean 64 25589.59 ( 0.00%) 24576.52 * -3.96%* Hmean 128 35667.64 ( 0.00%) 37590.20 * 5.39%* Hmean 256 65215.85 ( 0.00%) 64921.74 * -0.45%* Hmean 512 66035.57 ( 0.00%) 63812.48 * -3.37%* Hmean 1024 53391.64 ( 0.00%) 62356.50 * 16.79%* Like netperf, tbench4 benchmark also has a high success rate ~39% on the cache hot cpus, and the SIS overall success rate is ~45%. This benchmark runs a fast idling workload and makes the SIS idle cpumask quite unstable, which is a disaster to this feature. Even so, not much but still some improvement can be seen from the result. Conclusion ========== The overhead of maintaining the idle cpumasks seems acceptable, and this cost can trade for considerable throughput improvement once LLC becomes busier in which case less idle cpus are available. Signed-off-by: Abel Wu --- kernel/sched/fair.c | 74 +++++++++++++++++++++++++++++++++++++------- kernel/sched/sched.h | 3 ++ 2 files changed, 65 insertions(+), 12 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3aa699e9d4af..d06d59ac2f05 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6309,6 +6309,16 @@ static void update_idle_cpu(int cpu) } } +static inline bool should_scan_sibling(int cpu) +{ + return cmpxchg(&cpu_rq(cpu)->sis_scan_sibling, 1, 0); +} + +static inline void set_scan_sibling(int cpu) +{ + WRITE_ONCE(cpu_rq(cpu)->sis_scan_sibling, 1); +} + /* * Scans the local SMT mask to see if the entire core is idle, and records this * information in sd_llc_shared->has_idle_cores. @@ -6384,17 +6394,20 @@ static int select_idle_core(struct task_struct *p, int core, struct cpumask *cpu /* * Scan the local SMT mask for idle CPUs. */ -static int select_idle_smt(struct task_struct *p, int target) +static int select_idle_smt(struct task_struct *p, int core, struct cpumask *cpus, int exclude) { int cpu; - for_each_cpu_and(cpu, cpu_smt_mask(target), p->cpus_ptr) { - if (cpu == target) + for_each_cpu_and(cpu, cpu_smt_mask(core), p->cpus_ptr) { + if (exclude && cpu == core) continue; if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) return cpu; } + if (cpus) + cpumask_clear_cpu(core, cpus); + return -1; } @@ -6409,12 +6422,21 @@ static inline bool test_idle_cores(int cpu) return false; } +static inline bool should_scan_sibling(int cpu) +{ + return false; +} + +static inline void set_scan_sibling(int cpu) +{ +} + static inline int select_idle_core(struct task_struct *p, int core, struct cpumask *cpus, int *idle_cpu) { return __select_idle_cpu(core, p); } -static inline int select_idle_smt(struct task_struct *p, int target) +static inline int select_idle_smt(struct task_struct *p, int core, struct cpumask *cpus, int exclude) { return -1; } @@ -6434,6 +6456,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool struct rq *this_rq = this_rq(); int this = smp_processor_id(); struct sched_domain *this_sd = NULL; + bool scan_sibling = false; u64 time = 0; if (sched_feat(SIS_PROP) && !has_idle_core) { @@ -6479,20 +6502,31 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool } } - if (sched_feat(SIS_CORE) && sched_smt_active()) + if (sched_feat(SIS_CORE) && sched_smt_active()) { + /* + * Due to the nature of idle core scanning, has_idle_core + * hint should also consume the scan_sibling flag even + * though it doesn't use the flag when scanning. + */ + scan_sibling = should_scan_sibling(target); icpus = sched_domain_icpus(sd); + } cpumask_and(cpus, icpus ? icpus : sched_domain_span(sd), p->cpus_ptr); for_each_cpu_wrap(cpu, cpus, target + 1) { + if (!--nr) + break; + if (has_idle_core) { i = select_idle_core(p, cpu, cpus, &idle_cpu); if ((unsigned int)i < nr_cpumask_bits) return i; - + } else if (scan_sibling) { + idle_cpu = select_idle_smt(p, cpu, icpus, 0); + if ((unsigned int)idle_cpu < nr_cpumask_bits) + break; } else { - if (!--nr) - return -1; idle_cpu = __select_idle_cpu(cpu, p); if ((unsigned int)idle_cpu < nr_cpumask_bits) break; @@ -6503,9 +6537,25 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool set_idle_cores(target, false); if (icpus && idle_cpu == -1) { - /* Reset the idle cpu mask if a full scan fails */ - if (nr > 0) - cpumask_clear(icpus); + if (nr > 0 && (has_idle_core || scan_sibling)) { + /* + * Reset the idle cpu mask if a full scan fails, + * but ignore the !has_idle_core case which should + * have already been fixed during scan. + */ + if (has_idle_core) + cpumask_clear(icpus); + } else { + /* + * As for partial scan failures, it will probably + * fail again next time scanning from the same cpu. + * Due to the SIS_CORE rule of CORE granule update, + * some idle cpus can be missed in the mask. So it + * would be reasonable to scan SMT siblings as well + * if the scan is fail-prone. + */ + set_scan_sibling(target); + } } if (sched_feat(SIS_PROP) && this_sd && !has_idle_core) { @@ -6657,7 +6707,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) has_idle_core = test_idle_cores(target); if (!has_idle_core && cpus_share_cache(prev, target)) { - i = select_idle_smt(p, prev); + i = select_idle_smt(p, prev, NULL, 1); if ((unsigned int)i < nr_cpumask_bits) return i; } diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 1fc198be1ffd..c7f8ed5021e6 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -971,6 +971,9 @@ struct rq { #ifdef CONFIG_SMP unsigned int ttwu_pending; +#ifdef CONFIG_SCHED_SMT + int sis_scan_sibling; +#endif #endif u64 nr_switches;