From patchwork Tue Aug 1 20:41:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129497 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp34859vqx; Tue, 1 Aug 2023 14:56:09 -0700 (PDT) X-Google-Smtp-Source: APBJJlHyPCMpexjpP/LOqlrskRnn/QXEUI/TufIxMb7Kso5pG3tGNkgXaAaMMu8Q3qT0XWxp0S5p X-Received: by 2002:a05:6a00:1394:b0:666:c1ae:3b87 with SMTP id t20-20020a056a00139400b00666c1ae3b87mr21488874pfg.12.1690926968943; Tue, 01 Aug 2023 14:56:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690926968; cv=none; d=google.com; s=arc-20160816; b=JByL6Z6iFu1/Yj0Hz/IbaN93Vd15Pk47tkyZX7aJqCBVRBg8LR0Mc/VI8cQ/AIdGAD 9HSEcoy5Wr8LhntoJLxMiTDejm8zMJWnjf9RLio3Cj4n/Bxo7HaOq/kNvfxyxmd3N6cj svSCYobIOuFkIflSdz0Wk28npDvanVsIqj0IpLfUXecWQgPzuWoBtYwHhhaxPYWMLMzX I4xoXlHAI1ijSMyFN0/V6NRdF9U2AYfdiqG7uzhrJz7qB0Nwo3PcVvNfytsdi1O3huWR dqssVxFWeTDBTBQiVlTah3Tht7RIahypmYo3OReJ7fgc5WKstgx4zyGYQd2C4pWeRiPm fusA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=2Fw/6pc13hFz4431UNCf/XS3M9fAIWcYfqPnXE09Ku4=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=Ytk6c0ECb89L4ex9pZJ+0KKBuU+zUQ5/MN9c+2DSVxSnBriNrtqZVkT1/NOqDVtJ5V 8jIo4bzX/D/wLzHi5keQdrEInNK1flSnbnqT9rrVOPBiFEb+DFv85uWQ6EpOMiat6Wis mIX4IfstXPFUqCz+jPIMmjvkWiClB2ANYBqmKH3BwxyS0ngT5GeHeislNiiT5k6kACDN 5+ddK6/vODHG8xN8oi7wWL2E0JOq0G3XzZKFA8UF9ObRxf0CJWVeU9pY9m89ng0V+O3Q vs+4qO3oB9QMuMv4JFvp+Dm7AgKWAxFBkmm9gbbTl7Ecj+uj/25L4OMhEKUriG4ZeHmT KJTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=jf7h3twP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cv12-20020a056a0044cc00b0068761c460bdsi215279pfb.32.2023.08.01.14.55.55; Tue, 01 Aug 2023 14:56:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=jf7h3twP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232512AbjHAVZ3 (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230241AbjHAVY4 (ORCPT ); Tue, 1 Aug 2023 17:24:56 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 062492708 for ; Tue, 1 Aug 2023 14:24:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=2Fw/6pc13hFz4431UNCf/XS3M9fAIWcYfqPnXE09Ku4=; b=jf7h3twPdz2DT1GCwfUgdsHWvl 6gJ6KV5WtDzaSrLPLVmGcSsNI/aPU6YuG/xgdRcISp/8cdMuxLDYSSOACzvce0Xin0ei5eurCCVW6 pKhIniOu3a6lY3JXgczXG8BA4bNwQ7ibysONv7Fklk9bZ4pMTaX6+Jhxsn/5ua9oyqgzxNHtotBA2 m+Rej9y/VI678MskQgPdCshVY4X6Qmnwpqj76M6A/AW1o7OgItWsLTAKeedBx6E19GFALEo58HwfS 6ZyZ0DSbdXMfYqFv4nb7aPil+r8I1F8t7OlD0qIUnWQHwn4lbg194c89YhUGFLlx1PCBGeyPtrqCd QdZAOqSg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qQwqm-00B5yX-0o; Tue, 01 Aug 2023 21:24:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 569F43002D3; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 389C52028A2D8; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211811.828443100@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:22 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 1/9] sched: Simplify get_nohz_timer_target() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773065437466724073 X-GMAIL-MSGID: 1773065437466724073 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Joel Fernandes (Google) --- kernel/sched/core.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1097,25 +1097,22 @@ int get_nohz_timer_target(void) hk_mask = housekeeping_cpumask(HK_TYPE_TIMER); - rcu_read_lock(); + guard(rcu)(); + for_each_domain(cpu, sd) { for_each_cpu_and(i, sched_domain_span(sd), hk_mask) { if (cpu == i) continue; - if (!idle_cpu(i)) { - cpu = i; - goto unlock; - } + if (!idle_cpu(i)) + return i; } } if (default_cpu == -1) default_cpu = housekeeping_any_cpu(HK_TYPE_TIMER); - cpu = default_cpu; -unlock: - rcu_read_unlock(); - return cpu; + + return default_cpu; } /* From patchwork Tue Aug 1 20:41:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129501 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp41848vqx; Tue, 1 Aug 2023 15:08:37 -0700 (PDT) X-Google-Smtp-Source: APBJJlE4pWbQwj2AnJ57K4u95lSEu9HlGRiSCatDkwOWp+0mWxysEpdTfe8DsmsaZRzjnK22EKdu X-Received: by 2002:a05:6a20:8e0e:b0:135:293b:9b14 with SMTP id y14-20020a056a208e0e00b00135293b9b14mr17615726pzj.55.1690927717522; Tue, 01 Aug 2023 15:08:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690927717; cv=none; d=google.com; s=arc-20160816; b=yGzPT2Ehvl549EjjItSVq0XrNDRfpdKqRmSZKkcEv2SMDQV0bAfZUCz3kfXSinKra2 JQJPUWditHfU/Yi3Qx4gMB9NZs3H5p+G9NPux5Ab504k9WGmQiPkQjzHMp09AWuDhLlY iYIBY9ALwLr8ZOWU4YDrYPUbjeazu6kMWDQ2sMYgiJJN4MT3kW2isbwRnBkBYWJgFmBd RPqKE7qfnXil1L3ee+W6xirLVSmrlOgi585poE8Q3UMbqc0YAGRrh2QKuctvFtF8Ls/o 855FqxECjPeueA0n5dsMbsQFXwn6TzVhb8eTW63/XDiBaC/uxN2PzH3ZX8YqqmLKLe0M /44g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=+t+cIZqcnxPpRu93TZSyPqI5zhsAiI8Yqy+BlhriBHw=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=0HYOK7tnmfhi7El8FqRT1j1QSWW6xp64sKV4CjIp0qlPyx/c4FPT87wr0N1pT6SlAY aZB9izk0utK/76eZHvOn94aUNKPZ7yk2WGYAvjGcmQhkhMDrnM4Nh+iQE8iTojxYMfvY jKDF1hJ5rU0KTp2bBMGyiEswrXGlV2Tq/1XhEKPaz7hDoKGygQi6SbUek5c5hZL6YS4a MPker5aJNEvsteZQgrzAZf8YREKc7zsKiBh01CluYjrVEJcNCymLoOJ/CbaMNu4QiMau 73S5lPORek+hbSK+5oS5P7KKROSFDRyDFodGfcgTd6Mn9KDegCCe7rq1xhqkYKsLD4qt e9aQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=IM3WYqsN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z19-20020a63e113000000b00553a50c5d85si9453171pgh.510.2023.08.01.15.08.24; Tue, 01 Aug 2023 15:08:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=IM3WYqsN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232363AbjHAVZK (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232266AbjHAVYz (ORCPT ); Tue, 1 Aug 2023 17:24:55 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3922359E for ; Tue, 1 Aug 2023 14:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=+t+cIZqcnxPpRu93TZSyPqI5zhsAiI8Yqy+BlhriBHw=; b=IM3WYqsNA2BbiQtdFADa2gomoD qqrejQLUxpaa3PX8lxd6dkq4elfIaAVb/VgsWGSVE2hhf4aL/p6PEV18fehDaFl6cT7d2ZEbNRHyW JrhQXBi4Jktho0k0IEW+Y8ZTrBu3UP2Db1TKI2P00+TNAGyN3K5oXBLOXr5CeCJSbcrWv3KOvo665 cwPgWklfuD8p35guhVqMcNcyqtQ0iWmvxMBiwdez1fFBkELMfv8gM8DK/PKc/Y3tVe5gU95qdK2Tb hBiwxSEcc40v8QISM1Lp5siTkIAQZmHqL+XUAnVu9A9p6jukzbGjTn+nOYR9wy1QTwXLe2D4b4aCG cv3D41BA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qQwqm-00B5yc-2S; Tue, 01 Aug 2023 21:24:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 5893F300768; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3A78A202883B0; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211811.896559109@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:23 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 2/9] sched: Simplify sysctl_sched_uclamp_handler() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773066222276777766 X-GMAIL-MSGID: 1773066222276777766 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1801,7 +1801,8 @@ static int sysctl_sched_uclamp_handler(s int old_min, old_max, old_min_rt; int result; - mutex_lock(&uclamp_mutex); + guard(mutex)(&uclamp_mutex); + old_min = sysctl_sched_uclamp_util_min; old_max = sysctl_sched_uclamp_util_max; old_min_rt = sysctl_sched_uclamp_util_min_rt_default; @@ -1810,7 +1811,7 @@ static int sysctl_sched_uclamp_handler(s if (result) goto undo; if (!write) - goto done; + return 0; if (sysctl_sched_uclamp_util_min > sysctl_sched_uclamp_util_max || sysctl_sched_uclamp_util_max > SCHED_CAPACITY_SCALE || @@ -1846,16 +1847,12 @@ static int sysctl_sched_uclamp_handler(s * Otherwise, keep it simple and do just a lazy update at each next * task enqueue time. */ - - goto done; + return 0; undo: sysctl_sched_uclamp_util_min = old_min; sysctl_sched_uclamp_util_max = old_max; sysctl_sched_uclamp_util_min_rt_default = old_min_rt; -done: - mutex_unlock(&uclamp_mutex); - return result; } #endif From patchwork Tue Aug 1 20:41:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129493 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp34323vqx; Tue, 1 Aug 2023 14:54:48 -0700 (PDT) X-Google-Smtp-Source: APBJJlEhBLfc0OzX923Sww/Sa2Wxs9Se4mEn2O5vU5AvHeL20LumAhtqAvCoVWnmm/FfFMjly0Io X-Received: by 2002:a05:6808:1891:b0:3a4:34c8:f777 with SMTP id bi17-20020a056808189100b003a434c8f777mr17128829oib.16.1690926888178; Tue, 01 Aug 2023 14:54:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690926888; cv=none; d=google.com; s=arc-20160816; b=0hdlCB4P0yXJek0pUI2FtHvtQ3rNyM5LWmmZz05chdFR0+KJilI8fDJggCObJjxeHY Mv7wr7zIoBTRg5wof0oDvFNd5iMkQYc9G3iHTzW8+2FsXwzkbX0diM1oxO11APN94rSo OzVe7NmtpSQ0vkJMP5v2nxGxXzKHf2EOv8NRvB8+ErVyBdDdqQ8HSZc2ffDeCIA+meTj o+5ULy4eHDR5oWti+AniecL5bsS3Gsv5x2gijg8RrfJdu0OuHJtMpBzP/MFpO6QvymUv +TSaAqCs93iNugr53C3k+Hu0xw5jc6TWzHyPD1w9xV+PZ51BbZ7t4bkq+zirNmkMm+w7 Ghbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=Z/QyOBDSEB+zoR9GPvKsSeTCgad9VvZXAWajjLt+jzo=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=ZuCXo2wHDeNUi2P4N2YM66N4L/oHgZJM60yjhw8vJLuymvOe34rFOuqzqlCDp/OV4t PeexMPU46QD+meCxeQ0Mx04K2PsFReabjXtBMNa48NvjJ3PdkMiyCsoTl5xZWCpCz1FG 2/4qUeSx7S7C3bx+1n6/MIeaaAXHJe4rcApTPE14/99WhoyehE9ELGbJNvijsJapJ8RU c5bf1ql4kHA3Hv6J3Ykf6JbUzpVMO0PE/C14spSVdNTvWJTRFTUi3gCOyzHzRFedlXFv 0SHaxM5c0RMMqCNzCNPyHVbtpXl8DvhOgnrhtyL39+X92M4L6vfFzoUOTdKolbXcRRfZ IZ2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="Al/Ax4if"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i14-20020a636d0e000000b005307d9f387esi7530550pgc.536.2023.08.01.14.54.35; Tue, 01 Aug 2023 14:54:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="Al/Ax4if"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232270AbjHAVZE (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232263AbjHAVYz (ORCPT ); Tue, 1 Aug 2023 17:24:55 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE18F35A5 for ; Tue, 1 Aug 2023 14:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Z/QyOBDSEB+zoR9GPvKsSeTCgad9VvZXAWajjLt+jzo=; b=Al/Ax4if19A7a2TrJQQAx5hfN3 jqg+ksC+d20UaRIoWyI9FVSvx17aviTARLX3DxpTKJO7DNGF8MSnXB4mXyz7+rlnD8gTBntWtLV/K hAE6NSmjeMqZqHkdudwwOa8osYGnJq9Z38p0Bve15z5ihBn6bYNZmexYZS6dvTzC9d9gIk/PzL15F Y6VzxwEZXQinghsVeWDd1BAs2xUPy7ET2DfI7GoTkaVtUOdzCeD+ReoItHAbcno31GqJAqBl9WZDo 4FMje4CiSik7YQwcStueiaJPQ1dRuceH1JtbaVo8peHJv0nfT54H88ZRtJvgVv6ZCW/z+N06y61Mg hoCUurHQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qQwqm-00EvTp-0A; Tue, 01 Aug 2023 21:24:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 5A81130083E; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4047520292D01; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211811.964370836@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:24 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 3/9] sched: Simplify: migrate_swap_stop() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773065352895588712 X-GMAIL-MSGID: 1773065352895588712 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 23 +++++++---------------- kernel/sched/sched.h | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+), 16 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3258,7 +3258,6 @@ static int migrate_swap_stop(void *data) { struct migration_swap_arg *arg = data; struct rq *src_rq, *dst_rq; - int ret = -EAGAIN; if (!cpu_active(arg->src_cpu) || !cpu_active(arg->dst_cpu)) return -EAGAIN; @@ -3266,33 +3265,25 @@ static int migrate_swap_stop(void *data) src_rq = cpu_rq(arg->src_cpu); dst_rq = cpu_rq(arg->dst_cpu); - double_raw_lock(&arg->src_task->pi_lock, - &arg->dst_task->pi_lock); - double_rq_lock(src_rq, dst_rq); + guard(double_raw_spinlock)(&arg->src_task->pi_lock, &arg->dst_task->pi_lock); + guard(double_rq_lock)(src_rq, dst_rq); if (task_cpu(arg->dst_task) != arg->dst_cpu) - goto unlock; + return -EAGAIN; if (task_cpu(arg->src_task) != arg->src_cpu) - goto unlock; + return -EAGAIN; if (!cpumask_test_cpu(arg->dst_cpu, arg->src_task->cpus_ptr)) - goto unlock; + return -EAGAIN; if (!cpumask_test_cpu(arg->src_cpu, arg->dst_task->cpus_ptr)) - goto unlock; + return -EAGAIN; __migrate_swap_task(arg->src_task, arg->dst_cpu); __migrate_swap_task(arg->dst_task, arg->src_cpu); - ret = 0; - -unlock: - double_rq_unlock(src_rq, dst_rq); - raw_spin_unlock(&arg->dst_task->pi_lock); - raw_spin_unlock(&arg->src_task->pi_lock); - - return ret; + return 0; } /* --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2572,6 +2572,12 @@ static inline void double_rq_clock_clear static inline void double_rq_clock_clear_update(struct rq *rq1, struct rq *rq2) {} #endif +#define DEFINE_LOCK_GUARD_2(name, type, _lock, _unlock, ...) \ +__DEFINE_UNLOCK_GUARD(name, type, _unlock, type *lock2; __VA_ARGS__) \ +static inline class_##name##_t class_##name##_constructor(type *lock, type *lock2) \ +{ class_##name##_t _t = { .lock = lock, .lock2 = lock2 }, *_T = &_t; \ + _lock; return _t; } + #ifdef CONFIG_SMP static inline bool rq_order_less(struct rq *rq1, struct rq *rq2) @@ -2701,6 +2707,16 @@ static inline void double_raw_lock(raw_s raw_spin_lock_nested(l2, SINGLE_DEPTH_NESTING); } +static inline void double_raw_unlock(raw_spinlock_t *l1, raw_spinlock_t *l2) +{ + raw_spin_unlock(l1); + raw_spin_unlock(l2); +} + +DEFINE_LOCK_GUARD_2(double_raw_spinlock, raw_spinlock_t, + double_raw_lock(_T->lock, _T->lock2), + double_raw_unlock(_T->lock, _T->lock2)) + /* * double_rq_unlock - safely unlock two runqueues * @@ -2758,6 +2774,10 @@ static inline void double_rq_unlock(stru #endif +DEFINE_LOCK_GUARD_2(double_rq_lock, struct rq, + double_rq_lock(_T->lock, _T->lock2), + double_rq_unlock(_T->lock, _T->lock2)) + extern struct sched_entity *__pick_first_entity(struct cfs_rq *cfs_rq); extern struct sched_entity *__pick_last_entity(struct cfs_rq *cfs_rq); From patchwork Tue Aug 1 20:41:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129527 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp81850vqx; Tue, 1 Aug 2023 16:37:52 -0700 (PDT) X-Google-Smtp-Source: APBJJlHab5AdBCjB1XqW3QQLulLDw5ok+Qq0XTdTdjFdk03P+AtlpTl7vouk1VNvRUJ44znMAPFH X-Received: by 2002:a05:6a20:4412:b0:13e:25ec:f240 with SMTP id ce18-20020a056a20441200b0013e25ecf240mr7023298pzb.2.1690933072499; Tue, 01 Aug 2023 16:37:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690933072; cv=none; d=google.com; s=arc-20160816; b=Z29kpzNkPamaJ1voGIKyzjvNlEhR6pM5MPtU6OAq2uJafoVPdXD5xqMf5YHy3sy2OX kaq3+HU0tI0NkFHaQPArVpob+i7FFuZN0AMT45qJkyR6EO2PoiqIZulrwSJ7xVU6Lf/V kzQtKToyEve8p3RI8Zrx4kNo0llJbIhIwEslKTsrER0VEpmXIr/yf3C8VXKh5i6b9aX5 8KckgoXqkzfYu7+okZhNuGacy0F9Ro3Oi1dfEAXzPYMVj+nB56Cci2YvxsBJELKvk6q/ 2km5Woa0qO23bMYWAUof1WwSryEttnixHGk+yGDBLJrNqQZDae9+hLOnHXMiZT8PqY0g 52Pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=XnCmIaNvJgB5bekhtHHcxeVW5sv9v7ogZHNowp3XnxU=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=kPGaspqniZReEVrVtHLY6iD2gphwZkq5rnz1iYxEeG8dFSOtwBjfxymfTM4bpvcTGh X6YwCYKSMqO1kAxkFyxU9YmtRFns0VFoQ7994dZK3dL2Pz+cTxj08qojnmTz+jThN2ea Vfk7TtiayDhhQaYiV8vN1Rbrdfl0aEUz/nau7qYMeF4+8RoXAsbSdGj/i5GLdGRaO4X3 MEoMXAzYE9qzddNH6vAJGItgeqJ4e7I9CIzTcfB1Dchj71DyxoxLMY5V6ykJe/NqgDWP 0PrA/L7JvjxpENutVSnROmRIx4EQvy4Dn1Q/n8Rj8rqOPlei+Bk5rykmd+8nGptaXJgU t5aw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=grhO7SXH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i197-20020a639dce000000b0055ac8f8526dsi2506673pgd.530.2023.08.01.16.37.39; Tue, 01 Aug 2023 16:37:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=grhO7SXH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232499AbjHAVZ0 (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230029AbjHAVY4 (ORCPT ); Tue, 1 Aug 2023 17:24:56 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18A4C35A8 for ; Tue, 1 Aug 2023 14:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=XnCmIaNvJgB5bekhtHHcxeVW5sv9v7ogZHNowp3XnxU=; b=grhO7SXHynJfVX9hhJuZZMTp7F 2h7GK1gVLNebfvh7OneKfz6GSmm7RHut5djU0eFNRKHMe2FRvhg+FzBwaJ78Uo0sHHH6LkF8AT80w g8y0RsK9+1Iu6n7w3Vh6NUEZFwqA63p3VTG0yK8h+GjE9tvSqg4mpVxmiGV6jeSfsUWArAdJhsqOk 3O+CLGk4lfoghVPNgVudI1bsdSVYBKmW/6tUF3Mh0mn3bMfl3PRjv16/jTB5L479Ad3d+Z9bAFUUh EpRpYKlW9aKWPGtO5oraXAa0wyinNXesFXZ3SFNSHMg/77OLUg0ZKcEVgKqNGl0dzBvsJ58Z5jKYf KEGfVo+A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qQwqm-00EvTo-06; Tue, 01 Aug 2023 21:24:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 5C3463008C6; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4473A2029F9EC; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211812.032678917@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:25 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 4/9] sched: Simplify wake_up_if_idle() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773071837496223342 X-GMAIL-MSGID: 1773071837496223342 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 20 ++++++-------------- kernel/sched/sched.h | 15 +++++++++++++++ 2 files changed, 21 insertions(+), 14 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3872,21 +3872,13 @@ static void __ttwu_queue_wakelist(struct void wake_up_if_idle(int cpu) { struct rq *rq = cpu_rq(cpu); - struct rq_flags rf; - rcu_read_lock(); - - if (!is_idle_task(rcu_dereference(rq->curr))) - goto out; - - rq_lock_irqsave(rq, &rf); - if (is_idle_task(rq->curr)) - resched_curr(rq); - /* Else CPU is not idle, do nothing here: */ - rq_unlock_irqrestore(rq, &rf); - -out: - rcu_read_unlock(); + guard(rcu)(); + if (is_idle_task(rcu_dereference(rq->curr))) { + guard(rq_lock_irqsave)(rq); + if (is_idle_task(rq->curr)) + resched_curr(rq); + } } bool cpus_share_cache(int this_cpu, int that_cpu) --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1678,6 +1678,21 @@ rq_unlock(struct rq *rq, struct rq_flags raw_spin_rq_unlock(rq); } +DEFINE_LOCK_GUARD_1(rq_lock, struct rq, + rq_lock(_T->lock, &_T->rf), + rq_unlock(_T->lock, &_T->rf), + struct rq_flags rf) + +DEFINE_LOCK_GUARD_1(rq_lock_irq, struct rq, + rq_lock_irq(_T->lock, &_T->rf), + rq_unlock_irq(_T->lock, &_T->rf), + struct rq_flags rf) + +DEFINE_LOCK_GUARD_1(rq_lock_irqsave, struct rq, + rq_lock_irqsave(_T->lock, &_T->rf), + rq_unlock_irqrestore(_T->lock, &_T->rf), + struct rq_flags rf) + static inline struct rq * this_rq_lock_irq(struct rq_flags *rf) __acquires(rq->lock) From patchwork Tue Aug 1 20:41:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129500 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp36818vqx; Tue, 1 Aug 2023 15:00:46 -0700 (PDT) X-Google-Smtp-Source: APBJJlHhw3UloE04RNY1P1g4gPHoHdYLF2Q1C/72u2OVhQA8xrzRXV+TdZBewaMN87XPws8+7U5u X-Received: by 2002:a05:6a20:8f07:b0:13e:c5a3:97f with SMTP id b7-20020a056a208f0700b0013ec5a3097fmr1706026pzk.42.1690927245775; Tue, 01 Aug 2023 15:00:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690927245; cv=none; d=google.com; s=arc-20160816; b=UThWGptG8FYghkAHRJ5c4ZetavoDxp3GAy8XRsYLqYLdcdKZvqS1xwdBkDQNlNllp0 At0iG37qBKhchoIRomPJr1h6n0QCbWJ4zmEW5WOIIm0VEjRM1qHei1sgFS6d9JPjGv0L YFM0X5I+KirqkILQlpasZQ7duKR7/+aaQqmpX0INz87G2kBgMO9YBWQfxFb1OAeFeXFL CE4Xm2oG573Y08fj6uPapabSlxVyw0t9SZZQQqe3kFyIAh6RLfWkKWaVOebQk81UsIve GpYvZnHmHwqv3sg5+w1b4GCp14l97DYlmfWTC/sPgvwkC36pnW/OMwxIOpdNKtxqF0Hz BTHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=d7IoV3mp1lNO/qVCmLB7jmrWy1OfPyeEV/3PfRbFH8k=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=Z3AwKIgUfU/ZOZuRPtKsOZbOxYCaBwxRYU4pJ2tbFipEyEOIsAGXNNilaOUA+GrdQH oU5XM0/qScxW1FJsP87KENiA7glZ9zahb7P7wLAZNUvhAnx1kwDy8PcTQucCCWXFmi+M dIFn6I9n/gbbUoc1AyXiihsUuA9STtRry6vaD2Kt9r/iWSR+OH7t2+yZM9a1yrqOrism OI5dhzGH8EuRysZ1snkZIBs7Sff44A+r0THjD+56PE0A07W5NPRuoKTSK5Xb7nA14jx0 fbl4JgJ6qDfQjKsuG+QcNm8dCeoUrkYabssWqnymA2alg7Hz25EvmS+rttzsneysEXIc 7pxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=LzWkMvFW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a72-20020a63904b000000b00563fbc00007si7110216pge.262.2023.08.01.15.00.32; Tue, 01 Aug 2023 15:00:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=LzWkMvFW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232455AbjHAVZX (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230197AbjHAVY4 (ORCPT ); Tue, 1 Aug 2023 17:24:56 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 131CF35A7 for ; Tue, 1 Aug 2023 14:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=d7IoV3mp1lNO/qVCmLB7jmrWy1OfPyeEV/3PfRbFH8k=; b=LzWkMvFWmU8IhV/McwiylHZHRN hOc3QDgIdIkDW0kCnPGyVzxZwdQ059bMivNk9HB2WYMTJNthS0QyhqOEV07B+y3prixahIxIZ9nqm Y/ex7w/59uEiiEk309Fn65+kNiI4vQCABToj2JSSdPJbT5klwC1/gk16xNSpVybV/vYn4OCO8z8QC b/CmIW1zYbsX9V16oshrcw1y1jPGtFLryH2PwsLWi1KLfb9i/qctey2noT6c43uSWISHAO+BZaE63 52S5itpnBKNdkd8nEH0nCdiaG1S7riCGC+zzipbuHx+HLP1K2nAhf1rSK74vqRUyNINRa2c5sNglX gaDe74yA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qQwqn-00EvTu-1c; Tue, 01 Aug 2023 21:24:19 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BEE4030278C; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 492882028A2D9; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211812.101069260@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:26 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 5/9] sched: Simplify ttwu() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773065727259436191 X-GMAIL-MSGID: 1773065727259436191 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 221 +++++++++++++++++++++++++--------------------------- 1 file changed, 109 insertions(+), 112 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3706,14 +3706,14 @@ ttwu_stat(struct task_struct *p, int cpu struct sched_domain *sd; __schedstat_inc(p->stats.nr_wakeups_remote); - rcu_read_lock(); + + guard(rcu)(); for_each_domain(rq->cpu, sd) { if (cpumask_test_cpu(cpu, sched_domain_span(sd))) { __schedstat_inc(sd->ttwu_wake_remote); break; } } - rcu_read_unlock(); } if (wake_flags & WF_MIGRATED) @@ -4172,10 +4172,9 @@ bool ttwu_state_match(struct task_struct static int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) { - unsigned long flags; + guard(preempt)(); int cpu, success = 0; - preempt_disable(); if (p == current) { /* * We're waking current, this means 'p->on_rq' and 'task_cpu(p) @@ -4202,129 +4201,127 @@ try_to_wake_up(struct task_struct *p, un * reordered with p->state check below. This pairs with smp_store_mb() * in set_current_state() that the waiting thread does. */ - raw_spin_lock_irqsave(&p->pi_lock, flags); - smp_mb__after_spinlock(); - if (!ttwu_state_match(p, state, &success)) - goto unlock; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + smp_mb__after_spinlock(); + if (!ttwu_state_match(p, state, &success)) + break; - trace_sched_waking(p); + trace_sched_waking(p); - /* - * Ensure we load p->on_rq _after_ p->state, otherwise it would - * be possible to, falsely, observe p->on_rq == 0 and get stuck - * in smp_cond_load_acquire() below. - * - * sched_ttwu_pending() try_to_wake_up() - * STORE p->on_rq = 1 LOAD p->state - * UNLOCK rq->lock - * - * __schedule() (switch to task 'p') - * LOCK rq->lock smp_rmb(); - * smp_mb__after_spinlock(); - * UNLOCK rq->lock - * - * [task p] - * STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq - * - * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in - * __schedule(). See the comment for smp_mb__after_spinlock(). - * - * A similar smb_rmb() lives in try_invoke_on_locked_down_task(). - */ - smp_rmb(); - if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) - goto unlock; + /* + * Ensure we load p->on_rq _after_ p->state, otherwise it would + * be possible to, falsely, observe p->on_rq == 0 and get stuck + * in smp_cond_load_acquire() below. + * + * sched_ttwu_pending() try_to_wake_up() + * STORE p->on_rq = 1 LOAD p->state + * UNLOCK rq->lock + * + * __schedule() (switch to task 'p') + * LOCK rq->lock smp_rmb(); + * smp_mb__after_spinlock(); + * UNLOCK rq->lock + * + * [task p] + * STORE p->state = UNINTERRUPTIBLE LOAD p->on_rq + * + * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in + * __schedule(). See the comment for smp_mb__after_spinlock(). + * + * A similar smb_rmb() lives in try_invoke_on_locked_down_task(). + */ + smp_rmb(); + if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) + break; #ifdef CONFIG_SMP - /* - * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be - * possible to, falsely, observe p->on_cpu == 0. - * - * One must be running (->on_cpu == 1) in order to remove oneself - * from the runqueue. - * - * __schedule() (switch to task 'p') try_to_wake_up() - * STORE p->on_cpu = 1 LOAD p->on_rq - * UNLOCK rq->lock - * - * __schedule() (put 'p' to sleep) - * LOCK rq->lock smp_rmb(); - * smp_mb__after_spinlock(); - * STORE p->on_rq = 0 LOAD p->on_cpu - * - * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in - * __schedule(). See the comment for smp_mb__after_spinlock(). - * - * Form a control-dep-acquire with p->on_rq == 0 above, to ensure - * schedule()'s deactivate_task() has 'happened' and p will no longer - * care about it's own p->state. See the comment in __schedule(). - */ - smp_acquire__after_ctrl_dep(); + /* + * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be + * possible to, falsely, observe p->on_cpu == 0. + * + * One must be running (->on_cpu == 1) in order to remove oneself + * from the runqueue. + * + * __schedule() (switch to task 'p') try_to_wake_up() + * STORE p->on_cpu = 1 LOAD p->on_rq + * UNLOCK rq->lock + * + * __schedule() (put 'p' to sleep) + * LOCK rq->lock smp_rmb(); + * smp_mb__after_spinlock(); + * STORE p->on_rq = 0 LOAD p->on_cpu + * + * Pairs with the LOCK+smp_mb__after_spinlock() on rq->lock in + * __schedule(). See the comment for smp_mb__after_spinlock(). + * + * Form a control-dep-acquire with p->on_rq == 0 above, to ensure + * schedule()'s deactivate_task() has 'happened' and p will no longer + * care about it's own p->state. See the comment in __schedule(). + */ + smp_acquire__after_ctrl_dep(); - /* - * We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq - * == 0), which means we need to do an enqueue, change p->state to - * TASK_WAKING such that we can unlock p->pi_lock before doing the - * enqueue, such as ttwu_queue_wakelist(). - */ - WRITE_ONCE(p->__state, TASK_WAKING); + /* + * We're doing the wakeup (@success == 1), they did a dequeue (p->on_rq + * == 0), which means we need to do an enqueue, change p->state to + * TASK_WAKING such that we can unlock p->pi_lock before doing the + * enqueue, such as ttwu_queue_wakelist(). + */ + WRITE_ONCE(p->__state, TASK_WAKING); - /* - * If the owning (remote) CPU is still in the middle of schedule() with - * this task as prev, considering queueing p on the remote CPUs wake_list - * which potentially sends an IPI instead of spinning on p->on_cpu to - * let the waker make forward progress. This is safe because IRQs are - * disabled and the IPI will deliver after on_cpu is cleared. - * - * Ensure we load task_cpu(p) after p->on_cpu: - * - * set_task_cpu(p, cpu); - * STORE p->cpu = @cpu - * __schedule() (switch to task 'p') - * LOCK rq->lock - * smp_mb__after_spin_lock() smp_cond_load_acquire(&p->on_cpu) - * STORE p->on_cpu = 1 LOAD p->cpu - * - * to ensure we observe the correct CPU on which the task is currently - * scheduling. - */ - if (smp_load_acquire(&p->on_cpu) && - ttwu_queue_wakelist(p, task_cpu(p), wake_flags)) - goto unlock; + /* + * If the owning (remote) CPU is still in the middle of schedule() with + * this task as prev, considering queueing p on the remote CPUs wake_list + * which potentially sends an IPI instead of spinning on p->on_cpu to + * let the waker make forward progress. This is safe because IRQs are + * disabled and the IPI will deliver after on_cpu is cleared. + * + * Ensure we load task_cpu(p) after p->on_cpu: + * + * set_task_cpu(p, cpu); + * STORE p->cpu = @cpu + * __schedule() (switch to task 'p') + * LOCK rq->lock + * smp_mb__after_spin_lock() smp_cond_load_acquire(&p->on_cpu) + * STORE p->on_cpu = 1 LOAD p->cpu + * + * to ensure we observe the correct CPU on which the task is currently + * scheduling. + */ + if (smp_load_acquire(&p->on_cpu) && + ttwu_queue_wakelist(p, task_cpu(p), wake_flags)) + break; - /* - * If the owning (remote) CPU is still in the middle of schedule() with - * this task as prev, wait until it's done referencing the task. - * - * Pairs with the smp_store_release() in finish_task(). - * - * This ensures that tasks getting woken will be fully ordered against - * their previous state and preserve Program Order. - */ - smp_cond_load_acquire(&p->on_cpu, !VAL); + /* + * If the owning (remote) CPU is still in the middle of schedule() with + * this task as prev, wait until it's done referencing the task. + * + * Pairs with the smp_store_release() in finish_task(). + * + * This ensures that tasks getting woken will be fully ordered against + * their previous state and preserve Program Order. + */ + smp_cond_load_acquire(&p->on_cpu, !VAL); - cpu = select_task_rq(p, p->wake_cpu, wake_flags | WF_TTWU); - if (task_cpu(p) != cpu) { - if (p->in_iowait) { - delayacct_blkio_end(p); - atomic_dec(&task_rq(p)->nr_iowait); - } + cpu = select_task_rq(p, p->wake_cpu, wake_flags | WF_TTWU); + if (task_cpu(p) != cpu) { + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } - wake_flags |= WF_MIGRATED; - psi_ttwu_dequeue(p); - set_task_cpu(p, cpu); - } + wake_flags |= WF_MIGRATED; + psi_ttwu_dequeue(p); + set_task_cpu(p, cpu); + } #else - cpu = task_cpu(p); + cpu = task_cpu(p); #endif /* CONFIG_SMP */ - ttwu_queue(p, cpu, wake_flags); -unlock: - raw_spin_unlock_irqrestore(&p->pi_lock, flags); + ttwu_queue(p, cpu, wake_flags); + } out: if (success) ttwu_stat(p, task_cpu(p), wake_flags); - preempt_enable(); return success; } From patchwork Tue Aug 1 20:41:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129516 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp53611vqx; Tue, 1 Aug 2023 15:32:02 -0700 (PDT) X-Google-Smtp-Source: APBJJlEbCx3djr5l5JcL6ZkJjZbIfQpdxu8h2C8pJQQiuq6yOw+soFlXrhhioqAH8fKs15HPZLJK X-Received: by 2002:a05:651c:106:b0:2b6:ded8:6fc1 with SMTP id a6-20020a05651c010600b002b6ded86fc1mr3496892ljb.25.1690929122103; Tue, 01 Aug 2023 15:32:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690929122; cv=none; d=google.com; s=arc-20160816; b=F1pl8Fhzc315CYksEqD8lvgN9BdAIMwiju1UUOvIAXYGtxQytxWdOl3gpcAqvTmqFo QJISUQ2XhVuva2zdXZ2+9giM7CiKu3gklhsAGii/72JO+ft+L+FW/dXVfBnktWSkaPkO ad9PqJzpq1GbE8fq2mZPExlBMUzD8CbeOAeAKofnIRyf/dLnOEYF5g+inZNlEIYGPav9 K+pxkFDpwhHr0CTjtu5BfF+WXph65ESelyjVOG475DhADkR1a5OoOZsQp4rVObSnX1Zo 4hITb8x2URi/RgJxySBeTS9jSEeiUdQAkPkvoWbOwqflHmrmPGC4xa60NLxgrv5GRdJ0 HZOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=drbNuL55DYuL5wiscrlTnGAb/cO6SeGIhgtGTiA6PhQ=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=fLYYoRDi237dV0Whut3xuj84M1KbkDTEWGrx/k8ubC8uH/J+GcFypm1n9UWvYeW57v eYrbJpq8FEkyQfNB4i9VMyjIFphSg3B39ghVfrRIRerRS4tQqLkOZMdi46HW6fG03Eeo DX5tMaAvJXBkJ3TBx5o6kZDHU9ect50WqsQJYwjzeFdv+MIDbl0L+j4mnPQ6GR0NfEhi rGHapuBa1xRauHGhHVXbjpdSNO5tTaFRA7oDTqMBQ/puLYL3wTvaRMHmgCqGu7rrXQcU xp4tNHDcx0EPKW4srx1vRbJy1HHUalwwfZcwEic51p4JkYFeX8qji3Gz0PaQ9Fwygt4t XKSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=CmSlmMvB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y13-20020a170906524d00b009930c925bd9si8675045ejm.945.2023.08.01.15.31.38; Tue, 01 Aug 2023 15:32:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=CmSlmMvB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232257AbjHAVZH (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232258AbjHAVYz (ORCPT ); Tue, 1 Aug 2023 17:24:55 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC3D435A2 for ; Tue, 1 Aug 2023 14:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=drbNuL55DYuL5wiscrlTnGAb/cO6SeGIhgtGTiA6PhQ=; b=CmSlmMvBqFeZkasybngqkB/9/Z mUv7r6cmiKzOML5p3sP/7eNW16u9qXgad8LSgfbpyGh4CxsLLmw2+fNFo/ocTFfJNo0Roo/ELwr6o VWbMNAPshVBVeQplDvk7SN1Lgu5HGT6/O2moDQB/+MlNaKc8mQ0s4GL3cEOphzAOK24BFFOkgWJFk GzKQEzHnF7TRhjXrm0ER9eKzep+vUFLQNNxtbR+DusrIdrO6+z7R6eEDn9IfZESBYjm8ARA9ed+eh 4e2nwM1i8mQ175codWVx83vH1F5D6LSOw/I4VMx1Ei1MK7jL0k6BtcRWfsQ7puqN0FmXoKGXL4Zph igvkYxww==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qQwqn-00EvTv-1f; Tue, 01 Aug 2023 21:24:20 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BEDCA3006ED; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 4BEB720282F78; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211812.168490417@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:27 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 6/9] sched: Simplify sched_exec() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773067694968694900 X-GMAIL-MSGID: 1773067694968694900 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5431,23 +5431,20 @@ unsigned int nr_iowait(void) void sched_exec(void) { struct task_struct *p = current; - unsigned long flags; + struct migration_arg arg; int dest_cpu; - raw_spin_lock_irqsave(&p->pi_lock, flags); - dest_cpu = p->sched_class->select_task_rq(p, task_cpu(p), WF_EXEC); - if (dest_cpu == smp_processor_id()) - goto unlock; + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + dest_cpu = p->sched_class->select_task_rq(p, task_cpu(p), WF_EXEC); + if (dest_cpu == smp_processor_id()) + return; - if (likely(cpu_active(dest_cpu))) { - struct migration_arg arg = { p, dest_cpu }; + if (unlikely(!cpu_active(dest_cpu))) + return; - raw_spin_unlock_irqrestore(&p->pi_lock, flags); - stop_one_cpu(task_cpu(p), migration_cpu_stop, &arg); - return; + arg = (struct migration_arg){ p, dest_cpu }; } -unlock: - raw_spin_unlock_irqrestore(&p->pi_lock, flags); + stop_one_cpu(task_cpu(p), migration_cpu_stop, &arg); } #endif From patchwork Tue Aug 1 20:41:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129490 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp34079vqx; Tue, 1 Aug 2023 14:54:16 -0700 (PDT) X-Google-Smtp-Source: APBJJlGIiSf01ksHlxGs5gDia3Yjlr6xtzeaRhLSu+7gKy25reXa01PadDSUcNA0eJ7Hnrb6Fdyi X-Received: by 2002:a05:6a00:a85:b0:668:69fa:f78f with SMTP id b5-20020a056a000a8500b0066869faf78fmr16066817pfl.1.1690926855924; Tue, 01 Aug 2023 14:54:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690926855; cv=none; d=google.com; s=arc-20160816; b=WAFDF15YoDjbcaPLCW+WvsOF7fwl9EW0Wvol0vd8owZJ7/sheUjIaB/EJ/oX8Gwwga P93Jb/YEftsl27KYIoR3CntF3IwyrUWMHr1+BJ60R94epgHugIPp+bBBLfNbFYxk1Cyn ebk3Zlle7AFlKShDFPqUh8RDbjq+Y6sIuG1myxBwzdzh85kcy1FQlw2H2S28/DLDbszI 3bjsFQk9iSLoceOnFhVKQeHtsZgjZ83HyWUVgzvn8TbFruXBta7nF/ww+NQVAPrcu74g 30WWqK0iouwHFab3lFbSMRY3+9rVOnXCLU/uo+yOgJ3gIhfs73gAUzo8MaSPy3jfu8LE XZCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=q92jsVFPZSP8G7FOeDp8TiMSNcZFRY9w1smIqAIutpc=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=hLQBsPf1+ecTMyJMTZsGgknO/+8nNIwoNVqYot09aA7wr5TvSvVk/iofhyOoj8sdTe 6r408oM8BKqq0OQhGURqWLgdAHxhUOAG0lD8+oa2esZ4U0OM5I2dM5iaJBiHKLsLBRQq MXeJ8aQ+ZYCb5oUdQjdU3O4CQH8WNzVeOZ1c3yHuYKOg7JmuTEC1VB4Bw4kO0HX7v+Yw nDEkwNnzGJHTqA9GzwbiNH2Z4SaN7CLBPqAVb2WZS4gqGLxa5KxVUWylmO0HgBAkJLgF 646IFrVX/COQGiEPmnkAMUaMAHmmaG9z7+iFTDOATcwi3buM3v3v2eg5fUII6HhQCzWq 9I9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="Zs/CXT4S"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i14-20020a636d0e000000b005307d9f387esi7530550pgc.536.2023.08.01.14.54.01; Tue, 01 Aug 2023 14:54:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="Zs/CXT4S"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232273AbjHAVZR (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232272AbjHAVYz (ORCPT ); Tue, 1 Aug 2023 17:24:55 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62DC135AA for ; Tue, 1 Aug 2023 14:24:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=q92jsVFPZSP8G7FOeDp8TiMSNcZFRY9w1smIqAIutpc=; b=Zs/CXT4SulTjrAAiY5k/JWkVbm nhCxY6+yMoC+VlPJZokF8S4WA4fqGlAOTXENgeCoeHlkKzuq0wipZHItvz0AG72ro+VhW3Rlw8Fy+ ElTqg9gWrlTzSzFSOHOjUXPVGaa4AsavKRfIAD17HosObUFW+rCfubbLhI/BsR7UiLTr3ExI55yRp T0wjF+UVz+E7tCmZssp6z135YlHwBTB0VPKI0UC8Z+JzCUVCktqH0bmCT1imWr3KQA7lvam5+LWJT mWBpEaT7XJ96fmYyMKMQdX53gcJGDStd6CGGK0Ni5ERXhsLs7R/SAuGQH379LjysbNS7AvlYS07Iz M3/Z86xw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qQwqn-00EvTw-1h; Tue, 01 Aug 2023 21:24:19 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BEE5930310E; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 52AE820283BA4; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211812.236247952@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:28 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 7/9] sched: Simplify sched_tick_remote() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773065318793191696 X-GMAIL-MSGID: 1773065318793191696 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 43 ++++++++++++++++++------------------------- 1 file changed, 18 insertions(+), 25 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5651,9 +5651,6 @@ static void sched_tick_remote(struct wor struct tick_work *twork = container_of(dwork, struct tick_work, work); int cpu = twork->cpu; struct rq *rq = cpu_rq(cpu); - struct task_struct *curr; - struct rq_flags rf; - u64 delta; int os; /* @@ -5663,30 +5660,26 @@ static void sched_tick_remote(struct wor * statistics and checks timeslices in a time-independent way, regardless * of when exactly it is running. */ - if (!tick_nohz_tick_stopped_cpu(cpu)) - goto out_requeue; + if (tick_nohz_tick_stopped_cpu(cpu)) { + guard(rq_lock_irq)(rq); + struct task_struct *curr = rq->curr; + + if (cpu_online(cpu)) { + update_rq_clock(rq); + + if (!is_idle_task(curr)) { + /* + * Make sure the next tick runs within a + * reasonable amount of time. + */ + u64 delta = rq_clock_task(rq) - curr->se.exec_start; + WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3); + } + curr->sched_class->task_tick(rq, curr, 0); - rq_lock_irq(rq, &rf); - curr = rq->curr; - if (cpu_is_offline(cpu)) - goto out_unlock; - - update_rq_clock(rq); - - if (!is_idle_task(curr)) { - /* - * Make sure the next tick runs within a reasonable - * amount of time. - */ - delta = rq_clock_task(rq) - curr->se.exec_start; - WARN_ON_ONCE(delta > (u64)NSEC_PER_SEC * 3); + calc_load_nohz_remote(rq); + } } - curr->sched_class->task_tick(rq, curr, 0); - - calc_load_nohz_remote(rq); -out_unlock: - rq_unlock_irq(rq, &rf); -out_requeue: /* * Run the remote tick once per second (1Hz). This arbitrary From patchwork Tue Aug 1 20:41:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129508 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp50608vqx; Tue, 1 Aug 2023 15:25:46 -0700 (PDT) X-Google-Smtp-Source: APBJJlEc8rFaY9D+48VWiZ9XLP9TIZ62i7hh6OSGWlbHWqDkntFEVSP5m6WS7nZWfseITyGp7Atg X-Received: by 2002:a17:907:78c2:b0:99c:3b4:940f with SMTP id kv2-20020a17090778c200b0099c03b4940fmr3867276ejc.27.1690928746282; Tue, 01 Aug 2023 15:25:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690928746; cv=none; d=google.com; s=arc-20160816; b=oP3cbFVcnPly/Nlb/pWqZ/ZivW6XQeFRkpYpndb+PBMRLmstVE8vIZ/WCBWAOAzx0H Okv2xYgzWWzuxbSQP37YWHOXSt38JpLsXreKQ3xr79OEmZ2HOzfWMIoPKY/UvrglJu21 gL8whyt1ORoADuBwpv8flpqAO0oKve8U1vrfQaAgq67O0M0R2qjWmA0GD5EPsjgwk5O+ 5M9dJIDftbOczxuIhcl1+K56FXica1heUylqUFK3gk1PQNodhldruhX8V3bkSiqFyryj uDuhwsiL+KoiWbKS/7fSW1t5xj1qiXZQ9CFd31+54fd0AOQLhxHPuy9vhKYBDlQtE+XU aMQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=4+/Q1nIn7pjeaoOkqiN8ZoN9dOzRXj0Qx8pmucBG9KU=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=cAOTL7QFrhHN8TsPERUJY/kU4tUbDXbenmq42lVA8CF0tDUA38g69XN0mrlw98RvEI budXr+eVuIe177c0Gz0n/2krvA/JHtleBMrFdJaL49ElDOVNteib7xz7kltIGT9HqmkT WHrz6mYwQNc3ad23FmzDTHuUJDDrnYsXSuWxZc/j0ZGy4wqoolg1Rry9cxTGiEg6Gm6n rM+Lbyy3l+v7ArGNjpRmrsGpQWHe4m3EptKLW+W6ZQVC5L5rgd/yR13GXFpbJLwsXOk4 NlD51lt+iYELCeqp3/61nE0GSiyXXo2M9ScJhU/hcFzCXZBY4W88mAxFmCrkKuYAOEre O0mQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=J8IHK9Ue; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n9-20020a170906688900b0099bcfd32960si6607263ejr.161.2023.08.01.15.25.21; Tue, 01 Aug 2023 15:25:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=J8IHK9Ue; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232565AbjHAVZb (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232078AbjHAVY6 (ORCPT ); Tue, 1 Aug 2023 17:24:58 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C77F2707 for ; Tue, 1 Aug 2023 14:24:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=4+/Q1nIn7pjeaoOkqiN8ZoN9dOzRXj0Qx8pmucBG9KU=; b=J8IHK9UeVvDpDEqjIjdyb+ueyD MZu72D5VRlzK2tgFk6E2Q19XNTvN6w4R97+txCz5/kQUErEhc2yg8x/4L9rokdCMEfx4NHRRpIjhh BmupbnZfuzBBcOzvXYkmurDDQkqYJVR+7bqQqWfeszzL94RmSahcvlCY0sz71Hu3ShBcJmRn4HVIW oGqKaY55edDIPUaXUF0TKedyjgYdxXvApczceNkZD6d2Nlc7uS47doikWbWyqjHJnIhPCUlIvhs5b JHQUnCDYksFwdhUdRC8afFXVARdY0EHUsRoFgfPKWRw6tgUQMNnYUYoByV+O2zkAEZavOIg2Ys3Xh FHwmGJMg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qQwqn-00EvTt-1a; Tue, 01 Aug 2023 21:24:18 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BEDEB302781; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 5713220286FBA; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211812.304154828@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:29 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 8/9] sched: Simplify try_steal_cookie() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773067301081785167 X-GMAIL-MSGID: 1773067301081785167 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6229,19 +6229,19 @@ static bool try_steal_cookie(int this, i unsigned long cookie; bool success = false; - local_irq_disable(); - double_rq_lock(dst, src); + guard(irq)(); + guard(double_rq_lock)(dst, src); cookie = dst->core->core_cookie; if (!cookie) - goto unlock; + return false; if (dst->curr != dst->idle) - goto unlock; + return false; p = sched_core_find(src, cookie); if (!p) - goto unlock; + return false; do { if (p == src->core_pick || p == src->curr) @@ -6253,9 +6253,10 @@ static bool try_steal_cookie(int this, i if (p->core_occupation > dst->idle->core_occupation) goto next; /* - * sched_core_find() and sched_core_next() will ensure that task @p - * is not throttled now, we also need to check whether the runqueue - * of the destination CPU is being throttled. + * sched_core_find() and sched_core_next() will ensure + * that task @p is not throttled now, we also need to + * check whether the runqueue of the destination CPU is + * being throttled. */ if (sched_task_is_throttled(p, this)) goto next; @@ -6273,10 +6274,6 @@ static bool try_steal_cookie(int this, i p = sched_core_next(p, cookie); } while (p); -unlock: - double_rq_unlock(dst, src); - local_irq_enable(); - return success; } From patchwork Tue Aug 1 20:41:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 129488 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f41:0:b0:3e4:2afc:c1 with SMTP id v1csp31017vqx; Tue, 1 Aug 2023 14:46:45 -0700 (PDT) X-Google-Smtp-Source: APBJJlFjMit2wvGI0g8IWYCoT2CAjUA/PjmR+XYE3SnQ938Scg7pkUoxJ2CeX1ARj5Gq2j3YKL4e X-Received: by 2002:a17:906:2ca:b0:99b:49ce:b4cc with SMTP id 10-20020a17090602ca00b0099b49ceb4ccmr3428353ejk.64.1690926405618; Tue, 01 Aug 2023 14:46:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690926405; cv=none; d=google.com; s=arc-20160816; b=aQMSUwmpakycvPX3Tun8BMbtGrTKZbad4wK/7VLzTp8tu769Vj+FSinPhcNGiNJdW2 Vv9Pzdjdp1QYztRdq2wizgnOAfz/ct2/4lEHFruS0r1sa4H9ZJoprheoJ6laYpUyJOBJ RdEmNFJCcXB6JEhJnNqINuEcfIavmNGM19ikY0QQqLORqtnvUQLVEVlFEGSwbcn8n+Yp cqB5LPgiEpgn2Ncxp2kxJtZLGDPcy4OSorxYu14VvuF7gH1Ggo7o93c7BubNXsenGvbB aK042A78HscQ7J0eNNqFYY0poJohoILI7d1MRSc+sMMGmB2+IAIztyrqm6ywLfJG7IBy tMTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=tN5fF7Wyh35qBJFCdjEuKMaMwIiqau8XmW0j2x/SMjk=; fh=YNEt5Rwl98wFdDlZq5mCbJr8YZE8zeewTDacAVth8MY=; b=UdbKt81t9GF3+ikpbo2OAu0FdvaY5PjfDQ7T8TMoTaPyaspf0mTIA0O6DwpGRgi2pX qDBDlYXVmDLvhIZqZ3JW3q2uh66RfpgDmls0ZCe5pNahBhL1HFC0QhYdRbq9Iqv+bgdp +DE4HqHMLEcFAEN3mR6FPV6s/0DorUa3nPs7gZoAuvtBKw8X8AqTMEkueu6FcOW3kvVK CTyKgy4Iq3WfirUINbXMxgJjTGZUFzWHczDjMQ3fVdhvVx037h0lEQMUhetlVvChHm4R PIYmWBI8EA7bfj3qCQXR1zQJ3XlwPJQvAGO4dCVQdfuxnlkVVPzrzN6vat8WYc/rhjUB 2sQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=OBxsTC6y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m21-20020a1709066d1500b009930e13beb8si117963ejr.952.2023.08.01.14.46.21; Tue, 01 Aug 2023 14:46:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=OBxsTC6y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230162AbjHAVZB (ORCPT + 99 others); Tue, 1 Aug 2023 17:25:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232257AbjHAVYz (ORCPT ); Tue, 1 Aug 2023 17:24:55 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3BE035A4 for ; Tue, 1 Aug 2023 14:24:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=tN5fF7Wyh35qBJFCdjEuKMaMwIiqau8XmW0j2x/SMjk=; b=OBxsTC6ytcWiY/yE6bTaLRGenU UXNPiXEek3EpXmglUInzCpiKgnS3Xz9uHEbo7oY5WDfMfRjUz6q3DK9ojN+5Su+zHJowi8JZkdBJ8 rZtpQZszObYqufmw4gEPMk5APgE2bWfFNTlvKj10VcO2RwkEUplUW2a7aLx0tk2rTvhA1MBl95JeA MLf1QC02VFnIlh00Zr9IL404fYayP1cF/EJqh1gO6LAcLlbPHhXEWK/jbPzOwChgZjJR1QxrVvTp9 Ss9CbChT2TfWVE/c/meuiPSnO7OW5RaPYlArIXzJru3Hk79gP0WwXf/+mZQHrSIM7limw+vBxut7M /lGBWahg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qQwqn-00B5yk-GZ; Tue, 01 Aug 2023 21:24:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id BEE9330334C; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 5B5942026B3DC; Tue, 1 Aug 2023 23:24:15 +0200 (CEST) Message-ID: <20230801211812.371787909@infradead.org> User-Agent: quilt/0.66 Date: Tue, 01 Aug 2023 22:41:30 +0200 From: Peter Zijlstra To: mingo@redhat.com Cc: peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org Subject: [PATCH 9/9] sched: Simplify sched_core_cpu_{starting,deactivate}() References: <20230801204121.929256934@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1773064846516016790 X-GMAIL-MSGID: 1773064846516016790 Use guards to reduce gotos and simplify control flow. Signed-off-by: Peter Zijlstra (Intel) --- kernel/sched/core.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6331,20 +6331,24 @@ static void queue_core_balance(struct rq queue_balance_callback(rq, &per_cpu(core_balance_head, rq->cpu), sched_core_balance); } +DEFINE_LOCK_GUARD_1(core_lock, int, + sched_core_lock(*_T->lock, &_T->flags), + sched_core_unlock(*_T->lock, &_T->flags), + unsigned long flags) + static void sched_core_cpu_starting(unsigned int cpu) { const struct cpumask *smt_mask = cpu_smt_mask(cpu); struct rq *rq = cpu_rq(cpu), *core_rq = NULL; - unsigned long flags; int t; - sched_core_lock(cpu, &flags); + guard(core_lock)(&cpu); WARN_ON_ONCE(rq->core != rq); /* if we're the first, we'll be our own leader */ if (cpumask_weight(smt_mask) == 1) - goto unlock; + return; /* find the leader */ for_each_cpu(t, smt_mask) { @@ -6358,7 +6362,7 @@ static void sched_core_cpu_starting(unsi } if (WARN_ON_ONCE(!core_rq)) /* whoopsie */ - goto unlock; + return; /* install and validate core_rq */ for_each_cpu(t, smt_mask) { @@ -6369,29 +6373,25 @@ static void sched_core_cpu_starting(unsi WARN_ON_ONCE(rq->core != core_rq); } - -unlock: - sched_core_unlock(cpu, &flags); } static void sched_core_cpu_deactivate(unsigned int cpu) { const struct cpumask *smt_mask = cpu_smt_mask(cpu); struct rq *rq = cpu_rq(cpu), *core_rq = NULL; - unsigned long flags; int t; - sched_core_lock(cpu, &flags); + guard(core_lock)(&cpu); /* if we're the last man standing, nothing to do */ if (cpumask_weight(smt_mask) == 1) { WARN_ON_ONCE(rq->core != rq); - goto unlock; + return; } /* if we're not the leader, nothing to do */ if (rq->core != rq) - goto unlock; + return; /* find a new leader */ for_each_cpu(t, smt_mask) { @@ -6402,7 +6402,7 @@ static void sched_core_cpu_deactivate(un } if (WARN_ON_ONCE(!core_rq)) /* impossible */ - goto unlock; + return; /* copy the shared state to the new leader */ core_rq->core_task_seq = rq->core_task_seq; @@ -6424,9 +6424,6 @@ static void sched_core_cpu_deactivate(un rq = cpu_rq(t); rq->core = core_rq; } - -unlock: - sched_core_unlock(cpu, &flags); } static inline void sched_core_cpu_dying(unsigned int cpu)