From patchwork Fri Nov 24 22:32:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169593 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580439vqx; Fri, 24 Nov 2023 14:32:56 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTxCESt9/u881231x7Bf92awAl8C0YLqg2HVvxo8stR8RAgeZ4vemzRzLgtxWDtkmLpuy5 X-Received: by 2002:a05:6a00:9385:b0:6c6:b15:392 with SMTP id ka5-20020a056a00938500b006c60b150392mr5356055pfb.24.1700865176526; Fri, 24 Nov 2023 14:32:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865176; cv=none; d=google.com; s=arc-20160816; b=pXGZzALfjN7WMSAhm3hWl8X4JTLZQ6BFY5yMRHL5PVgTlhG65clCxbzVl09+ZR5kXh LbKZocARLe95F7dgF9s+b/3649LialPuc9Biuz3YpI0xARgC3XgvBLNEXrnj+xrIdUSV lPKQFqrU+8dqVANZwTgYOoq9COfsbA3tn2rmOUmm1YdpHSC+fo0a0QNbQ08Yz/uJhF+1 LjBqSojQsrRvWv1OS7mjTQQChTW0Jwc/OAYSfWb6z2PlUpepyUOk/i8sMZkJinuPCPP2 urTC5EsdxXQipaITJIDm1dW+Ij7GwIsMuwKcyZJ5QS8GRNIfpRtE69q8ymj0mKb+NNk4 j7KA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rYbpj2Bk4kub7HDN8+41XPy4sMNBEzeqtZNmcv3C1Ic=; fh=IDVMcnS6D26GUyxqaB9flHqld6ARLJ0tlhgNrVL7FDw=; b=XoFpXjxtZ+c9HHbxlbOv/fpwH8Inm/CmUMwKLADOiF5scsRJJ5tLPDlI7kCg/xyMX7 LgUYkX8lV3fo9KXt3NHMktH3ed20jdFhaFwHC8y/Y/M2toOYv+J9Pm46rWzbSQXRIVRg tx4gSQJvdYeG3/qu4gcAhhKgjD2cw+PAgVUBpfLBaqxRqxoxi0HkDVx5z8GFe+e9+e9r Umgso8Xmm9INIj/iF0wdKtI83wL0ALRWQ9IWgstKJBqR4wxow4OgobkC7gjBKdX2dEeI Ch5pBbiLRoLxuOjrex/lfivWYlfqTovCdg+WkUBpg81HJXzpOxsJvlWjf3i2ClbcBhl5 CEsA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HPa6VZmh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z32-20020a631920000000b005bd66bfc253si4452892pgl.148.2023.11.24.14.32.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:32:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=HPa6VZmh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 2CB2E80B01F1; Fri, 24 Nov 2023 14:32:55 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231512AbjKXWck (ORCPT + 99 others); Fri, 24 Nov 2023 17:32:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230074AbjKXWch (ORCPT ); Fri, 24 Nov 2023 17:32:37 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5756810FB for ; Fri, 24 Nov 2023 14:32:44 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 90621C433C7; Fri, 24 Nov 2023 22:32:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865164; bh=CXS0kf7QfS6H/bXiY0zJwq5tEcqIiy9tCvQovQrbkN4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HPa6VZmhmx1d/lcTIANzys+lP9tpG585RQPkJGTDl5qaYFfx4ng4+D3Ue51VjCGVl +YxcXCxqvYdyVGph2cL4+kMGm5zFSgpCHg5EBFOVd/8zwyRWjqOFBk6snNRHBTH1wR OFJBStdXapHsWXqoti/A4nXAHrp8NESBWDq5BJBGYQ1eOE1jlVlHkg3NaSap/6tNoS yb6GgWzrn1W2nuFIxs8TJFNr334MzPXjLxHCmzBN3R1FFILiFAcXnSztVzBPWlNXWN N/zdiw303k4a4e8NdAGfP9X1Loxo+a/jrmE/Qe/uz4Z/Bq5ZE758VybQJPlKasDCZI V7UcKca+vdGMQ== From: Frederic Weisbecker To: LKML Cc: Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Frederic Weisbecker Subject: [PATCH 1/7] x86: Fix CPUIDLE_FLAG_IRQ_ENABLE leaking timer reprogram Date: Fri, 24 Nov 2023 23:32:20 +0100 Message-ID: <20231124223226.24249-2-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:32:55 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486403023847160 X-GMAIL-MSGID: 1783486403023847160 From: Peter Zijlstra intel_idle_irq() re-enables IRQs very early. As a result, an interrupt may fire before mwait() is eventually called. If such an interrupt queues a timer, it may go unnoticed until mwait returns and the idle loop handles the tick re-evaluation. And monitoring TIF_NEED_RESCHED doesn't help because a local timer enqueue doesn't set that flag. The issue is mitigated by the fact that this idle handler is only invoked for shallow C-states when, presumably, the next tick is supposed to be close enough. There may still be rare cases though when the next tick is far away and the selected C-state is shallow, resulting in a timer getting ignored for a while. Fix this with using sti_mwait() whose IRQ-reenablement only triggers upon calling mwait(), dealing with the race while keeping the interrupt latency within acceptable bounds. Fixes: c227233ad64c (intel_idle: enable interrupts before C1 on Xeons) Signed-off-by: Peter Zijlstra Acked-by: Rafael J. Wysocki Signed-off-by: Frederic Weisbecker --- arch/x86/include/asm/mwait.h | 11 +++++++++-- drivers/idle/intel_idle.c | 19 +++++++------------ 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h index 778df05f8539..bae83810505b 100644 --- a/arch/x86/include/asm/mwait.h +++ b/arch/x86/include/asm/mwait.h @@ -115,8 +115,15 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo } __monitor((void *)¤t_thread_info()->flags, 0, 0); - if (!need_resched()) - __mwait(eax, ecx); + + if (!need_resched()) { + if (ecx & 1) { + __mwait(eax, ecx); + } else { + __sti_mwait(eax, ecx); + raw_local_irq_disable(); + } + } } current_clr_polling(); } diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index dcda0afecfc5..3e01a6b23e75 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -131,11 +131,12 @@ static unsigned int mwait_substates __initdata; #define MWAIT2flg(eax) ((eax & 0xFF) << 24) static __always_inline int __intel_idle(struct cpuidle_device *dev, - struct cpuidle_driver *drv, int index) + struct cpuidle_driver *drv, + int index, bool irqoff) { struct cpuidle_state *state = &drv->states[index]; unsigned long eax = flg2MWAIT(state->flags); - unsigned long ecx = 1; /* break on interrupt flag */ + unsigned long ecx = 1*irqoff; /* break on interrupt flag */ mwait_idle_with_hints(eax, ecx); @@ -159,19 +160,13 @@ static __always_inline int __intel_idle(struct cpuidle_device *dev, static __cpuidle int intel_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - return __intel_idle(dev, drv, index); + return __intel_idle(dev, drv, index, true); } static __cpuidle int intel_idle_irq(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - int ret; - - raw_local_irq_enable(); - ret = __intel_idle(dev, drv, index); - raw_local_irq_disable(); - - return ret; + return __intel_idle(dev, drv, index, false); } static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev, @@ -184,7 +179,7 @@ static __cpuidle int intel_idle_ibrs(struct cpuidle_device *dev, if (smt_active) __update_spec_ctrl(0); - ret = __intel_idle(dev, drv, index); + ret = __intel_idle(dev, drv, index, true); if (smt_active) __update_spec_ctrl(spec_ctrl); @@ -196,7 +191,7 @@ static __cpuidle int intel_idle_xstate(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { fpu_idle_fpregs(); - return __intel_idle(dev, drv, index); + return __intel_idle(dev, drv, index, true); } /** From patchwork Fri Nov 24 22:32:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169594 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580526vqx; Fri, 24 Nov 2023 14:33:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IEHUAps6S/RzCQwu11nUg6w4cTthkAKRcB6bEjRFYadMtjj7CQa9ubz+osybUnD9s30qp06 X-Received: by 2002:a05:6a21:7891:b0:18b:950d:de3b with SMTP id bf17-20020a056a21789100b0018b950dde3bmr5941766pzc.38.1700865185007; Fri, 24 Nov 2023 14:33:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865184; cv=none; d=google.com; s=arc-20160816; b=JSkXwa00HCs9LuEPWeGzNZStYKB3JSKCwNzHiiDSkFAo8fvi1vbrSb1cmJw/XVob2y mAPdfosfbqmcyYPSPRjqD/gpLizN2ptENFK6Q3lDdvDpin5GVrFDHB699lNsXYrH+fAy posQ6O2Ln1up6Mr3yeKn16DGiFvqsIeGU8HgNBCir0WUIoB5HDVYLhc4GSyM1wcxTR2c 5EDNgBolV6xIpLsmGHmLenvk4z9QoJNEa92aR9sxz8iyuGE9Gr1LFqQm8zOEZoXzGqRL 9mN5+kpphDE7g+ByUtJTn6IzSMyrXX6rVnR6Ux2t+vqZl914Csg5Uo3AzIB2PrzFeUEH G6Lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=moq9PRfsPBX9cKDCPNLdFWCMWjIroUd98HLlD4iBVI0=; fh=FP7vh/uLSL8T47BAINXnJeKRz+lJwSPdPP4Si4L9mlE=; b=QYFdicVCEjT1unW5fvooFyOu5qyp/ke7r6aW9u5+w0mrZrf+gvOMW+ZL8hys+H2pma m48+KUnQ/rTRHjiqpZAAAyZkf1zomLdq00uiG1bQjIdSwr8FNl2cRjj/yQDblo+K0QYB ivoOZIyFDiWYcXlo60Vx5TtFbm/v0F5VFj/LTt1MLoyOVVEAXbuEQywTZGI83hZjqBcH v5DGzWVjWARBJl9NxmwL1xZ5x1f9XvDXhIOYoGGEEPlGQ9IUmjMUub7u96sgoHRhwHXy +UpxVoDwK03Cx4gGMckIVC72fdrRnt228jE82XgMen+Y5hXKhLX2U6Omd0OIe9Pbenuk Xvcg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="q/lnFaSV"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id ml5-20020a17090b360500b00285494e7747si4433795pjb.167.2023.11.24.14.33.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:33:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="q/lnFaSV"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 754A38026C16; Fri, 24 Nov 2023 14:32:58 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230074AbjKXWcl (ORCPT + 99 others); Fri, 24 Nov 2023 17:32:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231532AbjKXWck (ORCPT ); Fri, 24 Nov 2023 17:32:40 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19F8710E2 for ; Fri, 24 Nov 2023 14:32:47 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7070AC433CA; Fri, 24 Nov 2023 22:32:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865166; bh=SruwsS2l79TbUPDZT8+hJEb5a4kT01b9VD/LZ7J5AKM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q/lnFaSVMQiMkEsslwmzjqzy22MQXd8r9dOLxCa7TLWnoQvJK3pQYuE+Q6CrbEoyR 39RMK7baBkVlSAzMhxWIcg0mYsY1sVWkYmD7FkCurSO46YSXqsjEIDzPqOUbgXsK63 FrjQ75lmSpZu6ETzAaNH/vz7tuQvdYh2RmvwuKVhJT3Osmh1h/AU7TIvg2inx3mdMd ZSEbZeLhQldnmaE5WqzqY3UPniXAsGIpPw26G6aZrTQOnaTyR73iSfEFzIgSYztgH1 740u66mQXDlK06o2pQUyCJzQf3vn4xdoPoqlU1Bn+bD2I/HX6a0CQkejm6GKWo+zh0 jjdqFVncadX3A== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra Subject: [PATCH 2/7] x86: Add a comment about the "magic" behind shadow sti before mwait Date: Fri, 24 Nov 2023 23:32:21 +0100 Message-ID: <20231124223226.24249-3-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:32:59 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486412216280617 X-GMAIL-MSGID: 1783486412216280617 Add a note to make sure we never miss and break the requirements behind it. Acked-by: Rafael J. Wysocki Signed-off-by: Frederic Weisbecker --- arch/x86/include/asm/mwait.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h index bae83810505b..920426d691ce 100644 --- a/arch/x86/include/asm/mwait.h +++ b/arch/x86/include/asm/mwait.h @@ -87,6 +87,15 @@ static __always_inline void __mwaitx(unsigned long eax, unsigned long ebx, :: "a" (eax), "b" (ebx), "c" (ecx)); } +/* + * Re-enable interrupts right upon calling mwait in such a way that + * no interrupt can fire _before_ the execution of mwait, ie: no + * instruction must be placed between "sti" and "mwait". + * + * This is necessary because if an interrupt queues a timer before + * executing mwait, it would otherwise go unnoticed and the next tick + * would not be reprogrammed accordingly before mwait ever wakes up. + */ static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx) { mds_idle_clear_cpu_buffers(); From patchwork Fri Nov 24 22:32:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169596 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580684vqx; Fri, 24 Nov 2023 14:33:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IGhWID1TrF1ak6TBXO0dedf6nF+kPuuC6ZWB2XQC+0NnB02VcaZYV+QtZ4EgmOLnZKvkLCo X-Received: by 2002:a05:6a00:8ce:b0:6c3:1b7d:ecee with SMTP id s14-20020a056a0008ce00b006c31b7deceemr5783483pfu.15.1700865207762; Fri, 24 Nov 2023 14:33:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865207; cv=none; d=google.com; s=arc-20160816; b=YWzJCcvoME1NGW4Qab2h3kVrKBV+1Q0xDNF/plADWfahpWNHpndYMff1ByNcZ7+Na7 4MlHqPl3mddGvQZYV4b0LKsLLXxPd4KRyY9OBEtYUggUgKn6MaJzXbAd5t1LyGw42pio SZNwnVUBiaYA+I0B9nU41+BCEDDr/EHLmH/wRT6PVsRlmEPYQRoVZLp8BtW7tXZYusrG jU24JN9zrSnGEDIuF2IaVmcRZBuUNmhKMYiblqrkK1wHuEZNOAqiWtsBp4jf+1Wn7y01 eCcNbtXLiD146NTdZ3rKHKv/2DYlqJHuO2FFbmnB3sln47Wl1Y/wXEvBnLuD4D4t02iP 0kBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=L9AD4npZHNE7p07jGPwHmr8Xy3EVUnv33DYi1PDLLmQ=; fh=kadRONg1sMHv05REhOXqG2OstPrHhRU52xPLYfj4eAI=; b=tVPwjM6ksszVt0bm07wYksW2H96vOGYxcr0vVVgb8e4h93VKhQmI4O1Z+gDz4cgEHs lcX8xdOHTGqwEXXYgHVBzGTQmb+OQMzi2JqnrMYc3HFCwz23Hh+ARUR5cJ6CPe5+eT99 Jfnr9v37nvq74h2hz3Mzu7NQbttZI0PfmBB6NOvANk24ofq03dFbzdQ4e/WP3jGe8RNx blDz0Sp7N8nm/i1bauTUc05C8apEU9+hfFY1WgNX0vPDfPxvJpY5rXOu0mpe1SKLRKH6 C09OdH1jPjUvoxxFXRnqc5lUYJrC0vIVC1cPYKolPAKK6wlWuNGtrDSwFb1YeRfEeF1d ztQA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rHOgFJue; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id cn4-20020a056a020a8400b005b9b68add85si4534793pgb.254.2023.11.24.14.33.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:33:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rHOgFJue; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 8E58880C4DBC; Fri, 24 Nov 2023 14:33:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231512AbjKXWct (ORCPT + 99 others); Fri, 24 Nov 2023 17:32:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231532AbjKXWco (ORCPT ); Fri, 24 Nov 2023 17:32:44 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC7B610E2 for ; Fri, 24 Nov 2023 14:32:49 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 287BFC433C8; Fri, 24 Nov 2023 22:32:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865169; bh=sUhE/qFaIDHh5RCqYIo7aaxz5Ifr+Yh3RMfcwpeT7hc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rHOgFJueqacJh3oSGholAxY008YeuYv2dLD8Hx1i+J42ckaCsPEue+U26TfZlARwm JZW9by+SVFIhonSoEAeY96DUR5A6LRpOk6l+X5lW7jWz8WuKOZORla5W+Tk1XuALgj MYwIK3qgK9tew6YwvquWXqWd54fnBITkSQXVnS19hVQgzWyRlYjXhPvU4Rg1N4BvgV AjpcEKOFBoBsU0hsUGCECscZ6jxHwYsEELkWxGucl5t5JB8X3Y+QjKZW/MajGaie0o MfNTU2ss+3GBqu3nevft7gW6YovD70zEZAYatkvvZs8AwuxvfnjJpAPewKS2KYK1MV H6HsE5Vo/TsRA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra , Marcelo Tosatti Subject: [PATCH 3/7] cpuidle: Remove unnecessary current_clr_polling_and_test() from haltpoll Date: Fri, 24 Nov 2023 23:32:22 +0100 Message-ID: <20231124223226.24249-4-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:33:22 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486435924197907 X-GMAIL-MSGID: 1783486435924197907 When cpuidle drivers ->enter() callback are called, the TIF_NR_POLLING flag is cleared already and TIF_NEED_RESCHED checked by call_cpuidle(). Therefore calling current_clr_polling_and_test() is redundant here and further setting of TIF_NEED_RESCHED will result in an IPI and thus an idle loop exit. This call can be safely removed. Cc: Marcelo Tosatti Acked-by: Rafael J. Wysocki Signed-off-by: Frederic Weisbecker --- drivers/cpuidle/cpuidle-haltpoll.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/drivers/cpuidle/cpuidle-haltpoll.c b/drivers/cpuidle/cpuidle-haltpoll.c index e66df22f9695..b641bc535102 100644 --- a/drivers/cpuidle/cpuidle-haltpoll.c +++ b/drivers/cpuidle/cpuidle-haltpoll.c @@ -28,11 +28,8 @@ static enum cpuhp_state haltpoll_hp_state; static int default_enter_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - if (current_clr_polling_and_test()) { - local_irq_enable(); - return index; - } arch_cpu_idle(); + return index; } From patchwork Fri Nov 24 22:32:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169595 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580568vqx; Fri, 24 Nov 2023 14:33:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IFRSzT3W+ZxsMtxreVcQiaRT++gi5jrr1wSJuvkUfiYL406+Z8dDx4M0f/QxE6hVe9AdSuE X-Received: by 2002:a17:902:ab17:b0:1cf:59df:241 with SMTP id ik23-20020a170902ab1700b001cf59df0241mr3714769plb.52.1700865192158; Fri, 24 Nov 2023 14:33:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865192; cv=none; d=google.com; s=arc-20160816; b=bP1Kcyfc1YEfNMzeqcASHeBo1LMp0PGrxFJL7nkWhKY8Z5BI9PdJ01X7eKPQRRWDto 9SwUFjkGWQOZgm256YWcO1yTW48eKMZl+FnYbUlhpfkFjN4k1r8DOxCgFj7I3nsgvs9a jadae8SU1vYXIm11DJ1xEnrLXPt0P+qKodWTmQttkSlET2Hdakp1l8EucYQI9LcMNBDs 7Srf545h72YfaOxYsd0w8zYwhsUmRylh36EcgHNqf+AiK9stYh+vkm2cgUjafDB4zNqG cTDTpRkjv1GclEiSLsNK6sCm0liIiqExXHcrdLl8IfJD33fz3iC1KFhXkZhGAb3hHy5i fl9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XIlF9RYnvaXIRTiaVCU3k8IeTF00dtud4a3/nx+W6Qc=; fh=FP7vh/uLSL8T47BAINXnJeKRz+lJwSPdPP4Si4L9mlE=; b=s0lAnP7hZF/RtCsJOHLB/59YYILvjFZrGe+IzbzF0k4E3Jl/qJm5LaZ1UEPQx+JMD0 cF3uhPBfY+VZnkua9OgQaizfUZsKYA3IMsIk8OVc3PtqfrwsYiL8HJUGqXpPjCk8DRH3 b8DmmGY8Q+4PaNg6bkcFc/q5mHGPj9wo+h41XR8LPKZPGQlaGump0yDrpmSm60euQouv tdESFLZdlYn6a7M9bubqllC/+5QEsInGF1QzXPIfkmJ3hG8hHsTSr8MydZJY66PCUnFj kaRO5M8QdUKJ9NVa1f+qlUH4npprqcA4IwFst1tvK8P4FdkXFfe1e4VKzGG5IdV9RC21 T1kg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oNVRPsBE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id h6-20020a170902f7c600b001cf920235edsi3632510plw.88.2023.11.24.14.33.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:33:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oNVRPsBE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id ECF8880FCE6F; Fri, 24 Nov 2023 14:33:08 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231585AbjKXWcw (ORCPT + 99 others); Fri, 24 Nov 2023 17:32:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231550AbjKXWct (ORCPT ); Fri, 24 Nov 2023 17:32:49 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC8321BD2 for ; Fri, 24 Nov 2023 14:32:52 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 12448C433CB; Fri, 24 Nov 2023 22:32:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865172; bh=EVmYFcQRtRqs/jn3gEWSri/kyXx/B+VGrmgyGEWXJ1s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oNVRPsBEfcRPpE1NgeqTaUNReFI/47tRe7hMPfCPcbXIperxi5Nm8V6Y+qJ67BzzK UO5BUefTkpk6nSk4x9pTJHkFtsAGiX8CF+u00My6GTx/dqRC8f9ZKv8ALqUKxreiHp a90rE9LdX2/bKdtIZfqY52sRMmvZSO4Zd2YDyFx6tF72H14hjDGP5xDBalSWtNErcg iZIPsU5ZxOc6vbVycnLLq5hU0w6v0vAp23AtyxfvadzG3TEUZnZCevR9DsxS0G7PQO QBpnNBP+hyd5c2hNULOYFDTVkzaLnWCz6YZQ/PYvuoppl6rLYBEH0lrHKqvvsYTKFq 6Nsxc4OF0NxqA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra Subject: [PATCH 4/7] cpuidle: s/CPUIDLE_FLAG_POLLING/CPUIDLE_FLAG_POLLING_SOFT Date: Fri, 24 Nov 2023 23:32:23 +0100 Message-ID: <20231124223226.24249-5-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:33:09 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486420018761023 X-GMAIL-MSGID: 1783486420018761023 In order to further distinguish software and hardware TIF_NEED_RESCHED polling cpuidle states, rename CPUIDLE_FLAG_POLLING to CPUIDLE_FLAG_POLLING_SOFT before introducing CPUIDLE_FLAG_POLLING_HARD and tag mwait users with it. This will allow cpuidle core to manage TIF_NR_POLLING on behalf of all kinds of TIF_NEED_RESCHED polling states while keeping a necessary distinction for the governors between software loops polling on TIF_NEED_RESCHED and hardware monitored writes to thread flags. Signed-off-by: Frederic Weisbecker --- Documentation/driver-api/pm/cpuidle.rst | 2 +- drivers/cpuidle/cpuidle-powernv.c | 2 +- drivers/cpuidle/cpuidle-pseries.c | 4 ++-- drivers/cpuidle/governors/ladder.c | 4 ++-- drivers/cpuidle/governors/menu.c | 8 ++++---- drivers/cpuidle/governors/teo.c | 8 ++++---- drivers/cpuidle/poll_state.c | 2 +- include/linux/cpuidle.h | 2 +- 8 files changed, 16 insertions(+), 16 deletions(-) diff --git a/Documentation/driver-api/pm/cpuidle.rst b/Documentation/driver-api/pm/cpuidle.rst index d477208604b8..5ad10dad8033 100644 --- a/Documentation/driver-api/pm/cpuidle.rst +++ b/Documentation/driver-api/pm/cpuidle.rst @@ -192,7 +192,7 @@ governors for computations related to idle state selection: :c:member:`flags` Flags representing idle state properties. Currently, governors only use - the ``CPUIDLE_FLAG_POLLING`` flag which is set if the given object + the ``CPUIDLE_FLAG_POLLING_SOFT`` flag which is set if the given object does not represent a real idle state, but an interface to a software "loop" that can be used in order to avoid asking the processor to enter any idle state at all. [There are other flags used by the ``CPUIdle`` diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c index 9ebedd972df0..675b8eb81ebd 100644 --- a/drivers/cpuidle/cpuidle-powernv.c +++ b/drivers/cpuidle/cpuidle-powernv.c @@ -158,7 +158,7 @@ static struct cpuidle_state powernv_states[CPUIDLE_STATE_MAX] = { .exit_latency = 0, .target_residency = 0, .enter = snooze_loop, - .flags = CPUIDLE_FLAG_POLLING }, + .flags = CPUIDLE_FLAG_POLLING_SOFT }, }; static int powernv_cpuidle_cpu_online(unsigned int cpu) diff --git a/drivers/cpuidle/cpuidle-pseries.c b/drivers/cpuidle/cpuidle-pseries.c index 14db9b7d985d..4e08c9a39172 100644 --- a/drivers/cpuidle/cpuidle-pseries.c +++ b/drivers/cpuidle/cpuidle-pseries.c @@ -271,7 +271,7 @@ static struct cpuidle_state dedicated_states[NR_DEDICATED_STATES] = { .exit_latency = 0, .target_residency = 0, .enter = &snooze_loop, - .flags = CPUIDLE_FLAG_POLLING }, + .flags = CPUIDLE_FLAG_POLLING_SOFT }, { /* CEDE */ .name = "CEDE", .desc = "CEDE", @@ -290,7 +290,7 @@ static struct cpuidle_state shared_states[] = { .exit_latency = 0, .target_residency = 0, .enter = &snooze_loop, - .flags = CPUIDLE_FLAG_POLLING }, + .flags = CPUIDLE_FLAG_POLLING_SOFT }, { /* Shared Cede */ .name = "Shared Cede", .desc = "Shared Cede", diff --git a/drivers/cpuidle/governors/ladder.c b/drivers/cpuidle/governors/ladder.c index 8e9058c4ea63..a5f462b60d50 100644 --- a/drivers/cpuidle/governors/ladder.c +++ b/drivers/cpuidle/governors/ladder.c @@ -69,7 +69,7 @@ static int ladder_select_state(struct cpuidle_driver *drv, struct ladder_device *ldev = this_cpu_ptr(&ladder_devices); struct ladder_device_state *last_state; int last_idx = dev->last_state_idx; - int first_idx = drv->states[0].flags & CPUIDLE_FLAG_POLLING ? 1 : 0; + int first_idx = drv->states[0].flags & CPUIDLE_FLAG_POLLING_SOFT ? 1 : 0; s64 latency_req = cpuidle_governor_latency_req(dev->cpu); s64 last_residency; @@ -133,7 +133,7 @@ static int ladder_enable_device(struct cpuidle_driver *drv, struct cpuidle_device *dev) { int i; - int first_idx = drv->states[0].flags & CPUIDLE_FLAG_POLLING ? 1 : 0; + int first_idx = drv->states[0].flags & CPUIDLE_FLAG_POLLING_SOFT ? 1 : 0; struct ladder_device *ldev = &per_cpu(ladder_devices, dev->cpu); struct ladder_device_state *lstate; struct cpuidle_state *state; diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c index b96e3da0fedd..98ec067805b6 100644 --- a/drivers/cpuidle/governors/menu.c +++ b/drivers/cpuidle/governors/menu.c @@ -320,7 +320,7 @@ static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, * it right away and keep the tick running if state[0] is a * polling one. */ - *stop_tick = !(drv->states[0].flags & CPUIDLE_FLAG_POLLING); + *stop_tick = !(drv->states[0].flags & CPUIDLE_FLAG_POLLING_SOFT); return 0; } @@ -365,7 +365,7 @@ static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, * Use a physical idle state, not busy polling, unless * a timer is going to trigger soon enough. */ - if ((drv->states[idx].flags & CPUIDLE_FLAG_POLLING) && + if ((drv->states[idx].flags & CPUIDLE_FLAG_POLLING_SOFT) && s->exit_latency_ns <= latency_req && s->target_residency_ns <= data->next_timer_ns) { predicted_ns = s->target_residency_ns; @@ -411,7 +411,7 @@ static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, * Don't stop the tick if the selected state is a polling one or if the * expected idle duration is shorter than the tick period length. */ - if (((drv->states[idx].flags & CPUIDLE_FLAG_POLLING) || + if (((drv->states[idx].flags & CPUIDLE_FLAG_POLLING_SOFT) || predicted_ns < TICK_NSEC) && !tick_nohz_tick_stopped()) { *stop_tick = false; @@ -492,7 +492,7 @@ static void menu_update(struct cpuidle_driver *drv, struct cpuidle_device *dev) * duration predictor do a better job next time. */ measured_ns = 9 * MAX_INTERESTING / 10; - } else if ((drv->states[last_idx].flags & CPUIDLE_FLAG_POLLING) && + } else if ((drv->states[last_idx].flags & CPUIDLE_FLAG_POLLING_SOFT) && dev->poll_time_limit) { /* * The CPU exited the "polling" state due to a time limit, so diff --git a/drivers/cpuidle/governors/teo.c b/drivers/cpuidle/governors/teo.c index 7244f71c59c5..f86e16d0ffac 100644 --- a/drivers/cpuidle/governors/teo.c +++ b/drivers/cpuidle/governors/teo.c @@ -354,7 +354,7 @@ static int teo_find_shallower_state(struct cpuidle_driver *drv, for (i = state_idx - 1; i >= 0; i--) { if (dev->states_usage[i].disable || - (no_poll && drv->states[i].flags & CPUIDLE_FLAG_POLLING)) + (no_poll && drv->states[i].flags & CPUIDLE_FLAG_POLLING_SOFT)) continue; state_idx = i; @@ -426,7 +426,7 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, * all. If state 1 is disabled, though, state 0 must be used * anyway. */ - if ((!idx && !(drv->states[0].flags & CPUIDLE_FLAG_POLLING) && + if ((!idx && !(drv->states[0].flags & CPUIDLE_FLAG_POLLING_SOFT) && teo_state_ok(0, drv)) || dev->states_usage[1].disable) { idx = 0; goto out_tick; @@ -584,7 +584,7 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, * the current candidate state is low enough and skip the timers * check in that case too. */ - if ((drv->states[0].flags & CPUIDLE_FLAG_POLLING) && + if ((drv->states[0].flags & CPUIDLE_FLAG_POLLING_SOFT) && drv->states[idx].target_residency_ns < RESIDENCY_THRESHOLD_NS) goto out_tick; @@ -616,7 +616,7 @@ static int teo_select(struct cpuidle_driver *drv, struct cpuidle_device *dev, * one or the expected idle duration is shorter than the tick period * length. */ - if ((!(drv->states[idx].flags & CPUIDLE_FLAG_POLLING) && + if ((!(drv->states[idx].flags & CPUIDLE_FLAG_POLLING_SOFT) && duration_ns >= TICK_NSEC) || tick_nohz_tick_stopped()) return idx; diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c index 9b6d90a72601..a2fe173de117 100644 --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -57,6 +57,6 @@ void cpuidle_poll_state_init(struct cpuidle_driver *drv) state->target_residency_ns = 0; state->power_usage = -1; state->enter = poll_idle; - state->flags = CPUIDLE_FLAG_POLLING; + state->flags = CPUIDLE_FLAG_POLLING_SOFT; } EXPORT_SYMBOL_GPL(cpuidle_poll_state_init); diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h index 3183aeb7f5b4..66b59868622c 100644 --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -78,7 +78,7 @@ struct cpuidle_state { /* Idle State Flags */ #define CPUIDLE_FLAG_NONE (0x00) -#define CPUIDLE_FLAG_POLLING BIT(0) /* polling state */ +#define CPUIDLE_FLAG_POLLING_SOFT BIT(0) /* polling state */ #define CPUIDLE_FLAG_COUPLED BIT(1) /* state applies to multiple cpus */ #define CPUIDLE_FLAG_TIMER_STOP BIT(2) /* timer is stopped on this state */ #define CPUIDLE_FLAG_UNUSABLE BIT(3) /* avoid using this state */ From patchwork Fri Nov 24 22:32:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169597 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580699vqx; Fri, 24 Nov 2023 14:33:29 -0800 (PST) X-Google-Smtp-Source: AGHT+IGy5KjTvPgn/G2x4bP9cnQU0OrZ+b8KJH4UWDvM0WnWjrmp5eS+YoTOQSFgkokUxSnrjy5g X-Received: by 2002:a92:ca0d:0:b0:35b:c7a:cdc5 with SMTP id j13-20020a92ca0d000000b0035b0c7acdc5mr6791826ils.26.1700865209357; Fri, 24 Nov 2023 14:33:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865209; cv=none; d=google.com; s=arc-20160816; b=uoi8cvEDLHTbdd204u1fC0yPL3zRC0RjMjqPNLfqZAUkRlxGlcOGzv7Ybb7rMTjwB6 7r660R0QPKIbT4TvGwKABrJoIqfcBI1M9GtpbZn3zh7HdOuxnL8ZOcy+KIvycjq1kwMW NZ/mgfOHlTPbaPlXg4lArNXt8FxntgLAD+1ZkpLfDT9wFbtu1uktkXzQkqewTOKVRRTx ckkImwyuijzRdt+Mmdodl3V+dDXMMkFhwQe40RKI9XZfi7lb0LRCwzWg41DxJ8caN3Lw b6B2H9VYW3sI7I+Qe91wp9XGFHWoZaVETCo8LggWeUnRbVnB9zJ1kA4gkR7frBxL4eCF qR2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=2FuwhzGM+VoMm2+Qkh9VlaZ5IO3Xv92E/AQxM9JLuzc=; fh=IDVMcnS6D26GUyxqaB9flHqld6ARLJ0tlhgNrVL7FDw=; b=H50Yyc4kyOz7PzbXRRqGufhmGPdaoYikwavGtHtjbefSeSOjV16qWkCtsVetkF6b5L ZG99OBMO5xiRo8sdg0QiGJMmjK66W8mdjNKe5qukpFCreMHCIb65zsJJyJqM6oT5kNOS pwBpn31r9T43+7XmMhokhob+DBbDRXdbY2SMpKfNJ7owrqXsysHo8MO8HtaEDOpcPzIb 2ECsQCXTFb7TbBmt8eOpSyGIZdskj2m3DcxNKrTl72lLcw2QkjH2pREmSAapo6IyyFlD oNSyy3g7HgugRQvdcIyuMZ+2d582/y7XLMxtmspK6wRjoYWF452wVuhYcEo43Vu7pmR2 0xwA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MhjYZ3hd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id x190-20020a6263c7000000b006cb997a5f83si4153302pfb.31.2023.11.24.14.33.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:33:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=MhjYZ3hd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id A93E98026C26; Fri, 24 Nov 2023 14:33:21 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231847AbjKXWdD (ORCPT + 99 others); Fri, 24 Nov 2023 17:33:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231688AbjKXWcw (ORCPT ); Fri, 24 Nov 2023 17:32:52 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7251919A3 for ; Fri, 24 Nov 2023 14:32:55 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B1C45C433CA; Fri, 24 Nov 2023 22:32:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865175; bh=ihqiqz3/q3v0Bf/+UnffiXh+sdvTju3BkjA83XED8zM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MhjYZ3hdtcsr4cC2W098SMu6zoephrorXSVCY5em38yU+gwCG+wIbyfqAO5PIbzKf tpgkQTEkCx2BhLpfxsUTVI1DINEJBJNxF8fz1n+KANI31MWexPQH3mJGySsf37sBIa 7JTR4ATQUG271FQF/loFOAG+Qy+pw6gXOY2hmEO/zE+7SbyQbgonGchdZuAWoBHXz3 NyqO5oLTc6q1G4XtfFP8o5ebtW7fl0Y+NHtxQN3xJsqpkEtLZz7lOm5gFGd9x4tN41 4aVIZc/aiXkcOGYn2HvpRnYA6ByplMGp8WiKQSYvKvElSVGswR05pETHCE4Ae89CMp f1LRSNULka0wQ== From: Frederic Weisbecker To: LKML Cc: Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Frederic Weisbecker Subject: [PATCH 5/7] cpuidle: Introduce CPUIDLE_FLAG_POLLING_HARD Date: Fri, 24 Nov 2023 23:32:24 +0100 Message-ID: <20231124223226.24249-6-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:33:21 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486437430508486 X-GMAIL-MSGID: 1783486437430508486 From: Peter Zijlstra Provide a way to tell the cpuidle core about states polling/monitoring TIF_NEED_RESCHED on the hardware level, monitor/mwait users being the only examples in use. This will allow cpuidle core to manage TIF_NR_POLLING on behalf of all kinds of TIF_NEED_RESCHED polling states while keeping a necessary distinction for the governors between software loops polling on TIF_NEED_RESCHED and hardware monitored writes to thread flags. [fweisbec: _ Initialize flag from acpi_processor_setup_cstates() instead of acpi_processor_setup_lpi_states(), as the latter seem to be about arm64... _ Rename CPUIDLE_FLAG_NO_IPI to CPUIDLE_FLAG_POLLING_HARD] Not-yet-signed-off-by: Peter Zijlstra Signed-off-by: Frederic Weisbecker --- drivers/acpi/processor_idle.c | 3 +++ drivers/idle/intel_idle.c | 5 ++++- include/linux/cpuidle.h | 3 ++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c index 3a34a8c425fe..a77a4d4b0dad 100644 --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -814,6 +814,9 @@ static int acpi_processor_setup_cstates(struct acpi_processor *pr) if (cx->type != ACPI_STATE_C3) drv->safe_state_index = count; } + + if (cx->entry_method == ACPI_CSTATE_FFH) + state->flags |= CPUIDLE_FLAG_POLLING_HARD; /* * Halt-induced C1 is not good for ->enter_s2idle, because it * re-enables interrupts on exit. Moreover, C1 is generally not diff --git a/drivers/idle/intel_idle.c b/drivers/idle/intel_idle.c index 3e01a6b23e75..bc56624fe0b5 100644 --- a/drivers/idle/intel_idle.c +++ b/drivers/idle/intel_idle.c @@ -1563,7 +1563,8 @@ static void __init intel_idle_init_cstates_acpi(struct cpuidle_driver *drv) if (cx->type > ACPI_STATE_C1) state->target_residency *= 3; - state->flags = MWAIT2flg(cx->address); + state->flags = MWAIT2flg(cx->address) | CPUIDLE_FLAG_POLLING_HARD; + if (cx->type > ACPI_STATE_C2) state->flags |= CPUIDLE_FLAG_TLB_FLUSHED; @@ -1836,6 +1837,8 @@ static bool __init intel_idle_verify_cstate(unsigned int mwait_hint) static void state_update_enter_method(struct cpuidle_state *state, int cstate) { + state->flags |= CPUIDLE_FLAG_POLLING_HARD; + if (state->flags & CPUIDLE_FLAG_INIT_XSTATE) { /* * Combining with XSTATE with IBRS or IRQ_ENABLE flags diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h index 66b59868622c..873fdf200dc3 100644 --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -78,13 +78,14 @@ struct cpuidle_state { /* Idle State Flags */ #define CPUIDLE_FLAG_NONE (0x00) -#define CPUIDLE_FLAG_POLLING_SOFT BIT(0) /* polling state */ +#define CPUIDLE_FLAG_POLLING_SOFT BIT(0) /* software need_resched() polling state */ #define CPUIDLE_FLAG_COUPLED BIT(1) /* state applies to multiple cpus */ #define CPUIDLE_FLAG_TIMER_STOP BIT(2) /* timer is stopped on this state */ #define CPUIDLE_FLAG_UNUSABLE BIT(3) /* avoid using this state */ #define CPUIDLE_FLAG_OFF BIT(4) /* disable this state by default */ #define CPUIDLE_FLAG_TLB_FLUSHED BIT(5) /* idle-state flushes TLBs */ #define CPUIDLE_FLAG_RCU_IDLE BIT(6) /* idle-state takes care of RCU */ +#define CPUIDLE_FLAG_POLLING_HARD BIT(7) /* hardware need_resched() polling state */ struct cpuidle_device_kobj; struct cpuidle_state_kobj; From patchwork Fri Nov 24 22:32:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169598 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580804vqx; Fri, 24 Nov 2023 14:33:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IFnKnIznQx9ZZYRLG4m6lndoWtSHqZ0qvZLxQKEXy1rm+2gCAL6W15e94Y9UpV2O/w2Tsd0 X-Received: by 2002:a17:902:a413:b0:1cf:7439:6908 with SMTP id p19-20020a170902a41300b001cf74396908mr5447792plq.11.1700865225859; Fri, 24 Nov 2023 14:33:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865225; cv=none; d=google.com; s=arc-20160816; b=LdRLaJokQvsbfDldXKJr+GIGkqzJUxuqVZfvf+6CR4OCXw0bEvldRh2WsX7LRaTDgM D4NiKrfibFBLRyG+qk5qivuOqH/E3Stln41fNiz8voHkl/g0/BBAlGxL3x/Cfj44ZKds dU1iWSPIKzvRixq8m5IVF4XjyuzPsrLJBYk4BJ+mgYvsk8VwBX7raIGmgSw9hSEhqvHN 1GjUucbjxjnchoFziyunERtOAyIyOwlKCToQ018fPY0WbRHVn/AV5OrX37SaE2D/8RpP NVtaRgKjiTzyxnn+xsVxUBemIwsnzI+Evj04Fj5ie6+KJc7y+40d1Z386uXoXZbEYBz+ FHQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Vf8Kv9NRq261AwJiu7Tb9WYPig3gsfChRee3H0Kd2Js=; fh=IDVMcnS6D26GUyxqaB9flHqld6ARLJ0tlhgNrVL7FDw=; b=bZKBn5VfbFJ5pznW7pfN6tTxLhZDYfmDQqv65LM88uYu0zWBg4f0TA+20Fom8oTrc5 U5Ed76/dZ7Jal0gFMnIwi9orVYAtHJnCe4iE7EpwKIveZtycDWK8sbC9XEH0VHzN8qVh 3uduPj5onQgvTxtU10aNqrAD7Ae7xmvi9yqhTj71wMcYL37Hugs2VGjPB1QWEQEsUlAk Nw+QK9UECpzmE5e70rPBgAZnD7hmoPaZVb4Voio0PF82CTvViGJj2dklNhCcQSDwSweo tfvqIZj6AT9F1yS28lESisAsO4q6j+qoVgwrp5JPQk5QXqdtKHTSrmEEydMAdKK/KiDS 9Qug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=TdpYx+ds; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id b8-20020a170902a9c800b001c9f54d26bfsi4012830plr.516.2023.11.24.14.33.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:33:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=TdpYx+ds; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 966818227434; Fri, 24 Nov 2023 14:33:42 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231493AbjKXWdK (ORCPT + 99 others); Fri, 24 Nov 2023 17:33:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231649AbjKXWc5 (ORCPT ); Fri, 24 Nov 2023 17:32:57 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32C8019BE for ; Fri, 24 Nov 2023 14:32:58 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71637C433C8; Fri, 24 Nov 2023 22:32:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865177; bh=IDZpN9p+abqwRVqn7d74rxuHAxIwi6lNakqu6PbLwr4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TdpYx+dsmek1O7PqlvFr1zhMJUwSjosoIXnXKZkctP0hjHH/SVECx05aIORgVwWqC WFpVWYWwNx8FBUpfGOj57Fa55M70+wzkU6+Ir5gIyK7ofBxAstic450rksm4fhI0l9 zGIPnDKyKGfg6bvihi1XsDdhxTucl4C/XVNbdZZsrWFo5GLLYiEgBUptU0L6RdkpTS 2s/3d36DlSW5h8BJmc6wPiyAFb86n72wzeaO6EAimDNnQhZnSudFoLMAAIGWAloiwb oVGtZEaGs0/xq6g6quyArN9lX4PN3OsgexnWvMwXd2+Ph8aJSm5qv7OhX0lUceUSY0 xw5gyEAO4HKBA== From: Frederic Weisbecker To: LKML Cc: Peter Zijlstra , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Frederic Weisbecker Subject: [PATCH 6/7] cpuidle: Handle TIF_NR_POLLING on behalf of CPUIDLE_FLAG_POLLING_HARD states Date: Fri, 24 Nov 2023 23:32:25 +0100 Message-ID: <20231124223226.24249-7-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:33:42 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486455130120426 X-GMAIL-MSGID: 1783486455130120426 From: Peter Zijlstra The current handling of TIF_NR_POLLING is a bit of a maze: 1) A common brief part in the generic idle loop sets TIF_NR_POLLING while cpuidle selects an appropriate state and the tick is evaluated and then stopped. Summary: One pair of set/clear 2) The state cpuidle is then called with TIF_NR_POLLING cleared but if the state polls on need_resched() (software or hardware), it sets again TIF_NR_POLLING and clears it when it completes. Summary: another pair of set/clear 3) goto 1) However those costly atomic operations, fully ordered RmW for some of them, could be avoided if the cpuidle core knew in advance if the target state polls on need_resched(). If so, TIF_NR_POLLING could simply be set once before entering the idle loop and cleared once after idle loop exit. Start dealing with that with handling TIF_NR_POLLING on behalf of CPUIDLE_FLAG_POLLING_HARD states. [fweisbec: _ Handle broadcast properly _ Ignore mwait_idle() as it can be used by default_idle_call()] Not-yet-signed-off-by: Peter Zijlstra Signed-off-by: Frederic Weisbecker --- arch/x86/include/asm/mwait.h | 3 +-- drivers/cpuidle/cpuidle.c | 22 +++++++++++++++++++- include/linux/sched/idle.h | 7 ++++++- kernel/sched/idle.c | 40 +++++++++++++----------------------- 4 files changed, 42 insertions(+), 30 deletions(-) diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h index 920426d691ce..3634d00e5c37 100644 --- a/arch/x86/include/asm/mwait.h +++ b/arch/x86/include/asm/mwait.h @@ -116,7 +116,7 @@ static __always_inline void __sti_mwait(unsigned long eax, unsigned long ecx) */ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned long ecx) { - if (static_cpu_has_bug(X86_BUG_MONITOR) || !current_set_polling_and_test()) { + if (static_cpu_has_bug(X86_BUG_MONITOR) || !need_resched()) { if (static_cpu_has_bug(X86_BUG_CLFLUSH_MONITOR)) { mb(); clflush((void *)¤t_thread_info()->flags); @@ -134,7 +134,6 @@ static __always_inline void mwait_idle_with_hints(unsigned long eax, unsigned lo } } } - current_clr_polling(); } /* diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c index 737a026ef58a..49078cc83f4a 100644 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -213,10 +213,10 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, int index) { int entered_state; - struct cpuidle_state *target_state = &drv->states[index]; bool broadcast = !!(target_state->flags & CPUIDLE_FLAG_TIMER_STOP); ktime_t time_start, time_end; + bool polling; instrumentation_begin(); @@ -236,6 +236,23 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, broadcast = false; } + polling = target_state->flags & CPUIDLE_FLAG_POLLING_HARD; + + /* + * If the target state doesn't poll on need_resched(), this is + * the last check after which further TIF_NEED_RESCHED remote setting + * will involve an IPI. + */ + if (!polling && current_clr_polling_and_test()) { + if (broadcast) + tick_broadcast_exit(); + dev->last_residency_ns = 0; + local_irq_enable(); + instrumentation_end(); + return -EBUSY; + } + + if (target_state->flags & CPUIDLE_FLAG_TLB_FLUSHED) leave_mm(dev->cpu); @@ -335,6 +352,9 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, dev->states_usage[index].rejected++; } + if (!polling) + __current_set_polling(); + instrumentation_end(); return entered_state; diff --git a/include/linux/sched/idle.h b/include/linux/sched/idle.h index 478084f9105e..50c13531f5d8 100644 --- a/include/linux/sched/idle.h +++ b/include/linux/sched/idle.h @@ -68,6 +68,8 @@ static __always_inline bool __must_check current_set_polling_and_test(void) static __always_inline bool __must_check current_clr_polling_and_test(void) { + bool ret; + __current_clr_polling(); /* @@ -76,7 +78,10 @@ static __always_inline bool __must_check current_clr_polling_and_test(void) */ smp_mb__after_atomic(); - return unlikely(tif_need_resched()); + ret = unlikely(tif_need_resched()); + if (ret) + __current_set_polling(); + return ret; } #else diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c index 565f8374ddbb..4e554b4e3781 100644 --- a/kernel/sched/idle.c +++ b/kernel/sched/idle.c @@ -94,11 +94,12 @@ void __cpuidle default_idle_call(void) stop_critical_timings(); ct_cpuidle_enter(); - arch_cpu_idle(); + arch_cpu_idle(); // XXX assumes !polling ct_cpuidle_exit(); start_critical_timings(); trace_cpu_idle(PWR_EVENT_EXIT, smp_processor_id()); + __current_set_polling(); } local_irq_enable(); instrumentation_end(); @@ -107,31 +108,14 @@ void __cpuidle default_idle_call(void) static int call_cpuidle_s2idle(struct cpuidle_driver *drv, struct cpuidle_device *dev) { + int ret; + if (current_clr_polling_and_test()) return -EBUSY; - return cpuidle_enter_s2idle(drv, dev); -} - -static int call_cpuidle(struct cpuidle_driver *drv, struct cpuidle_device *dev, - int next_state) -{ - /* - * The idle task must be scheduled, it is pointless to go to idle, just - * update no idle residency and return. - */ - if (current_clr_polling_and_test()) { - dev->last_residency_ns = 0; - local_irq_enable(); - return -EBUSY; - } - - /* - * Enter the idle state previously returned by the governor decision. - * This function will block until an interrupt occurs and will take - * care of re-enabling the local interrupts - */ - return cpuidle_enter(drv, dev, next_state); + ret = cpuidle_enter_s2idle(drv, dev); + __current_set_polling(); + return ret; } /** @@ -198,7 +182,7 @@ static void cpuidle_idle_call(void) tick_nohz_idle_stop_tick(); next_state = cpuidle_find_deepest_state(drv, dev, max_latency_ns); - call_cpuidle(drv, dev, next_state); + cpuidle_enter(drv, dev, next_state); } else { bool stop_tick = true; @@ -212,7 +196,12 @@ static void cpuidle_idle_call(void) else tick_nohz_idle_retain_tick(); - entered_state = call_cpuidle(drv, dev, next_state); + /* + * Enter the idle state previously returned by the governor decision. + * This function will block until an interrupt occurs and will take + * care of re-enabling the local interrupts. + */ + entered_state = cpuidle_enter(drv, dev, next_state); /* * Give the governor an opportunity to reflect on the outcome */ @@ -220,7 +209,6 @@ static void cpuidle_idle_call(void) } exit_idle: - __current_set_polling(); /* * It is up to the idle functions to reenable local interrupts From patchwork Fri Nov 24 22:32:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Weisbecker X-Patchwork-Id: 169599 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:ce62:0:b0:403:3b70:6f57 with SMTP id o2csp1580819vqx; Fri, 24 Nov 2023 14:33:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IHTV9WHnCXczCEr41XoHgF7mRMSVuZto0DgELNPmWGVGwHTSUHCEt+Q+Cj+yQBjX8i5oWYx X-Received: by 2002:a17:902:dacc:b0:1cf:a70b:39b7 with SMTP id q12-20020a170902dacc00b001cfa70b39b7mr3606398plx.43.1700865228307; Fri, 24 Nov 2023 14:33:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700865228; cv=none; d=google.com; s=arc-20160816; b=JgZuvNz24Bc4gChJWnyoLKPem7FVYGrfMHnkUE31cKVC+czi/LnuPBbGTkJjZPjKf0 myH2PfYLkli5d4pP1kENKPLvXwbAPTr7NZ+DlBUQQFtyuEKF6p+EQm+okD9P0DwtQDbF belBQZYQ4QYe3c/yX+/C/XdzxRtusFQZE2DXgpBDnAt3+22fO6smE1Uu077kl1dkWNBl zUUGEAPviX6r20HKvbzN6lqu6NZkmLwzrUnvzpxXIivKiJVXJb8u3zC6nAXmfG8N3qHC Bpq1op7OA3ik0LYHavszXpDb6pC5eXh0inZIJ+6UeMnVzBdFfYBm1Vgp7ty8XLVV43C/ 70lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4yOFnHb474rWjPxsHqMvh84l3hTm8YWaV2xwbY226lI=; fh=FP7vh/uLSL8T47BAINXnJeKRz+lJwSPdPP4Si4L9mlE=; b=sk7F31jzKwOuJO0zAm3z4fTLGsQ3/wzmmMZ/3n2I4opk66x8QwM9kIRKIkLSkoYwP1 1014iDJkFkCw1qYQvrY8yyxbXwq/yB76OrLCgIP62udk4tKz2hwfOS/OzkdG/YBLU5kj jSYM+4CEs02vRrK8DQ4/Gsl4F0/KTXFO27WOaQfqd22iJJaUGHxTEYHdPO+dzkw11zQe 00VN1Rx9jbffROMCakwcqASkvsOi6DapoCZaBQ87c79+XQqGotxoWT0urvIIVRIJ9M6G blVVxmkFyeJ3BZgWLnJ67kyGENBqTiL6zhgMJTSeqCryUvXzVtIb/A6gpTwwWO4lCk4y 15qg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LG5Q5EEg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id l10-20020a170903120a00b001cc31c32e0fsi4473579plh.91.2023.11.24.14.33.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Nov 2023 14:33:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=LG5Q5EEg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 67E1980C711C; Fri, 24 Nov 2023 14:33:45 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235330AbjKXWdW (ORCPT + 99 others); Fri, 24 Nov 2023 17:33:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231927AbjKXWdE (ORCPT ); Fri, 24 Nov 2023 17:33:04 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A45421BE2 for ; Fri, 24 Nov 2023 14:33:00 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 32778C433CB; Fri, 24 Nov 2023 22:32:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1700865180; bh=qkwnYSoqlBcfb84gH/3LY3O2XgUdAI1yXrt+QgVwpng=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LG5Q5EEgKX6cIW/WbSemc+jpnC4AVH6XFhH2IavHYRsaAdG4Yf+u7scHj7+B1U+6v xAy6ws8g/2QZl9G9qSIivZfAAFEPDvLaEuigQutSypLnKxUcY7qegY5zBggdPdK5mM z0/EUYh2gJ00egk/QgMjyJQAAnXHEv6Ubxa7m8wa9lXW2EnpwOYlhxQmPMusty05N7 tBwsmU262FNzth+F8s3JnhxTU7QRrb25NzWeROZxbquU5igA3thQ+tVAKccsE2imGA 4zsxcSoRgJUEWm/QC/O8cjb2+EireCnBmjStyIuN4gdVp1mYDxLynBoxICn14o/EXz JCheLVxxPDzAA== From: Frederic Weisbecker To: LKML Cc: Frederic Weisbecker , "Rafael J . Wysocki" , Daniel Lezcano , linux-pm@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra Subject: [PATCH 7/7] cpuidle: Handle TIF_NR_POLLING on behalf of software polling idle states Date: Fri, 24 Nov 2023 23:32:26 +0100 Message-ID: <20231124223226.24249-8-frederic@kernel.org> X-Mailer: git-send-email 2.42.1 In-Reply-To: <20231124223226.24249-1-frederic@kernel.org> References: <20231124223226.24249-1-frederic@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 24 Nov 2023 14:33:45 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783486457842109891 X-GMAIL-MSGID: 1783486457842109891 Software polling idle states set again TIF_NR_POLLING and clear it upon exit. This involves error prone duplicated code and wasted cycles performing atomic operations, sometimes RmW fully ordered. To avoid this, benefit instead from the same generic TIF_NR_POLLING handling that is currently in use for hardware polling states. Signed-off-by: Frederic Weisbecker --- drivers/cpuidle/cpuidle-powernv.c | 10 ---------- drivers/cpuidle/cpuidle-pseries.c | 11 ----------- drivers/cpuidle/cpuidle.c | 4 ++-- drivers/cpuidle/poll_state.c | 30 ++++++++++++------------------ 4 files changed, 14 insertions(+), 41 deletions(-) diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c index 675b8eb81ebd..b88bbf7ead41 100644 --- a/drivers/cpuidle/cpuidle-powernv.c +++ b/drivers/cpuidle/cpuidle-powernv.c @@ -71,8 +71,6 @@ static int snooze_loop(struct cpuidle_device *dev, { u64 snooze_exit_time; - set_thread_flag(TIF_POLLING_NRFLAG); - local_irq_enable(); snooze_exit_time = get_tb() + get_snooze_timeout(dev, drv, index); @@ -81,21 +79,13 @@ static int snooze_loop(struct cpuidle_device *dev, HMT_very_low(); while (!need_resched()) { if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) { - /* - * Task has not woken up but we are exiting the polling - * loop anyway. Require a barrier after polling is - * cleared to order subsequent test of need_resched(). - */ - clear_thread_flag(TIF_POLLING_NRFLAG); dev->poll_time_limit = true; - smp_mb(); break; } } HMT_medium(); ppc64_runlatch_on(); - clear_thread_flag(TIF_POLLING_NRFLAG); local_irq_disable(); diff --git a/drivers/cpuidle/cpuidle-pseries.c b/drivers/cpuidle/cpuidle-pseries.c index 4e08c9a39172..0ae76512b740 100644 --- a/drivers/cpuidle/cpuidle-pseries.c +++ b/drivers/cpuidle/cpuidle-pseries.c @@ -39,8 +39,6 @@ int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv, { u64 snooze_exit_time; - set_thread_flag(TIF_POLLING_NRFLAG); - pseries_idle_prolog(); raw_local_irq_enable(); snooze_exit_time = get_tb() + snooze_timeout; @@ -50,21 +48,12 @@ int snooze_loop(struct cpuidle_device *dev, struct cpuidle_driver *drv, HMT_low(); HMT_very_low(); if (likely(snooze_timeout_en) && get_tb() > snooze_exit_time) { - /* - * Task has not woken up but we are exiting the polling - * loop anyway. Require a barrier after polling is - * cleared to order subsequent test of need_resched(). - */ dev->poll_time_limit = true; - clear_thread_flag(TIF_POLLING_NRFLAG); - smp_mb(); break; } } HMT_medium(); - clear_thread_flag(TIF_POLLING_NRFLAG); - raw_local_irq_disable(); pseries_idle_epilog(); diff --git a/drivers/cpuidle/cpuidle.c b/drivers/cpuidle/cpuidle.c index 49078cc83f4a..9eb811b5d8b6 100644 --- a/drivers/cpuidle/cpuidle.c +++ b/drivers/cpuidle/cpuidle.c @@ -236,8 +236,8 @@ noinstr int cpuidle_enter_state(struct cpuidle_device *dev, broadcast = false; } - polling = target_state->flags & CPUIDLE_FLAG_POLLING_HARD; - + polling = (target_state->flags & (CPUIDLE_FLAG_POLLING_SOFT | + CPUIDLE_FLAG_POLLING_HARD)); /* * If the target state doesn't poll on need_resched(), this is * the last check after which further TIF_NEED_RESCHED remote setting diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c index a2fe173de117..3bfa251b344a 100644 --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -13,35 +13,29 @@ static int __cpuidle poll_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - u64 time_start; - - time_start = local_clock_noinstr(); + u64 time_start = local_clock_noinstr(); + unsigned int loop_count = 0; + u64 limit; dev->poll_time_limit = false; raw_local_irq_enable(); - if (!current_set_polling_and_test()) { - unsigned int loop_count = 0; - u64 limit; - limit = cpuidle_poll_time(drv, dev); + limit = cpuidle_poll_time(drv, dev); - while (!need_resched()) { - cpu_relax(); - if (loop_count++ < POLL_IDLE_RELAX_COUNT) - continue; + while (!need_resched()) { + cpu_relax(); + if (loop_count++ < POLL_IDLE_RELAX_COUNT) + continue; - loop_count = 0; - if (local_clock_noinstr() - time_start > limit) { - dev->poll_time_limit = true; - break; - } + loop_count = 0; + if (local_clock_noinstr() - time_start > limit) { + dev->poll_time_limit = true; + break; } } raw_local_irq_disable(); - current_clr_polling(); - return index; }