From patchwork Mon Dec 19 15:14:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Krzysztof Kozlowski X-Patchwork-Id: 34609 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2447283wrn; Mon, 19 Dec 2022 07:16:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXseKBpoN+r5CfgTaEWF+kksdWOAD1zTbr9Pb+iavX81UqshH03Gb9FKxCZQDxO7xKmZlywr X-Received: by 2002:a05:6402:448f:b0:47b:16f5:61dc with SMTP id er15-20020a056402448f00b0047b16f561dcmr2285319edb.37.1671462988673; Mon, 19 Dec 2022 07:16:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671462988; cv=none; d=google.com; s=arc-20160816; b=ExKIt1P0Q/8gWzkKKzG06/s4uuKo27vaZp2xL7n/DvuGHMLxVSW1JH9npvuOFIx+i1 hBGbiSDHxTawVKUPSsQfHa90YA0hyrJAcl27wbzZfy3hpBIMLAM6b92cj+Ucm84vve9E JGDrF3xw8168b1zpSCOb+3XG6IaUS5WH8EcTdSXvaMI+HqGDrgOTFzyY44pCdJXsVWPy Abhxt0sph1IhgZKsf0yGBYVU1YanN+uwWrVVcerTF2JNUUZeI1tgVhHvRMLiWqh/XJV4 SH6Cg0MltifJ4sipqFQERwK0stPA+gjhekPD8o/YkQ4zEq+CekS+LuWJPmcugCICgaCf 0cbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5GU2NjmgE9pvJfcbP7Wbuf1L4Bg6ALcOGzwayLNEdCc=; b=XwxeLy+dgJA2X2lzNNgL3VxrdlrkQQYAM4Loe5k+TTJuvM4YE6+VbsclWy2aaksRYb yIycJKFNb7UlhRn5byZ4vpUTFwDht7HqH26X+3/fVwD5tKCp9Jvy+ugkN+Xzc2lGeIAZ /7fQK3eCODuvnGnehuGhk5rypj56+f4qiyhxS62pkwkVKD6AVChL23K2znGb/K/Asmil mM9j2i4R/YqVSxDLsXFpLCmldtZmlkGbDSunGwLwKKh73AcfNSTYSbXAczTxAbK8JUV5 nz/H1Id8LT8LIAL7g5hsv6DwK8hXGrN+ulVFkz4jTwboLNbWpzrKVhRbmfJOKpATQOqG jF8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=M96PQCEK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a23-20020a05640213d700b00458b42ff246si6631578edx.236.2022.12.19.07.16.04; Mon, 19 Dec 2022 07:16:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=M96PQCEK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232883AbiLSPPV (ORCPT + 99 others); Mon, 19 Dec 2022 10:15:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232833AbiLSPPP (ORCPT ); Mon, 19 Dec 2022 10:15:15 -0500 Received: from mail-lf1-x12c.google.com (mail-lf1-x12c.google.com [IPv6:2a00:1450:4864:20::12c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0C04E0C3 for ; Mon, 19 Dec 2022 07:15:13 -0800 (PST) Received: by mail-lf1-x12c.google.com with SMTP id o6so9234841lfi.5 for ; Mon, 19 Dec 2022 07:15:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5GU2NjmgE9pvJfcbP7Wbuf1L4Bg6ALcOGzwayLNEdCc=; b=M96PQCEKu708zYSAKfnI3vYYuOYQPIDYBpF3ko4XOFDTuP8CChmqa/eBMxnob9Xvvu 13bJnQo4DxOWhMM/jR1CrFAF5M9LqPtdN/7YNIKVERNdhI1SDlDl3cc86hykawp9nDlW A8LaZVRmgg3mME0zszFlmghrDv2CYA9UQGulYue18w+tfHQ41K2i09gLezGgefdt2Xat yA3iZWLVPKyr3leBBG1RmA/uIihnVnrApGFWHFy9C1HuHrxB4Ntk1TXktxvLrCwRmM4l HKEKzORBxoGPpeRsegGr5/AeKG3CT40OE9h/5COkm0Ke5wzV7SIHlnUWUrpTu1NnSrTp 0XRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5GU2NjmgE9pvJfcbP7Wbuf1L4Bg6ALcOGzwayLNEdCc=; b=SXgwbuPeZPBfZUHxqXd1xTAE3622lGTtwLw8Ryb/RaujtXPFxM8qlRVJxsJ8/Kdg2g sVy4tmUJk2uDTFTayYhq1QbS1V+zXT2U7VX+nk4bR3ruTWqhkNNteTvv82qNBm96Vu4q 27yXBi4CL3dfFTMlVO1nRcdEklmBImIOqYAVeziDqUp+aS3IaVee5jpGxjbHpQgNA/kq Vsiauqh1vhCsAMEuyVNrOXW18DumSyr3zzTuNTPE9RvN1vMvYFUFRnBmB1F7IVrLB+Ew 5EQJf/JWNF1fAXxilfDiJRqsj7tMp3EX2lC/LEoelBOBMnKO1K7sB58jVcbBXmV6seGG 9tXA== X-Gm-Message-State: ANoB5pnlFPvAUQ/B6raEspDhzCdP2E7ZZvDtMva3TKsCux8YqHQXxqfv tix/1R2wtGemAZmfUAE42+dsVw== X-Received: by 2002:ac2:560a:0:b0:4b6:f0ea:4f49 with SMTP id v10-20020ac2560a000000b004b6f0ea4f49mr7591911lfd.9.1671462912024; Mon, 19 Dec 2022 07:15:12 -0800 (PST) Received: from krzk-bin.NAT.warszawa.vectranet.pl (088156142067.dynamic-2-waw-k-3-2-0.vectranet.pl. [88.156.142.67]) by smtp.gmail.com with ESMTPSA id e1-20020a05651236c100b004bd8534ebbcsm1109894lfs.37.2022.12.19.07.15.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Dec 2022 07:15:11 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Len Brown , Pavel Machek , Greg Kroah-Hartman , Kevin Hilman , Ulf Hansson , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v2 1/5] PM: domains: Add GENPD_FLAG_RT_SAFE for PREEMPT_RT Date: Mon, 19 Dec 2022 16:14:59 +0100 Message-Id: <20221219151503.385816-2-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> References: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752655974882048629?= X-GMAIL-MSGID: =?utf-8?q?1752655974882048629?= Realtime kernels with PREEMPT_RT must use raw_spinlock_t for domains which are invoked from CPU idle (thus from atomic section). Example is cpuidle PSCI domain driver which itself is PREEMPT_RT safe, but is being called as part of cpuidle. Add a flag allowing a power domain provider to indicate it is RT safe. The flag is supposed to be used with existing GENPD_FLAG_IRQ_SAFE. Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- Independently from Adrien, I encountered the same problem around genpd when using PREEMPT_RT kernel. Previous patch by Adrien: https://lore.kernel.org/all/20220615203605.1068453-1-athierry@redhat.com/ --- drivers/base/power/domain.c | 59 +++++++++++++++++++++++++++++++++++-- include/linux/pm_domain.h | 13 ++++++++ 2 files changed, 70 insertions(+), 2 deletions(-) diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c index 967bcf9d415e..4dfce1d476f4 100644 --- a/drivers/base/power/domain.c +++ b/drivers/base/power/domain.c @@ -119,6 +119,48 @@ static const struct genpd_lock_ops genpd_spin_ops = { .unlock = genpd_unlock_spin, }; +static void genpd_lock_rawspin(struct generic_pm_domain *genpd) + __acquires(&genpd->rslock) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&genpd->rslock, flags); + genpd->rlock_flags = flags; +} + +static void genpd_lock_nested_rawspin(struct generic_pm_domain *genpd, + int depth) + __acquires(&genpd->rslock) +{ + unsigned long flags; + + raw_spin_lock_irqsave_nested(&genpd->rslock, flags, depth); + genpd->rlock_flags = flags; +} + +static int genpd_lock_interruptible_rawspin(struct generic_pm_domain *genpd) + __acquires(&genpd->rslock) +{ + unsigned long flags; + + raw_spin_lock_irqsave(&genpd->rslock, flags); + genpd->rlock_flags = flags; + return 0; +} + +static void genpd_unlock_rawspin(struct generic_pm_domain *genpd) + __releases(&genpd->rslock) +{ + raw_spin_unlock_irqrestore(&genpd->rslock, genpd->rlock_flags); +} + +static const struct genpd_lock_ops genpd_rawspin_ops = { + .lock = genpd_lock_rawspin, + .lock_nested = genpd_lock_nested_rawspin, + .lock_interruptible = genpd_lock_interruptible_rawspin, + .unlock = genpd_unlock_rawspin, +}; + #define genpd_lock(p) p->lock_ops->lock(p) #define genpd_lock_nested(p, d) p->lock_ops->lock_nested(p, d) #define genpd_lock_interruptible(p) p->lock_ops->lock_interruptible(p) @@ -126,6 +168,8 @@ static const struct genpd_lock_ops genpd_spin_ops = { #define genpd_status_on(genpd) (genpd->status == GENPD_STATE_ON) #define genpd_is_irq_safe(genpd) (genpd->flags & GENPD_FLAG_IRQ_SAFE) +#define genpd_is_rt_safe(genpd) (genpd_is_irq_safe(genpd) && \ + (genpd->flags & GENPD_FLAG_RT_SAFE)) #define genpd_is_always_on(genpd) (genpd->flags & GENPD_FLAG_ALWAYS_ON) #define genpd_is_active_wakeup(genpd) (genpd->flags & GENPD_FLAG_ACTIVE_WAKEUP) #define genpd_is_cpu_domain(genpd) (genpd->flags & GENPD_FLAG_CPU_DOMAIN) @@ -1838,6 +1882,12 @@ static int genpd_add_subdomain(struct generic_pm_domain *genpd, return -EINVAL; } + if (!genpd_is_rt_safe(genpd) && genpd_is_rt_safe(subdomain)) { + WARN(1, "Parent %s of subdomain %s must be RT safe\n", + genpd->name, subdomain->name); + return -EINVAL; + } + link = kzalloc(sizeof(*link), GFP_KERNEL); if (!link) return -ENOMEM; @@ -2008,8 +2058,13 @@ static void genpd_free_data(struct generic_pm_domain *genpd) static void genpd_lock_init(struct generic_pm_domain *genpd) { if (genpd->flags & GENPD_FLAG_IRQ_SAFE) { - spin_lock_init(&genpd->slock); - genpd->lock_ops = &genpd_spin_ops; + if (genpd->flags & GENPD_FLAG_RT_SAFE) { + raw_spin_lock_init(&genpd->rslock); + genpd->lock_ops = &genpd_rawspin_ops; + } else { + spin_lock_init(&genpd->slock); + genpd->lock_ops = &genpd_spin_ops; + } } else { mutex_init(&genpd->mlock); genpd->lock_ops = &genpd_mtx_ops; diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h index 1cd41bdf73cf..0a1600244963 100644 --- a/include/linux/pm_domain.h +++ b/include/linux/pm_domain.h @@ -61,6 +61,14 @@ * GENPD_FLAG_MIN_RESIDENCY: Enable the genpd governor to consider its * components' next wakeup when determining the * optimal idle state. + * + * GENPD_FLAG_RT_SAFE: When used with GENPD_FLAG_IRQ_SAFE, this informs + * genpd that its backend callbacks, ->power_on|off(), + * do not use other spinlocks. They might use + * raw_spinlocks or other pre-emption-disable + * methods, all of which are PREEMPT_RT safe. Note + * that, a genpd having this flag set, requires its + * masterdomains to also have it set. */ #define GENPD_FLAG_PM_CLK (1U << 0) #define GENPD_FLAG_IRQ_SAFE (1U << 1) @@ -69,6 +77,7 @@ #define GENPD_FLAG_CPU_DOMAIN (1U << 4) #define GENPD_FLAG_RPM_ALWAYS_ON (1U << 5) #define GENPD_FLAG_MIN_RESIDENCY (1U << 6) +#define GENPD_FLAG_RT_SAFE (1U << 7) enum gpd_status { GENPD_STATE_ON = 0, /* PM domain is on */ @@ -164,6 +173,10 @@ struct generic_pm_domain { spinlock_t slock; unsigned long lock_flags; }; + struct { + raw_spinlock_t rslock; + unsigned long rlock_flags; + }; }; }; From patchwork Mon Dec 19 15:15:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Krzysztof Kozlowski X-Patchwork-Id: 34611 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2447801wrn; Mon, 19 Dec 2022 07:17:15 -0800 (PST) X-Google-Smtp-Source: AA0mqf7+o9iTw/cghry1WNftQ3U9Luz5ew2K6zhfaK0zMggOO66uvRoez1Z4Z/Xy4KlBQxgnz/5z X-Received: by 2002:a05:6402:540b:b0:461:e82b:be07 with SMTP id ev11-20020a056402540b00b00461e82bbe07mr37702847edb.2.1671463035298; Mon, 19 Dec 2022 07:17:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671463035; cv=none; d=google.com; s=arc-20160816; b=CLWbU0iKfc/Xc7PE0a3cbx9WxBZGoxbXXiIDbAajiEmxbkVea0x/sp7ylZhG0S5Zit CSj/swTSFWM3B9Sq3rjPR+B6Lw4IIPkTTrMNpfUFEjcg/VlNdHHjKh52Y1LiZjgV2iqM lsdylq7Z8BxdSSRqeCrqBqgblRoZQ2vj17Cxjh3d7hF3zixHU5YKsvyeSKv3n/mBtdrQ b5zonQlbJgM9fqZX2DbZmDfen633kGEHl9Wu2JuNw0ZS/JbWr7B63HkYyCrAIofSwqzt J0Unrb0DiL5W3EDboCHhnlkFhE2C5efK0XnOBdRx4nskqElMlryxRPVXbBDnICCAXBYi kK8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=710npLST/xPzuXiih2c1d9pFv9jXZIz92buQVuG0oAE=; b=hpjCM9Lgsugkn3j416UJbL2nZh26YLUPU8Gyb7Xcyl8M5JrA0RNxkQted6SQIjlWZt mmEjLdol0Zgw177iVcZ9S3YeVdnAcgDEqjlD9ooIln4xjZT8SDqX6lORdaBMOngsG+OC 3ANPyUfUWxSTdZR5lr1bbFjsOtXkQSJ8ygqZDura91XhnEHOCwq1gG4r/ni18LVjMUTz m737Cojw/a0z0uyiVgDkJVS8AO78ehPwEvC61N5/Fb0y31SJxj3i6wWo7ZtWUfa9rr2s tqm4zfXTiLGHIKdZ7BPMbpBBTtTW0TIXGeYXWi2J5Eyr4jFB+clHI+LSmZnNG3G7A/YA WR+w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uEio8Rqg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bx28-20020a0564020b5c00b00472be93436csi8159234edb.371.2022.12.19.07.16.51; Mon, 19 Dec 2022 07:17:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=uEio8Rqg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232604AbiLSPP2 (ORCPT + 99 others); Mon, 19 Dec 2022 10:15:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232850AbiLSPPQ (ORCPT ); Mon, 19 Dec 2022 10:15:16 -0500 Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C68ACB7DB for ; Mon, 19 Dec 2022 07:15:14 -0800 (PST) Received: by mail-lf1-x12a.google.com with SMTP id 1so14128767lfz.4 for ; Mon, 19 Dec 2022 07:15:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=710npLST/xPzuXiih2c1d9pFv9jXZIz92buQVuG0oAE=; b=uEio8RqgLj7kCKnMrT/ndnvx/qtape0C21ameHDn5oxR1frx1ZVQ/UjjKEP/ibeWWh /FrAeFueaLx7BNbF218RzyfLQhs6ts56jbLBL2inGVZ8II/hhK38SdmB46mvrjRjuhWs LneaquS3DyNt64GkklXVgbkS2zL9bf6bEp74uvqATEfY07UyjJTqmZM+QFcOLM4uy8Bo CE/mOMKX0Cwc47pRYdikRdnVWrHoU5LoezIzgPZcUuo1I6KJTFkMPX9rvq5ht+fmfcMC ONvGwjkQVW0VtSI8uJhE12tdg/7q+CM2fWOMU6PmvL3H+i1ZDUpFhpWuX439KLEKV4MS LZ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=710npLST/xPzuXiih2c1d9pFv9jXZIz92buQVuG0oAE=; b=kw+b73cq/61U1ewy4Yr7tJlGnvuEGbMm+xMxYTE1zoy060BFXfDjX3MNOoC5jMLDMf rNTIfyCku5Pk/bSwyq8r3WLNjcO6lAKcgYIigkpVZMCq2VEqSVJq7QlEiw/jHaOe1hU4 4IlJRLsFcgQuMhtLizrJ1G32qtz70NepR9e5z8OXz0gm4Rahh5WMHIi288oIY3+t8usi aQXAfH+uP1ym3ukIs02VCljT2Qf8mQnaYxIWppd5dp03MFQyMBKNWnflYueYPmKX1MbJ 5S37i78w02oge1cunMWQoVOj48auuT1nw+pY25XeTt/9ppCZ1OHiDGmTx4Kxc6cQdgw2 e2JQ== X-Gm-Message-State: ANoB5pmJT1frRtwImj3vpt525FB+lPliWseSzNatQMHvsaYABIcKOc39 FdwZXqqE4jc5ZAX0NJf1CyZg8w== X-Received: by 2002:a05:6512:261b:b0:4b4:dd3d:4310 with SMTP id bt27-20020a056512261b00b004b4dd3d4310mr16841834lfb.19.1671462913205; Mon, 19 Dec 2022 07:15:13 -0800 (PST) Received: from krzk-bin.NAT.warszawa.vectranet.pl (088156142067.dynamic-2-waw-k-3-2-0.vectranet.pl. [88.156.142.67]) by smtp.gmail.com with ESMTPSA id e1-20020a05651236c100b004bd8534ebbcsm1109894lfs.37.2022.12.19.07.15.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Dec 2022 07:15:12 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Len Brown , Pavel Machek , Greg Kroah-Hartman , Kevin Hilman , Ulf Hansson , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v2 2/5] cpuidle: psci: Mark as PREEMPT_RT safe Date: Mon, 19 Dec 2022 16:15:00 +0100 Message-Id: <20221219151503.385816-3-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> References: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752656023504134768?= X-GMAIL-MSGID: =?utf-8?q?1752656023504134768?= The PSCI cpuidle power domain in power_off callback uses __this_cpu_write() so it is PREEMPT_RT safe. This allows to use it in Realtime kernels and solves errors like: BUG: scheduling while atomic: swapper/2/0/0x00000002 Hardware name: Qualcomm Technologies, Inc. Robotics RB5 (DT) Call trace: dump_backtrace.part.0+0xe0/0xf0 show_stack+0x18/0x40 dump_stack_lvl+0x68/0x84 dump_stack+0x18/0x34 __schedule_bug+0x60/0x80 __schedule+0x628/0x800 schedule_rtlock+0x28/0x5c rtlock_slowlock_locked+0x360/0xd30 rt_spin_lock+0x88/0xb0 genpd_lock_nested_spin+0x1c/0x30 genpd_power_off.part.0.isra.0+0x20c/0x2a0 genpd_runtime_suspend+0x150/0x2bc __rpm_callback+0x48/0x170 rpm_callback+0x6c/0x7c rpm_suspend+0x108/0x660 __pm_runtime_suspend+0x4c/0x8c __psci_enter_domain_idle_state.constprop.0+0x54/0xe0 psci_enter_domain_idle_state+0x18/0x2c cpuidle_enter_state+0x8c/0x4e0 cpuidle_enter+0x38/0x50 do_idle+0x248/0x2f0 cpu_startup_entry+0x24/0x30 secondary_start_kernel+0x130/0x154 __secondary_switched+0xb0/0xb4 Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- drivers/cpuidle/cpuidle-psci-domain.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/cpuidle/cpuidle-psci-domain.c b/drivers/cpuidle/cpuidle-psci-domain.c index c80cf9ddabd8..d15a91fb7048 100644 --- a/drivers/cpuidle/cpuidle-psci-domain.c +++ b/drivers/cpuidle/cpuidle-psci-domain.c @@ -62,7 +62,8 @@ static int psci_pd_init(struct device_node *np, bool use_osi) if (!pd_provider) goto free_pd; - pd->flags |= GENPD_FLAG_IRQ_SAFE | GENPD_FLAG_CPU_DOMAIN; + pd->flags |= GENPD_FLAG_IRQ_SAFE | GENPD_FLAG_RT_SAFE | \ + GENPD_FLAG_CPU_DOMAIN; /* Allow power off when OSI has been successfully enabled. */ if (use_osi) From patchwork Mon Dec 19 15:15:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Krzysztof Kozlowski X-Patchwork-Id: 34613 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2448614wrn; Mon, 19 Dec 2022 07:18:37 -0800 (PST) X-Google-Smtp-Source: AMrXdXvnqhpwFz9qOULpqz83ESQoS55V23PPu5hWsJb12CcsP1XfZKv7BUWC+OQuoW6eyvjyoQXj X-Received: by 2002:a05:6402:1944:b0:475:7dbf:7040 with SMTP id f4-20020a056402194400b004757dbf7040mr13010623edz.42.1671463117091; Mon, 19 Dec 2022 07:18:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671463117; cv=none; d=google.com; s=arc-20160816; b=uciUlG3KmyITM4cZqcOiA7sjpu8BKBovZOuR7YeW86N9gwe+6do38SWd9sb3DRsBYh LaJJ1DX5hvwUX9QVEUOnUSXVi4VkzI1hEq5XCejiEcUCFYegmRssNFSPyTob3q6U01m9 pwPEXIdV6WZ0CwfzuoNkkLhH+Nb+sfW5Qi9y0+T2fF5HNhV/4xef9dB7ODVaVHMqbNYq OicvOlwEf/q8ZV/KAUbBsf7pQ6obWZGnValryLGi8Yfyk8XgYsdHxwsab0Va6FyiR2H7 tQB5vu6DS7kAGvlBcnI2bHFl8Zc2Gj4JMb1H7zlGlBMzrueYxnQfah8mCdWkAhSaXhTD uIfg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NLwTozcUqVuaex+an++z/NMr4xtzIfovpnza53ZqTDE=; b=Q+UsRnBhZdFWeoCTpEw9Dko70uZOLm3JIeXQkYBhfq2NGaML9bWSkDVCJPk6o4Qxv3 I2R58I3tj5Fgcg5WSv5UpKvSuLAKUtM9H4OL4iLckmcXGnNMlzHVnMVXyASQd1KHGYTA 0TGWW1mz33LEt4Dg2Wa4tXHofPawtVh6pCujge/DTXdSDCd5RxSkpz/BvzUnDboVooRV VIqRrW55SoccuAwUWhQGVAsPyV0BReMm8uUCpokxbZqPWU50mWflqVRbKc9QlTzMezlX NH8gemTplGdnJKLk3M+bR4wbSBtDCaM42PCzRreyovJCGgdMDqj4kZYvP5/zKyaXYk+0 irFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ph1MMT+V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n13-20020a05640205cd00b0047b5ecde3d7si1140593edx.257.2022.12.19.07.18.12; Mon, 19 Dec 2022 07:18:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=Ph1MMT+V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232924AbiLSPPc (ORCPT + 99 others); Mon, 19 Dec 2022 10:15:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232837AbiLSPPR (ORCPT ); Mon, 19 Dec 2022 10:15:17 -0500 Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F28F110540 for ; Mon, 19 Dec 2022 07:15:15 -0800 (PST) Received: by mail-lf1-x133.google.com with SMTP id bf43so14113494lfb.6 for ; Mon, 19 Dec 2022 07:15:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NLwTozcUqVuaex+an++z/NMr4xtzIfovpnza53ZqTDE=; b=Ph1MMT+VKo3WhiyGRG1FZKkXojebF95OOhoxbbAdyKuxvtcuJHNFcTz+GQryrGBEJH JgoEcPjmfRrTHRZ6gQrzc0a97FezxX/lSEA5WkJkWTxfOmkD7BWFipsPNylStoKPLclU 4juNW9JRwUbpvvXYTgXaUp4kKivBGFRoR05rQgeYzBZ5+4gdktxjA3Mp7NsfrUCuZiAn 271GvZZQJo0W+1njeKjlKWlG7nJIT4KFHgVz9W1wNIB5EsXZdM/BJwAYkkXEkioVughp VFbbXbXy939GC7TRbJxNCrsD7XBrkTuQQLbJyg0zNVx+iQq/cWbFGjEYLrgt05etQPnn gOkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NLwTozcUqVuaex+an++z/NMr4xtzIfovpnza53ZqTDE=; b=xNy30C73Rux6pUa5LkOFj9nR1Ccog7udwxmoZ46x+GgoC47gMetqPWjHaBP+5F57wY af+MW4tNnKRvz0+9qbHnh8FZZu6RDap/xglrY7opoSdAwh9A+Ky4uMgGju96jNyY5RbO xYtwa6DPmxWVaYfm+dvO7A2U71tzvBbJHBKAtVii133mB0x4pP6xWW9otwlHWT+7+B1M pmhXoCOMSPeECvtA7PvmqKKWvaR/TRhjywO8RSdxFMPL3upqJe6zNiVS5UpsOooR077b ZJ80eUuPTt5qm4RW4fsjFTXMfAuYtoUmIzjlLBEGj0BRollsDi3NjLIkqrk01eiSR2+h +B9g== X-Gm-Message-State: AFqh2koAwnnPvh7n9yni0CZYhm03KAnU1f1V57qDyt+zZwOA5rLpVZL1 gflz/vPnGxF990CIWNdHtrSg4TmxVSEc8o5k X-Received: by 2002:a05:6512:695:b0:4c3:3a78:eab6 with SMTP id t21-20020a056512069500b004c33a78eab6mr4004090lfe.65.1671462914357; Mon, 19 Dec 2022 07:15:14 -0800 (PST) Received: from krzk-bin.NAT.warszawa.vectranet.pl (088156142067.dynamic-2-waw-k-3-2-0.vectranet.pl. [88.156.142.67]) by smtp.gmail.com with ESMTPSA id e1-20020a05651236c100b004bd8534ebbcsm1109894lfs.37.2022.12.19.07.15.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Dec 2022 07:15:14 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Len Brown , Pavel Machek , Greg Kroah-Hartman , Kevin Hilman , Ulf Hansson , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v2 3/5] cpuidle: psci: Do not suspend topology CPUs on PREEMPT_RT Date: Mon, 19 Dec 2022 16:15:01 +0100 Message-Id: <20221219151503.385816-4-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> References: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752656109308285796?= X-GMAIL-MSGID: =?utf-8?q?1752656109308285796?= Realtime kernels are not supposed to go into deep sleep modes (neither system nor CPUs) because the latencies might become unpredictable. Therefore actual power suspending of CPU topology is not that important. On the other hand, this runtime PM of CPU topology is not compatible with PREEMPT_RT: 1. Core cpuidle path disables IRQs. 2. Core cpuidle calls cpuidle-psci. 3. cpuidle-psci in __psci_enter_domain_idle_state() calls pm_runtime_put_sync_suspend() and pm_runtime_get_sync() which use spinlocks (which are sleeping on PREEMPT_RT): BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:46 in_atomic(): 1, irqs_disabled(): 128, non_block: 0, pid: 0, name: swapper/0 preempt_count: 1, expected: 0 RCU nest depth: 0, expected: 0 1 lock held by swapper/0/0: #0: ffff20524613c1a0 (&dev->power.lock){+.+.}-{3:3}, at: __pm_runtime_suspend+0x30/0xac irq event stamp: 18776 hardirqs last enabled at (18775): [] tick_nohz_idle_enter+0x7c/0x17c hardirqs last disabled at (18776): [] do_idle+0xe0/0x300 softirqs last enabled at (4310): [] __local_bh_enable_ip+0x98/0x280 softirqs last disabled at (4288): [] cgroup_idr_alloc.constprop.0+0x0/0xd0 Preemption disabled at: [] schedule_preempt_disabled+0x20/0x30 CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.1.0-rt5-00322-gb49b67f1d8dc #1 Hardware name: Qualcomm Technologies, Inc. Robotics RB5 (DT) Call trace: dump_backtrace.part.0+0xe0/0xf0 show_stack+0x18/0x40 dump_stack_lvl+0x8c/0xb8 dump_stack+0x18/0x34 __might_resched+0x17c/0x214 rt_spin_lock+0x5c/0x100 __pm_runtime_suspend+0x30/0xac __psci_enter_domain_idle_state.constprop.0+0x60/0x104 psci_enter_domain_idle_state+0x18/0x2c cpuidle_enter_state+0x220/0x37c cpuidle_enter+0x38/0x50 do_idle+0x258/0x300 cpu_startup_entry+0x28/0x30 rest_init+0x104/0x180 arch_post_acpi_subsys_init+0x0/0x18 start_kernel+0x6fc/0x73c __primary_switched+0xbc/0xc4 Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- drivers/cpuidle/cpuidle-psci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c index 57bc3e3ae391..9d971cc4b12b 100644 --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -72,7 +72,7 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev, ct_irq_enter_irqson(); if (s2idle) dev_pm_genpd_suspend(pd_dev); - else + else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) pm_runtime_put_sync_suspend(pd_dev); ct_irq_exit_irqson(); @@ -85,7 +85,7 @@ static int __psci_enter_domain_idle_state(struct cpuidle_device *dev, ct_irq_enter_irqson(); if (s2idle) dev_pm_genpd_resume(pd_dev); - else + else if (!IS_ENABLED(CONFIG_PREEMPT_RT)) pm_runtime_get_sync(pd_dev); ct_irq_exit_irqson(); From patchwork Mon Dec 19 15:15:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Krzysztof Kozlowski X-Patchwork-Id: 34610 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2447709wrn; Mon, 19 Dec 2022 07:17:07 -0800 (PST) X-Google-Smtp-Source: AMrXdXs9qXKgJw3Z9eqw0B0sIZF6llh+SPfapETGwt26d0vvLbuhS0/KeMmbD8Mxo9CXWbZc8VGR X-Received: by 2002:a05:6402:3811:b0:472:1b3b:35f1 with SMTP id es17-20020a056402381100b004721b3b35f1mr31485002edb.21.1671463026962; Mon, 19 Dec 2022 07:17:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671463026; cv=none; d=google.com; s=arc-20160816; b=MHe8r0coDiCCyFq6ouyv9sFqjBSup8KdjHWntwQmz1k/JvefWHaTOjmSazLm+iZn6Y P/wh7UiWmeMeZh1takopfSV18P/cTp+SSe8j5NVa4rBqR9Ge/gDRnXy9vA2LRGO4iwMH ZNCQq3aYSr6YP+V+ljfWx+RyF/RVHXnkh7Dq6kVUnZZBHWv55TCriWDAxoMh/Kufgm94 EKEWbjjKQlP4oz0ziHvRyCCHD6k33CUK2zyI7eUD6oMGQAY6IpvUkONDD2Vte3l1h6D3 IG21gtHUCxayoNunj760/7giYyHsWl3/sdknudRO5148i1uEWklhtoNXrAh3nnbuY+QY ys5Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=y2D0MZ2hxPTHwXNnho8HOkobrXZQciZPG6h42YTOjRw=; b=HnQLXd/l+mbgcUO5FqB8XqdqMmdJSX3ZdL4tzBqNmZZ5yvMEsjbsJw5jIGkl0bw3ZL JVM6Q5toT119TjRF5+YKUMIvzTlQLSokrf2j+j2ZYYTcGQEd4mmPHzAfQMIAJzwqKVQv GPjQ7BFk3XNFYd+Ht0DLnFF+kdnt+BR5BcYHB1sfkQ2/Zg+XXGbIYFsTZjzM3TTBHBL6 PgZABwKi5vw28+T6ji2M8tFCovsh286VhMu+B/9uauolkth0j4dQc+7/FxKjQvQLanKP lhLfMRckW2r8dsppOlWvmKbezImqUFj3YUTFCHH4DK/68Nc9ILdHAZ++JD+YFu5NHU3P gazw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ucUJppIg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f13-20020a056402068d00b0047338d1bfcfsi8580948edy.166.2022.12.19.07.16.43; Mon, 19 Dec 2022 07:17:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=ucUJppIg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232904AbiLSPPh (ORCPT + 99 others); Mon, 19 Dec 2022 10:15:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232852AbiLSPPS (ORCPT ); Mon, 19 Dec 2022 10:15:18 -0500 Received: from mail-lf1-x12f.google.com (mail-lf1-x12f.google.com [IPv6:2a00:1450:4864:20::12f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7A201260E for ; Mon, 19 Dec 2022 07:15:17 -0800 (PST) Received: by mail-lf1-x12f.google.com with SMTP id y25so14088815lfa.9 for ; Mon, 19 Dec 2022 07:15:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y2D0MZ2hxPTHwXNnho8HOkobrXZQciZPG6h42YTOjRw=; b=ucUJppIgRBgtpDQOSNoFKkTpre/wj89djnkxbJXoXfrK0vOY/QJLX1iUpqT5cY5qV/ GF+Dl82w3xJ9mA3oFvYtRV0YhivF59Kq+GNvnC3Jqv4aY+ohv2cJJWZlvBeaEZEzK6ou HIlLEZWK5gd1rofobutljGLWfO3nULqGfRQ0ICl88S/eoknin4PISCvnL7lw7XSStvto AE4nkyesvYnJaizYJIHpktZOUcmnuiuncKvdsRPUYSWCKWWwaxvzUs4Jljp2D+ucASqX xQDAJUDjIsYIBgPxGqZ7U5LO8uF94J/koNWBdV2fQu+agiJ3nra/ype6SXUSompNmfk6 bBwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y2D0MZ2hxPTHwXNnho8HOkobrXZQciZPG6h42YTOjRw=; b=44PTQrHESGhjLXFf6gc/oUz7pFnKii4wGemyO16+4Imi25F+hszxmtQW/iNJhKqPxn jRDEjwYb0Ot+2whoFeiF6Gki7qOp8ddi68vnXcL2jjj6IncxWNa7dOugpjIHa+dQDQtF S/qwuCNC2HsdlPXI16aM4qH+yOjcaNA8K3o6i5c8VUXHvgUIGJakuTeCdAU4uFbSZTBz 4XcCq+WtMwcdD4jJd2XLxUlpuJX1/l13/Lu9X5rjHjDETwqte08CAbZT2sJ8RHuEI4vQ kHBiaaLDLxixaQqc6D5fT6rAguiIIhNY9D7lmxWxMcqDbCRPeMzh23inE8kWoGpwqDpV PV5g== X-Gm-Message-State: ANoB5plWfcDLIBcDhQJXlzBm+yhe4dTBYcAqqfOnosL9De3hoYrc9DLe CTUpsfDHiCX8jJBVndgulEM5Tw== X-Received: by 2002:a05:6512:a94:b0:4b7:113:9296 with SMTP id m20-20020a0565120a9400b004b701139296mr14233163lfu.14.1671462915529; Mon, 19 Dec 2022 07:15:15 -0800 (PST) Received: from krzk-bin.NAT.warszawa.vectranet.pl (088156142067.dynamic-2-waw-k-3-2-0.vectranet.pl. [88.156.142.67]) by smtp.gmail.com with ESMTPSA id e1-20020a05651236c100b004bd8534ebbcsm1109894lfs.37.2022.12.19.07.15.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Dec 2022 07:15:15 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Len Brown , Pavel Machek , Greg Kroah-Hartman , Kevin Hilman , Ulf Hansson , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v2 4/5] PM: Allow calling dev_pm_domain_set() with raw spinlock Date: Mon, 19 Dec 2022 16:15:02 +0100 Message-Id: <20221219151503.385816-5-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> References: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752656015143504017?= X-GMAIL-MSGID: =?utf-8?q?1752656015143504017?= device_pm_check_callbacks() uses dev->power spinlock, which on PREEMPT_RT sleeps. However some PM domains on PREEMPT_RT might be using raw spinlocks as genpd_lock(), thus dev_pm_domain_set() must not call device_pm_check_callbacks(). In fact device_pm_check_callbacks() is not strictly related to dev_pm_domain_set() and calls for these two can be made separately. Add new helper dev_pm_domain_set_no_cb() which will only set PM domain but will not check the callbacks, leaving the checl to the caller. Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- drivers/base/power/common.c | 27 +++++++++++++++++++++++++-- include/linux/pm_domain.h | 3 +++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/drivers/base/power/common.c b/drivers/base/power/common.c index 72115917e0bd..f81cab6990ad 100644 --- a/drivers/base/power/common.c +++ b/drivers/base/power/common.c @@ -218,6 +218,30 @@ EXPORT_SYMBOL_GPL(dev_pm_domain_start); * This function must be called with the device lock held. */ void dev_pm_domain_set(struct device *dev, struct dev_pm_domain *pd) +{ + if (dev->pm_domain == pd) + return; + + dev_pm_domain_set_no_cb(dev, pd); + device_pm_check_callbacks(dev); +} +EXPORT_SYMBOL_GPL(dev_pm_domain_set); + +/** + * dev_pm_domain_set_no_cb - Set PM domain of a device. + * @dev: Device whose PM domain is to be set. + * @pd: PM domain to be set, or NULL. + * + * Sets the PM domain the device belongs to. The PM domain of a device needs + * to be set before its probe finishes (it's bound to a driver). + * + * This is exactly like dev_pm_domain_set(), however device_pm_check_callbacks() + * is not called and the caller is responsible to invoke + * device_pm_check_callbacks() with device lock held. + * + * This function must be called with the device lock held. + */ +void dev_pm_domain_set_no_cb(struct device *dev, struct dev_pm_domain *pd) { if (dev->pm_domain == pd) return; @@ -225,6 +249,5 @@ void dev_pm_domain_set(struct device *dev, struct dev_pm_domain *pd) WARN(pd && device_is_bound(dev), "PM domains can only be changed for unbound devices\n"); dev->pm_domain = pd; - device_pm_check_callbacks(dev); } -EXPORT_SYMBOL_GPL(dev_pm_domain_set); +EXPORT_SYMBOL_GPL(dev_pm_domain_set_no_cb); diff --git a/include/linux/pm_domain.h b/include/linux/pm_domain.h index 0a1600244963..352d0c76bfec 100644 --- a/include/linux/pm_domain.h +++ b/include/linux/pm_domain.h @@ -438,6 +438,7 @@ struct device *dev_pm_domain_attach_by_name(struct device *dev, void dev_pm_domain_detach(struct device *dev, bool power_off); int dev_pm_domain_start(struct device *dev); void dev_pm_domain_set(struct device *dev, struct dev_pm_domain *pd); +void dev_pm_domain_set_no_cb(struct device *dev, struct dev_pm_domain *pd); #else static inline int dev_pm_domain_attach(struct device *dev, bool power_on) { @@ -460,6 +461,8 @@ static inline int dev_pm_domain_start(struct device *dev) } static inline void dev_pm_domain_set(struct device *dev, struct dev_pm_domain *pd) {} +static inline void dev_pm_domain_set_no_cb(struct device *dev, + struct dev_pm_domain *pd) {} #endif #endif /* _LINUX_PM_DOMAIN_H */ From patchwork Mon Dec 19 15:15:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Krzysztof Kozlowski X-Patchwork-Id: 34612 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2448528wrn; Mon, 19 Dec 2022 07:18:27 -0800 (PST) X-Google-Smtp-Source: AA0mqf5gVLy9XzCEQKBDIew0zJu7Ywnrx/sxXl3VzAbutSMAm2yILchnSKXq310sojlN+dMZouyO X-Received: by 2002:a17:906:a050:b0:7a4:bbce:ddae with SMTP id bg16-20020a170906a05000b007a4bbceddaemr36842543ejb.59.1671463106866; Mon, 19 Dec 2022 07:18:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671463106; cv=none; d=google.com; s=arc-20160816; b=RryyQ4DMEfhmsS9JVIu4dPNBIsaWeaL/OiBvaybpXYUdJE21WcCDoRlUCMW9MC/5J5 csLXPf77rRLC2zHhhp74Vy2lzWLzQtbXuzLPdj7xHYjX9hmH5ZnI8NcoTxIP7OJVefux EwfffpIoUBXlo06gDKUe90RPSxQndFu4bjn6+FYLf6igdygopn0F5pgZ643aZf9F7xud d/9+trbss4/F5L3bJZ/IrlMH7GAjF6fM4csOxCnJawJMUIzRCecgLrdxMujQ3km6oVkB 7nbWKW/4ZLBNC4ILk81r5OFaoL3pXW4WL6x0sjyrJGpqolJG99X/c1Hu5sK9k1N2MLPL /eOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=m8Uj/f0x4H68pfp6EBuS49q8mUm/dd1ntZTx2c5gQRw=; b=CIvIvqUBPoQ1DPyRKqY3+waFKNyclFamFLs5iWLmwYw+uISm5msZKHweUVr3XjnMCE /FyWjn4znE7D4CuV1LfvB8EJmI4qpjbWvKSDtBD/SpPwU330N07pVHnLpbS0MpEMS3hc O7tdr4yx/JJbUhZsKoc+EcP7z5/xEoFYAx+v/aF/UnaF+QdJ8m1zaFHoyNHcrgJtDTcB E14WS6OAQyxJIocE1/5bYMNGq8GQJc+BjcVmN4MyuCBkB4gafZWNm7zMtBIf+R7lzBTu AZJFxHfmUGgIIvlvI3gd96HykgZMgO1RuiV2yd4mZ+BFjzIzPxDULOKGe/QY8+R+3whP 9VbA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=joGdkJgx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ib6-20020a1709072c6600b007c144e9f270si8471785ejc.708.2022.12.19.07.18.03; Mon, 19 Dec 2022 07:18:26 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=joGdkJgx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232939AbiLSPPm (ORCPT + 99 others); Mon, 19 Dec 2022 10:15:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232876AbiLSPPT (ORCPT ); Mon, 19 Dec 2022 10:15:19 -0500 Received: from mail-lf1-x135.google.com (mail-lf1-x135.google.com [IPv6:2a00:1450:4864:20::135]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48CE7B7DB for ; Mon, 19 Dec 2022 07:15:18 -0800 (PST) Received: by mail-lf1-x135.google.com with SMTP id b3so14153283lfv.2 for ; Mon, 19 Dec 2022 07:15:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=m8Uj/f0x4H68pfp6EBuS49q8mUm/dd1ntZTx2c5gQRw=; b=joGdkJgxChVHrhqXxh4D8sJDaKpzxBgUcFgLtPczIcYGrPWnS0gYSfGGb/xwur1BZA HXUG1BhIHBctF7bFBpebP9+UCrd1/wX6sQ06q9+cIYwobrrc2Zzo3WsBKRD3558LWzFQ /N47j8fJwjN1I4eRjd0Qr07xS6EOzoIv8QhoPs4d+muMgyaWlNH3o7v1b1Xr6gxm8Bsn zsRaxMej9uybQ/82hjh0l+j0+akCnf8EYggcdPI5wQ4zTU9cGGyx+pH3DhxSdkmYX4n/ boQmP89KDmwLe8i18IoxJ121DOicW9mnYp04EIuXSuoqBPamMqwF9q353kDRDAlSKDOO ifvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=m8Uj/f0x4H68pfp6EBuS49q8mUm/dd1ntZTx2c5gQRw=; b=7x2rGYUa0cwuIR2NYaIF+nO0g3O9dwGM9vvON6YRkwljNesnlM3zKyYhibnBTVOr02 zrkyxSvNX0JHSzBuNGXaTn9bNLQIJdpQbI5CCx1TDHG57Ki2GZ314EH3qBXUbpH+jmu1 VPgGiAfuHtpPUTDFZggnetT0o93Rd7ideBlrEXVuRrNsni+x6tb3XLRGtw+vDDjIgBuD ljyKmR/G1yOAcS9FqRNmbwh8g3JATui4tYiHnCHOtH7WQNblFabfkAxA0iLY6wUmzDPa uhonWqleCL8Oy+kwQGGT7up2ShuXwuBQnjAXR4Rcf1n7d8wWmnAKTuxIrHaTNtvBdWFA rScA== X-Gm-Message-State: ANoB5plJzZfE/a20hAA0KJAEV60/z3Zua1hsogdavzO5PS/58OhDVkHM P7oS6gJKQHpYUgxGlkS7G3Lx0w== X-Received: by 2002:a05:6512:2c85:b0:4a4:68b9:607f with SMTP id dw5-20020a0565122c8500b004a468b9607fmr11616697lfb.10.1671462916643; Mon, 19 Dec 2022 07:15:16 -0800 (PST) Received: from krzk-bin.NAT.warszawa.vectranet.pl (088156142067.dynamic-2-waw-k-3-2-0.vectranet.pl. [88.156.142.67]) by smtp.gmail.com with ESMTPSA id e1-20020a05651236c100b004bd8534ebbcsm1109894lfs.37.2022.12.19.07.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Dec 2022 07:15:16 -0800 (PST) From: Krzysztof Kozlowski To: "Rafael J. Wysocki" , Len Brown , Pavel Machek , Greg Kroah-Hartman , Kevin Hilman , Ulf Hansson , Daniel Lezcano , Lorenzo Pieralisi , Sudeep Holla , linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Krzysztof Kozlowski , Adrien Thierry , Brian Masney , linux-rt-users@vger.kernel.org Subject: [PATCH v2 5/5] PM: domains: Do not call device_pm_check_callbacks() when holding genpd_lock() Date: Mon, 19 Dec 2022 16:15:03 +0100 Message-Id: <20221219151503.385816-6-krzysztof.kozlowski@linaro.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> References: <20221219151503.385816-1-krzysztof.kozlowski@linaro.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752656098669504499?= X-GMAIL-MSGID: =?utf-8?q?1752656098669504499?= If PM domain on PREEMPT_RT is marked as GENPD_FLAG_RT_SAFE(), the genpd_lock() will be a raw spin lock, thus device_pm_check_callbacks() must be called outside of the domain lock. This solves on PREEMPT_RT: [ BUG: Invalid wait context ] 6.1.0-rt5-00325-g8a5f56bcfcca #8 Tainted: G W ----------------------------- swapper/0/1 is trying to lock: ffff76e045dec9a0 (&dev->power.lock){+.+.}-{3:3}, at: device_pm_check_callbacks+0x20/0xf0 other info that might help us debug this: context-{5:5} 3 locks held by swapper/0/1: #0: ffff76e045deb8e8 (&dev->mutex){....}-{4:4}, at: __device_attach+0x38/0x1c0 #1: ffffa92b81f825e0 (gpd_list_lock){+.+.}-{4:4}, at: __genpd_dev_pm_attach+0x7c/0x250 #2: ffff76e04105c7a0 (&genpd->rslock){....}-{2:2}, at: genpd_lock_rawspin+0x1c/0x30 stack backtrace: CPU: 5 PID: 1 Comm: swapper/0 Tainted: G W 6.1.0-rt5-00325-g8a5f56bcfcca #8 Hardware name: Qualcomm Technologies, Inc. Robotics RB5 (DT) Call trace: dump_backtrace.part.0+0xe0/0xf0 show_stack+0x18/0x40 dump_stack_lvl+0x8c/0xb8 dump_stack+0x18/0x34 __lock_acquire+0x938/0x2100 lock_acquire.part.0+0x104/0x28c lock_acquire+0x68/0x84 rt_spin_lock+0x40/0x100 device_pm_check_callbacks+0x20/0xf0 dev_pm_domain_set+0x54/0x64 genpd_add_device+0x258/0x340 __genpd_dev_pm_attach+0xa8/0x250 genpd_dev_pm_attach_by_id+0xc4/0x190 genpd_dev_pm_attach_by_name+0x3c/0x60 dev_pm_domain_attach_by_name+0x20/0x30 dt_idle_attach_cpu+0x24/0x90 psci_cpuidle_probe+0x300/0x4b0 platform_probe+0x68/0xe0 really_probe+0xbc/0x2dc __driver_probe_device+0x78/0xe0 driver_probe_device+0x3c/0x160 __device_attach_driver+0xb8/0x140 bus_for_each_drv+0x78/0xd0 __device_attach+0xa8/0x1c0 device_initial_probe+0x14/0x20 bus_probe_device+0x9c/0xa4 device_add+0x3b4/0x8dc platform_device_add+0x114/0x234 platform_device_register_full+0x108/0x1a4 psci_idle_init+0x6c/0xb0 do_one_initcall+0x74/0x450 kernel_init_freeable+0x2e0/0x350 kernel_init+0x24/0x130 ret_from_fork+0x10/0x20 Cc: Adrien Thierry Cc: Brian Masney Cc: linux-rt-users@vger.kernel.org Signed-off-by: Krzysztof Kozlowski --- drivers/base/power/domain.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/base/power/domain.c b/drivers/base/power/domain.c index 4dfce1d476f4..db499ba40497 100644 --- a/drivers/base/power/domain.c +++ b/drivers/base/power/domain.c @@ -1666,10 +1666,14 @@ static int genpd_add_device(struct generic_pm_domain *genpd, struct device *dev, if (ret) goto out; + + /* PREEMPT_RT: Must be outside of genpd_lock */ + device_pm_check_callbacks(dev); + genpd_lock(genpd); genpd_set_cpumask(genpd, gpd_data->cpu); - dev_pm_domain_set(dev, &genpd->domain); + dev_pm_domain_set_no_cb(dev, &genpd->domain); genpd->device_count++; if (gd)