From patchwork Tue Aug 22 18:01:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rob Clark X-Patchwork-Id: 136582 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b82d:0:b0:3f2:4152:657d with SMTP id z13csp3798274vqi; Tue, 22 Aug 2023 11:03:59 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGiGFixQMb32de+jWGKVdhwArXxD9rxnRL6xaUWPmHGzghx2T/S5601AvI3nO/wdjweOJo8 X-Received: by 2002:a17:906:10c:b0:99b:eca2:47a8 with SMTP id 12-20020a170906010c00b0099beca247a8mr9975493eje.38.1692727438958; Tue, 22 Aug 2023 11:03:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692727438; cv=none; d=google.com; s=arc-20160816; b=b9jX5l0ZMwZEW170TTx9daqLu9ADOzcMeCKA/O5YAG2xRHEVrdqSOUR3JZkQVFeoFl +DlUPPupLRNY80LkHC4IdHTnxFFd2GH8fzRucXejssmGPtVt8drFj976ci+d47acHUGf C8lLHjqSz3+5ZRkKaA/XCXdBmN7sFxQw3jdzzcRK9Gcp3aS1aNQX/ZzgBN1jbxcuUaFw Ud/hzASA6vZ+N2ZM8oRWRwespR6EB9jXNQtGzgoRhUGva+eQuVY+P1fglY7SuxG74qh7 g9k5z858608f2L6/nOd0o9BkjzWyYgYkYYpjAJiH8jEXTJKnlIFpGTe2LId/SXp1yACA //2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WHzqSThmy913fuGKpn4uXf/XGbug/XgU3SVaHteOTFw=; fh=rfUYmKK1dhOd4arqnW6KpdJILWMd2Xj1MpRfQMmcO1s=; b=ojNBDW/0kSEtMejVE2H0rWri8qJyrMSAKofB6iThKLZr5qlcbHdl2jrJsWPkdWGfFE 0eMo7Jjo1t8vXh2kO5oqiNTkUtdKMM8aCWQr8CEa1/s6rlDg8eTuufvR4cv8Fwrpwu41 NWyg3vMJ/EfP75ZbbSPm385w3o+uvuTz1xVsql4oYS7Ry50xUTx2NzEQ3tsr+qzxGUtX ytlJpp26FCqnF7Ija5UGVnmfPuJtxbL7MfOOwUbyfoMczHv2DuB9WSRYKaNGH0FNaFll f4yaXjSnnuAJBx0jEaG9UjQS+zH49Me5nazCpN2sB7XQ6ugGCjrvK1naLQTZTOmLxDGT gzTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="WPZoI+C/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h3-20020a1709063b4300b0098d373fa9e4si7502774ejf.1007.2023.08.22.11.03.16; Tue, 22 Aug 2023 11:03:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="WPZoI+C/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229683AbjHVSCm (ORCPT + 99 others); Tue, 22 Aug 2023 14:02:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229658AbjHVSCg (ORCPT ); Tue, 22 Aug 2023 14:02:36 -0400 Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B9D0CD7; Tue, 22 Aug 2023 11:02:34 -0700 (PDT) Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1bc0d39b52cso31428515ad.2; Tue, 22 Aug 2023 11:02:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692727353; x=1693332153; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WHzqSThmy913fuGKpn4uXf/XGbug/XgU3SVaHteOTFw=; b=WPZoI+C/3/TAgQYvvcQN3m8uDX7PrfAY02kIgdL0Oz9khOGf+e8dzjXhTDYiVuc9Bk DUaUbyZ1NO3qataURGw8oCNtmXl5E9xzTCYEa8GS6krPVvd9IgMBN1dUteEHAWOV74pd Me89kMXmtLHqKx7okwKC3CWYKkUIA/gRf/diLBrvf7Q7AHME/4vDRy99D2eKMuA/r+bS 9HYEbFHW7hu/1+TiAvjKZOp+TUQPGZnMAgbSRGaT4HI59DfoGKIvmWuYUqZpMxJBoaii 6EY4fesEl1gdxAe7MZvh6USiVcO9QnZVbyZxsTMDg+DqOLSAeWfOTbp0atTPWyptegbg 5XVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692727353; x=1693332153; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WHzqSThmy913fuGKpn4uXf/XGbug/XgU3SVaHteOTFw=; b=EuGwcHLN+sillOtvhj9O1J028iPKCFSRW3Yayj5uKNdUnFjiBcn4Ojt0vy+lCP3TS1 FW03VOr9hqMJcEZXobzT/Wqfe91OhRLCc8c5q8rLVCfZjfTZbLksSgUed/swsRAaV3yH tw5hMYcmj/eTi1buO0IBvYtauw8byWSnYTvJF/ZfU4lbyMbVlwy+0pMtPjHcGn9I08BZ 8DiZOJPWEciiIreKTf0HZzqKK7kJON0L0vN1BgJb7uEVIs4vjGuQepiPnLR2usN2Wa7U cIMuobFXkFbaAnH0aObRcIxW+fA+IFsQ+PnHCL9JLxB2p3LfC5OkgsZCx6fOkKTXURtY gyVw== X-Gm-Message-State: AOJu0YxaHm2iRaAZ3MSg/SAbW7XwUvtyoJHBJ5Q5IJH3daYrY5jHjrfm bdZnyi+b2FjQEkicFKeXn6afaOlC+V8= X-Received: by 2002:a17:902:a414:b0:1bd:b8c8:98f8 with SMTP id p20-20020a170902a41400b001bdb8c898f8mr6778141plq.4.1692727353437; Tue, 22 Aug 2023 11:02:33 -0700 (PDT) Received: from localhost ([2a00:79e1:abd:4a00:6c80:7c10:75a0:44f4]) by smtp.gmail.com with ESMTPSA id t7-20020a170902e84700b001b898595be7sm9347196plg.291.2023.08.22.11.02.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 11:02:32 -0700 (PDT) From: Rob Clark To: dri-devel@lists.freedesktop.org Cc: linux-arm-msm@vger.kernel.org, freedreno@lists.freedesktop.org, Rob Clark , MyungJoo Ham , Kyungmin Park , Chanwoo Choi , linux-pm@vger.kernel.org (open list:DEVICE FREQUENCY (DEVFREQ)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH v5 02/11] PM / devfreq: Teach lockdep about locking order Date: Tue, 22 Aug 2023 11:01:49 -0700 Message-ID: <20230822180208.95556-3-robdclark@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230822180208.95556-1-robdclark@gmail.com> References: <20230822180208.95556-1-robdclark@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774953366988970798 X-GMAIL-MSGID: 1774953366988970798 From: Rob Clark This will make it easier to catch places doing allocations that can trigger reclaim under devfreq->lock. Because devfreq->lock is held over various devfreq_dev_profile callbacks, there might be some fallout if those callbacks do allocations that can trigger reclaim, but I've looked through the various callback implementations and don't see anything obvious. If it does trigger any lockdep splats, those should be fixed. Signed-off-by: Rob Clark --- drivers/devfreq/devfreq.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/devfreq/devfreq.c b/drivers/devfreq/devfreq.c index e5558ec68ce8..81add6064406 100644 --- a/drivers/devfreq/devfreq.c +++ b/drivers/devfreq/devfreq.c @@ -817,6 +817,12 @@ struct devfreq *devfreq_add_device(struct device *dev, } mutex_init(&devfreq->lock); + + /* Teach lockdep about lock ordering wrt. shrinker: */ + fs_reclaim_acquire(GFP_KERNEL); + might_lock(&devfreq->lock); + fs_reclaim_release(GFP_KERNEL); + devfreq->dev.parent = dev; devfreq->dev.class = devfreq_class; devfreq->dev.release = devfreq_dev_release; From patchwork Tue Aug 22 18:01:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rob Clark X-Patchwork-Id: 136604 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a924:0:b0:3f4:cb6f:45e2 with SMTP id c4csp3907693vqw; Tue, 22 Aug 2023 11:53:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHQJq/anT9Y8DbvEzjg7SnBonhsTdYqqHsn5Ic/pPE/TudVL78zqehNkBBQHFP7MYVg8K1Y X-Received: by 2002:aa7:c507:0:b0:526:e585:e25a with SMTP id o7-20020aa7c507000000b00526e585e25amr7367842edq.38.1692730418578; Tue, 22 Aug 2023 11:53:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692730418; cv=none; d=google.com; s=arc-20160816; b=G+u4XLwHRFcguKIl6dNEaLbX1RDJFR6RNPy3aLU33vy8jHWb4LNtkqgaapnPDKdHJS BKG+eSLRZJAcahoBNSqkMmIYUp7bXWfu/Q5AlMfidVecke/htcBYpoappzzeM1vSuLWR mpJfmPPgWFxVuez2KkVYe6rSbctGuXMsaoQnX8zVsTnhxD1YOhwDFhuALuoTh31YrbEC Ki1d9dp1io+VWpcyKAy8kdlAlFHc7Jw8mXPBX6L3vGkCnIuWBlVMaZqnefC++5NLraZr aAlKsHGQYKAHBNUgX9QfAsV+LEd1D72CdFWkGFJ3oOujaqHE+JoQz2fMEmbv4ZUH2cFG mLRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jwi88nXkWgcM9/OEm1nLYgH6sllE6KibtJsTy1TkfRY=; fh=PhOWso5T0i1U8K2ZXlBMmXjRpM1No+RaW/OZqtsRyl0=; b=AEQDwQ6PQZIUCES/WOzT9rRuPsanZt886TsyTIY8zJj1OQnMCdQ+x162y4ZKoBZhzV c66iQb222iImQYzKECDxTlWX4ZkG5dJKziBcvWpOqzPoCE6h4mQ9LShcotwKF86bqZk9 xSEL2cFGKHW0/DPxxG0DujNMcazLM5NsgsYNaf/7QqWKWGx8CMr2lVcnzGWpxlIcdOd6 JDLMkwu79VYRz2LO7Ako4ke+ESOj30j88alPcOQZ+fRBy4xj1wlqcwyNTgnJQnMJ6lYQ epLsuCIYkGHCg1T27Fz/7wdrTKfw/xp5WJXIGvMVwe9LHoMkof/7RcHuSied8sn/nqfj 5n0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=EG7ioNIU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u12-20020a50eacc000000b005233fd1111fsi8245558edp.398.2023.08.22.11.53.12; Tue, 22 Aug 2023 11:53:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=EG7ioNIU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229646AbjHVSCl (ORCPT + 99 others); Tue, 22 Aug 2023 14:02:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229693AbjHVSCi (ORCPT ); Tue, 22 Aug 2023 14:02:38 -0400 Received: from mail-pj1-x102b.google.com (mail-pj1-x102b.google.com [IPv6:2607:f8b0:4864:20::102b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6777F185; Tue, 22 Aug 2023 11:02:36 -0700 (PDT) Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-26d63b608d4so1902404a91.2; Tue, 22 Aug 2023 11:02:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692727356; x=1693332156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jwi88nXkWgcM9/OEm1nLYgH6sllE6KibtJsTy1TkfRY=; b=EG7ioNIUIceTPWyCUvSdDcnxfTj11NqrOW3xhYTfkYwJeISXyeQAe8qhaUeVGhVC33 YInKNYGuzwqKylb23Id2fHc1EPI5GkHMBGZVNiiRrOaFM2inyQVotYc88yB0+ap5oteZ ptoJaasGdzceRe9xgY0as2WIHBcbq5sHJ28cxQPpSUM9Nx94Lt86TyIO0a/iqQqKbsvV 0S1hll/g6wPR4QafMBuY3Zkluaf/CmLiCjfHMPMkiN5EgvL99GYELAUPgznmSs/62Vz7 MEXK1KDX83QEqQSsL5W/re7Dlon2s6xwTWtGdXR92ZOLcX2szBCkNrLxrjwcTiQca89R 4k1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692727356; x=1693332156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jwi88nXkWgcM9/OEm1nLYgH6sllE6KibtJsTy1TkfRY=; b=hzpDgQ+uzenzt5jZGYJMzrFcJP02EHS3D6xxQj5qNIaHaM/nevkX7Wd3/PAdmxLm2d sZ7f7ACUQRuDQgRsO6fu+H1SDB7RRxZ6kT6wYAqLGqU6tOgy+u73A2I4w+ny7wS43ygA ESjLm9dsPuIeIhYfAiEuv103uJU6A7ksBEyIZC/5+bwOwTeIbrljFYt57GsVQFq1zBMS IE8Z42AQbL7eS1y6WsWumt++igY87eWbPtpDZlQNxeiJokN4Wvokq6oPx5GUXSMQVQAG fIc/BnKuH7aL1eLxJbLY4Rsj7KdxxUc/vzTGa2fLKX2erTAsbGQKUe8PPuPVtMZcmJz7 HdJQ== X-Gm-Message-State: AOJu0Yy6hpH3e2pEhBYy8BVjNf4zKdQ0Lvk+K5wZozzdWrlGv7s0DTK6 AYRZJRnyhmtzpkAuXYAU+PM= X-Received: by 2002:a17:90b:4c81:b0:26b:219f:3399 with SMTP id my1-20020a17090b4c8100b0026b219f3399mr6286786pjb.35.1692727355656; Tue, 22 Aug 2023 11:02:35 -0700 (PDT) Received: from localhost ([2a00:79e1:abd:4a00:6c80:7c10:75a0:44f4]) by smtp.gmail.com with ESMTPSA id n10-20020a17090a670a00b0025c1cfdb93esm8183747pjj.13.2023.08.22.11.02.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 11:02:35 -0700 (PDT) From: Rob Clark To: dri-devel@lists.freedesktop.org Cc: linux-arm-msm@vger.kernel.org, freedreno@lists.freedesktop.org, Rob Clark , "Rafael J . Wysocki" , Pavel Machek , Len Brown , Greg Kroah-Hartman , linux-pm@vger.kernel.org (open list:HIBERNATION (aka Software Suspend, aka swsusp)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH v5 03/11] PM / QoS: Fix constraints alloc vs reclaim locking Date: Tue, 22 Aug 2023 11:01:50 -0700 Message-ID: <20230822180208.95556-4-robdclark@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230822180208.95556-1-robdclark@gmail.com> References: <20230822180208.95556-1-robdclark@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774956491272804085 X-GMAIL-MSGID: 1774956491272804085 From: Rob Clark In the process of adding lockdep annotation for drm GPU scheduler's job_run() to detect potential deadlock against shrinker/reclaim, I hit this lockdep splat: ====================================================== WARNING: possible circular locking dependency detected 6.2.0-rc8-debug+ #558 Tainted: G W ------------------------------------------------------ ring0/125 is trying to acquire lock: ffffffd6d6ce0f28 (dev_pm_qos_mtx){+.+.}-{3:3}, at: dev_pm_qos_update_request+0x38/0x68 but task is already holding lock: ffffff8087239208 (&gpu->active_lock){+.+.}-{3:3}, at: msm_gpu_submit+0xec/0x178 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #4 (&gpu->active_lock){+.+.}-{3:3}: __mutex_lock+0xcc/0x3c8 mutex_lock_nested+0x30/0x44 msm_gpu_submit+0xec/0x178 msm_job_run+0x78/0x150 drm_sched_main+0x290/0x370 kthread+0xf0/0x100 ret_from_fork+0x10/0x20 -> #3 (dma_fence_map){++++}-{0:0}: __dma_fence_might_wait+0x74/0xc0 dma_resv_lockdep+0x1f4/0x2f4 do_one_initcall+0x104/0x2bc kernel_init_freeable+0x344/0x34c kernel_init+0x30/0x134 ret_from_fork+0x10/0x20 -> #2 (mmu_notifier_invalidate_range_start){+.+.}-{0:0}: fs_reclaim_acquire+0x80/0xa8 slab_pre_alloc_hook.constprop.0+0x40/0x25c __kmem_cache_alloc_node+0x60/0x1cc __kmalloc+0xd8/0x100 topology_parse_cpu_capacity+0x8c/0x178 get_cpu_for_node+0x88/0xc4 parse_cluster+0x1b0/0x28c parse_cluster+0x8c/0x28c init_cpu_topology+0x168/0x188 smp_prepare_cpus+0x24/0xf8 kernel_init_freeable+0x18c/0x34c kernel_init+0x30/0x134 ret_from_fork+0x10/0x20 -> #1 (fs_reclaim){+.+.}-{0:0}: __fs_reclaim_acquire+0x3c/0x48 fs_reclaim_acquire+0x54/0xa8 slab_pre_alloc_hook.constprop.0+0x40/0x25c __kmem_cache_alloc_node+0x60/0x1cc kmalloc_trace+0x50/0xa8 dev_pm_qos_constraints_allocate+0x38/0x100 __dev_pm_qos_add_request+0xb0/0x1e8 dev_pm_qos_add_request+0x58/0x80 dev_pm_qos_expose_latency_limit+0x60/0x13c register_cpu+0x12c/0x130 topology_init+0xac/0xbc do_one_initcall+0x104/0x2bc kernel_init_freeable+0x344/0x34c kernel_init+0x30/0x134 ret_from_fork+0x10/0x20 -> #0 (dev_pm_qos_mtx){+.+.}-{3:3}: __lock_acquire+0xe00/0x1060 lock_acquire+0x1e0/0x2f8 __mutex_lock+0xcc/0x3c8 mutex_lock_nested+0x30/0x44 dev_pm_qos_update_request+0x38/0x68 msm_devfreq_boost+0x40/0x70 msm_devfreq_active+0xc0/0xf0 msm_gpu_submit+0x10c/0x178 msm_job_run+0x78/0x150 drm_sched_main+0x290/0x370 kthread+0xf0/0x100 ret_from_fork+0x10/0x20 other info that might help us debug this: Chain exists of: dev_pm_qos_mtx --> dma_fence_map --> &gpu->active_lock Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&gpu->active_lock); lock(dma_fence_map); lock(&gpu->active_lock); lock(dev_pm_qos_mtx); *** DEADLOCK *** 3 locks held by ring0/123: #0: ffffff8087251170 (&gpu->lock){+.+.}-{3:3}, at: msm_job_run+0x64/0x150 #1: ffffffd00b0e57e8 (dma_fence_map){++++}-{0:0}, at: msm_job_run+0x68/0x150 #2: ffffff8087251208 (&gpu->active_lock){+.+.}-{3:3}, at: msm_gpu_submit+0xec/0x178 stack backtrace: CPU: 6 PID: 123 Comm: ring0 Not tainted 6.2.0-rc8-debug+ #559 Hardware name: Google Lazor (rev1 - 2) with LTE (DT) Call trace: dump_backtrace.part.0+0xb4/0xf8 show_stack+0x20/0x38 dump_stack_lvl+0x9c/0xd0 dump_stack+0x18/0x34 print_circular_bug+0x1b4/0x1f0 check_noncircular+0x78/0xac __lock_acquire+0xe00/0x1060 lock_acquire+0x1e0/0x2f8 __mutex_lock+0xcc/0x3c8 mutex_lock_nested+0x30/0x44 dev_pm_qos_update_request+0x38/0x68 msm_devfreq_boost+0x40/0x70 msm_devfreq_active+0xc0/0xf0 msm_gpu_submit+0x10c/0x178 msm_job_run+0x78/0x150 drm_sched_main+0x290/0x370 kthread+0xf0/0x100 ret_from_fork+0x10/0x20 The issue is that dev_pm_qos_mtx is held in the runpm suspend/resume (or freq change) path, but it is also held across allocations that could recurse into shrinker. Solve this by changing dev_pm_qos_constraints_allocate() into a function that can be called unconditionally before the device qos object is needed and before aquiring dev_pm_qos_mtx. This way the allocations can be done without holding the mutex. In the case that we raced with another thread to allocate the qos object, detect this *after* acquiring the dev_pm_qos_mtx and simply free the redundant allocations. Suggested-by: Rafael J. Wysocki Signed-off-by: Rob Clark Acked-by: Rafael J. Wysocki --- drivers/base/power/qos.c | 76 +++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 20 deletions(-) diff --git a/drivers/base/power/qos.c b/drivers/base/power/qos.c index 8e93167f1783..7e95760d16dc 100644 --- a/drivers/base/power/qos.c +++ b/drivers/base/power/qos.c @@ -185,27 +185,33 @@ static int apply_constraint(struct dev_pm_qos_request *req, } /* - * dev_pm_qos_constraints_allocate + * dev_pm_qos_constraints_allocate: Allocate and initializes qos constraints * @dev: device to allocate data for * - * Called at the first call to add_request, for constraint data allocation - * Must be called with the dev_pm_qos_mtx mutex held + * Called to allocate constraints before dev_pm_qos_mtx mutex is held. Should + * be matched with a call to dev_pm_qos_constraints_set() once dev_pm_qos_mtx + * is held. */ -static int dev_pm_qos_constraints_allocate(struct device *dev) +static struct dev_pm_qos *dev_pm_qos_constraints_allocate(struct device *dev) { struct dev_pm_qos *qos; struct pm_qos_constraints *c; struct blocking_notifier_head *n; - qos = kzalloc(sizeof(*qos), GFP_KERNEL); + /* + * If constraints are already allocated, we can skip speculatively + * allocating a new one, as we don't have to work about qos transitioning + * from non-null to null. The constraints are only freed on device + * removal. + */ + if (dev->power.qos) + return NULL; + + qos = kzalloc(sizeof(*qos) + 3 * sizeof(*n), GFP_KERNEL); if (!qos) - return -ENOMEM; + return NULL; - n = kzalloc(3 * sizeof(*n), GFP_KERNEL); - if (!n) { - kfree(qos); - return -ENOMEM; - } + n = (struct blocking_notifier_head *)(qos + 1); c = &qos->resume_latency; plist_head_init(&c->list); @@ -227,11 +233,29 @@ static int dev_pm_qos_constraints_allocate(struct device *dev) INIT_LIST_HEAD(&qos->flags.list); + return qos; +} + +/* + * dev_pm_qos_constraints_set: Ensure dev->power.qos is set + * + * If dev->power.qos is already set, free the newly allocated qos constraints. + * Otherwise set dev->power.qos. Must be called with dev_pm_qos_mtx held. + * + * This split unsynchronized allocation and synchronized set moves allocation + * out from under dev_pm_qos_mtx, so that lockdep does does not get angry about + * drivers which use dev_pm_qos in paths related to shrinker/reclaim. + */ +static void dev_pm_qos_constraints_set(struct device *dev, struct dev_pm_qos *qos) +{ + if (dev->power.qos) { + kfree(qos); + return; + } + spin_lock_irq(&dev->power.lock); dev->power.qos = qos; spin_unlock_irq(&dev->power.lock); - - return 0; } static void __dev_pm_qos_hide_latency_limit(struct device *dev); @@ -309,7 +333,6 @@ void dev_pm_qos_constraints_destroy(struct device *dev) dev->power.qos = ERR_PTR(-ENODEV); spin_unlock_irq(&dev->power.lock); - kfree(qos->resume_latency.notifiers); kfree(qos); out: @@ -341,7 +364,7 @@ static int __dev_pm_qos_add_request(struct device *dev, if (IS_ERR(dev->power.qos)) ret = -ENODEV; else if (!dev->power.qos) - ret = dev_pm_qos_constraints_allocate(dev); + ret = -ENOMEM; trace_dev_pm_qos_add_request(dev_name(dev), type, value); if (ret) @@ -388,9 +411,11 @@ static int __dev_pm_qos_add_request(struct device *dev, int dev_pm_qos_add_request(struct device *dev, struct dev_pm_qos_request *req, enum dev_pm_qos_req_type type, s32 value) { + struct dev_pm_qos *qos = dev_pm_qos_constraints_allocate(dev); int ret; mutex_lock(&dev_pm_qos_mtx); + dev_pm_qos_constraints_set(dev, qos); ret = __dev_pm_qos_add_request(dev, req, type, value); mutex_unlock(&dev_pm_qos_mtx); return ret; @@ -535,14 +560,15 @@ EXPORT_SYMBOL_GPL(dev_pm_qos_remove_request); int dev_pm_qos_add_notifier(struct device *dev, struct notifier_block *notifier, enum dev_pm_qos_req_type type) { + struct dev_pm_qos *qos = dev_pm_qos_constraints_allocate(dev); int ret = 0; mutex_lock(&dev_pm_qos_mtx); + dev_pm_qos_constraints_set(dev, qos); + if (IS_ERR(dev->power.qos)) ret = -ENODEV; - else if (!dev->power.qos) - ret = dev_pm_qos_constraints_allocate(dev); if (ret) goto unlock; @@ -903,12 +929,22 @@ s32 dev_pm_qos_get_user_latency_tolerance(struct device *dev) */ int dev_pm_qos_update_user_latency_tolerance(struct device *dev, s32 val) { - int ret; + struct dev_pm_qos *qos = dev_pm_qos_constraints_allocate(dev); + int ret = 0; mutex_lock(&dev_pm_qos_mtx); - if (IS_ERR_OR_NULL(dev->power.qos) - || !dev->power.qos->latency_tolerance_req) { + dev_pm_qos_constraints_set(dev, qos); + + if (IS_ERR(dev->power.qos)) + ret = -ENODEV; + else if (!dev->power.qos) + ret = -ENOMEM; + + if (ret) + goto out; + + if (!dev->power.qos->latency_tolerance_req) { struct dev_pm_qos_request *req; if (val < 0) { From patchwork Tue Aug 22 18:01:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rob Clark X-Patchwork-Id: 136605 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a924:0:b0:3f4:cb6f:45e2 with SMTP id c4csp3907694vqw; Tue, 22 Aug 2023 11:53:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH2xdJ0MfdthnduV2NcGlIVeZV7fv/M5uqLjw/H1usM/uT9yuKo48di0scuq0dGUUeBd9H+ X-Received: by 2002:a17:906:310d:b0:99d:f7f3:520f with SMTP id 13-20020a170906310d00b0099df7f3520fmr11145631ejx.33.1692730418739; Tue, 22 Aug 2023 11:53:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692730418; cv=none; d=google.com; s=arc-20160816; b=cN+senbhbIv0m6LdyhYrtGi3oM5wCpSCLlJWJ+23O8xgccMTcBKEbOpeLdYnP+cXPE aoX0dLc7SR7SRnmDR1AF+W67NeAeuO56oWQlkDsO8oZFnCHrzFf13m0xM6HH7beqRUhX 0clyMn5TdSldnPp7YEmpXyagn0BOFCoiO4BZDyPBC+O5M3e21YpTkM0PE2CRZ83bPaOV Q9HsDn233E8OmssoELJYxF2wkiu/uWkHzUKdzseFB6/7TxlsWaDMpLZX8GzGep917KJX aNdVKLcblCaAztqIBue/VxSN7LVUnEdQdYgM1LfCOZgI+VTPJ+K5SQqcmDGcvLKXibzL 9t9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+XShSbqbLSjOdDhersXOLs7lzCQuSfnKt14AX57BEfg=; fh=1UHN2vYFDg1VELwt62RkkNSS7rhAuJQ/UuEEWkjgt1M=; b=lkhQYN61PwFMZhOBIyCFmMipY/9N6pJ3+HL5ElHcpfhf9/k+fz65JDUGl3v/BEq9uR RGSt3r5DyBVXPCGj0aOxexFv21z27TYxUkJS1HTV/M5W71KcdcA2+ObpEo0sSE64z2Wq SJdvH2mbdzp4jvKk2mMOS+wyuCR1g2dlDYMeF3awVHP7gzrJhW6Cvy+XhNmDYJgiUg5I biJ/DVMv4YX2d2fMYwiGIoZEKW6hU07spLlkgf5QBh1El+FpGG9LzbkVkmp/H9ZwPIDR XInfAfxXId5p+kx4c8iP1Z0Wew0nX1rreUPkLHIBKws83N7w7c675/Tevhib5k7kwUa+ +Dwg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=pUZ+14WT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c12-20020a170906340c00b00988796c2019si7865307ejb.591.2023.08.22.11.53.14; Tue, 22 Aug 2023 11:53:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=pUZ+14WT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229709AbjHVSCs (ORCPT + 99 others); Tue, 22 Aug 2023 14:02:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229645AbjHVSCl (ORCPT ); Tue, 22 Aug 2023 14:02:41 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B86F5CD7; Tue, 22 Aug 2023 11:02:38 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id 98e67ed59e1d1-269304c135aso3288749a91.3; Tue, 22 Aug 2023 11:02:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692727358; x=1693332158; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+XShSbqbLSjOdDhersXOLs7lzCQuSfnKt14AX57BEfg=; b=pUZ+14WTjGDCf1BQn529IVVAU3hq39BWMLQDcxyh3ZnNRBdW80UywIS+1pmIIApzlC qwS842G1JCCQnTm2GE6oMZBJelvEJUhgD2Vt6d+kHraK7Dnkl31KoMaN4WQv2KPpm59X 7MZU2Ldos4yc3gUepBLpO9lBcwusarlG6zxLEYeX6CEbONCv3dq6NYXvPJNGCVz2TlEF 8IDQFIcKMxuxoF8DTAbGfanLO9KXBeSmSHFjrWlyJLnWnxFR5OMc7gC1wmwHSesOW1y9 4VgfyDskCki8nCRAB6ulGOZbQL7awkJoyYTayPWZF9yyiFx8tdwu5nGGw1vPvf66bNRo FFIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692727358; x=1693332158; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+XShSbqbLSjOdDhersXOLs7lzCQuSfnKt14AX57BEfg=; b=coOmU9ytivoc1NcpdkDzsolaYIgqSOm0fvm8Txhro3Y0/zVFVyqF2jekMuFKcBoXsk wVWuPoq+LvfvrvZZwNHmxdBE0uiTrg2UGEyDlFkSxHRwUAFNntOMWJ2KY7ksOPKwVpF/ nRvIEqMWqw04wfeZW4bKf5LRf2TDKXge9s0acOsTjd33hKtXxxfQboilcsBHzRvGLBEE KIfiLtakaWo3RukEpDfcSd84ZSfedxLJ0uGP+fhX2Zqm6o+HLEgG0P2mTXt+COmGiJZ9 YhsdvKaePK9OZugetOE/TIVA6aHKBY8th3j7ncM/ECiPhcWBIJCAEcZbvpCtzOM5xevt 6CGQ== X-Gm-Message-State: AOJu0YzOO0qy3s4fF2AwscHei1mTF4bBUEUje3dgNtpC8Y0Cwt5gB+4I gTwbOSHVV+9JbyHIWRDV9dc= X-Received: by 2002:a17:90b:2304:b0:269:46d7:f1db with SMTP id mt4-20020a17090b230400b0026946d7f1dbmr9862231pjb.32.1692727358080; Tue, 22 Aug 2023 11:02:38 -0700 (PDT) Received: from localhost ([2a00:79e1:abd:4a00:6c80:7c10:75a0:44f4]) by smtp.gmail.com with ESMTPSA id cm1-20020a17090afa0100b0026940eb686bsm9910775pjb.20.2023.08.22.11.02.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Aug 2023 11:02:37 -0700 (PDT) From: Rob Clark To: dri-devel@lists.freedesktop.org Cc: linux-arm-msm@vger.kernel.org, freedreno@lists.freedesktop.org, Rob Clark , "Rafael J. Wysocki" , Pavel Machek , Len Brown , Greg Kroah-Hartman , linux-pm@vger.kernel.org (open list:HIBERNATION (aka Software Suspend, aka swsusp)), linux-kernel@vger.kernel.org (open list) Subject: [PATCH v5 04/11] PM / QoS: Decouple request alloc from dev_pm_qos_mtx Date: Tue, 22 Aug 2023 11:01:51 -0700 Message-ID: <20230822180208.95556-5-robdclark@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230822180208.95556-1-robdclark@gmail.com> References: <20230822180208.95556-1-robdclark@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774956491592389098 X-GMAIL-MSGID: 1774956491592389098 From: Rob Clark Similar to the previous patch, move the allocation out from under dev_pm_qos_mtx, by speculatively doing the allocation and handle any race after acquiring dev_pm_qos_mtx by freeing the redundant allocation. Signed-off-by: Rob Clark --- drivers/base/power/qos.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/base/power/qos.c b/drivers/base/power/qos.c index 7e95760d16dc..09834f3354d7 100644 --- a/drivers/base/power/qos.c +++ b/drivers/base/power/qos.c @@ -930,8 +930,12 @@ s32 dev_pm_qos_get_user_latency_tolerance(struct device *dev) int dev_pm_qos_update_user_latency_tolerance(struct device *dev, s32 val) { struct dev_pm_qos *qos = dev_pm_qos_constraints_allocate(dev); + struct dev_pm_qos_request *req = NULL; int ret = 0; + if (!qos->latency_tolerance_req) + req = kzalloc(sizeof(*req), GFP_KERNEL); + mutex_lock(&dev_pm_qos_mtx); dev_pm_qos_constraints_set(dev, qos); @@ -945,8 +949,6 @@ int dev_pm_qos_update_user_latency_tolerance(struct device *dev, s32 val) goto out; if (!dev->power.qos->latency_tolerance_req) { - struct dev_pm_qos_request *req; - if (val < 0) { if (val == PM_QOS_LATENCY_TOLERANCE_NO_CONSTRAINT) ret = 0; @@ -954,17 +956,15 @@ int dev_pm_qos_update_user_latency_tolerance(struct device *dev, s32 val) ret = -EINVAL; goto out; } - req = kzalloc(sizeof(*req), GFP_KERNEL); if (!req) { ret = -ENOMEM; goto out; } ret = __dev_pm_qos_add_request(dev, req, DEV_PM_QOS_LATENCY_TOLERANCE, val); - if (ret < 0) { - kfree(req); + if (ret < 0) goto out; - } dev->power.qos->latency_tolerance_req = req; + req = NULL; } else { if (val < 0) { __dev_pm_qos_drop_user_request(dev, DEV_PM_QOS_LATENCY_TOLERANCE); @@ -976,6 +976,7 @@ int dev_pm_qos_update_user_latency_tolerance(struct device *dev, s32 val) out: mutex_unlock(&dev_pm_qos_mtx); + kfree(req); return ret; } EXPORT_SYMBOL_GPL(dev_pm_qos_update_user_latency_tolerance);