From patchwork Sat Mar 4 03:10:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 64151 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp763414wrd; Fri, 3 Mar 2023 20:16:35 -0800 (PST) X-Google-Smtp-Source: AK7set/xBi6MJt6mvuzP+3zfs8x6Qp+t8jIJWp6gPjhdZdscJmtlHuA7LXscPuHUVetYrjTBIzQZ X-Received: by 2002:a17:906:c001:b0:8b2:d70c:34ae with SMTP id e1-20020a170906c00100b008b2d70c34aemr3948457ejz.71.1677903395203; Fri, 03 Mar 2023 20:16:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677903395; cv=none; d=google.com; s=arc-20160816; b=e6rNU462q3vI8p/xG1u9sFqS7zzWprBaR5LW/TOasaM0UeXjVe2Cp0WPpqOHrYHhFK pzSAK/tG2J7YYJCnY3Fs2BDp+poD2x/2UsILYLgrPoayTwZlduJrr5m+jXNxmnW7bleD FrgFkKfb990RMzn5EQBpiX6t2gK5maRpNE28WzPB2HlbVFcmJ0iqI854hcZED+mMG8z1 CKP+4EhIyPpLzo71rAnVaGsW8vviX7CQucfHF0cdVCXh15YvqLedp5LBGvcIklLzH7Me Zc+03f4JcbaSjfyhTJ9QQJ3QHPuISzqpzgJuebM3fdalwMtSvjKKzeY2tH0LP3nabPRO 4s0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:from:subject :message-id:references:mime-version:in-reply-to:date:dkim-signature; bh=s3BTWvMS4HbVHXG0hGuYCfsLy+8T8n3QOZ6K5OLKcZQ=; b=cIm293RL18cids8Wh649a36F1UftJ8pljhRtceju+OftXI6A3ax1iv79AOi05ET96J B7xf9hskfU1cmCyh11fY/0PiDhZvrSV2b82NwFCXgDuDDz/u6Mp8a/zs2J/MwdSQu+Tg 7xnu5sYMrw/4xpopjxWeeuEmn7CVtW8zGKAsywFRbJHAdP+6ZO45DAB3AFNts82PKXtM 7cldjJHko3ZkYI+3mMx/xbXA7OztmMDRLYg9TOI6bjUq5opi/fd5kLnaTpJKbys3jo77 KlzWdgaQenPZVduwV1pRNjUKBmW5Fyqvhj4moht5bfUZwXjSVwGjMm5kr/IMPGK0olgO LnNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=PYqykcRT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v9-20020a170906338900b008c03cbdabb1si3764761eja.633.2023.03.03.20.16.02; Fri, 03 Mar 2023 20:16:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=PYqykcRT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229656AbjCDDKn (ORCPT + 99 others); Fri, 3 Mar 2023 22:10:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229437AbjCDDKk (ORCPT ); Fri, 3 Mar 2023 22:10:40 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ABDA7D30D for ; Fri, 3 Mar 2023 19:10:39 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id h14-20020aa786ce000000b005a89856900eso2258338pfo.14 for ; Fri, 03 Mar 2023 19:10:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1677899439; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=s3BTWvMS4HbVHXG0hGuYCfsLy+8T8n3QOZ6K5OLKcZQ=; b=PYqykcRThs1t2MTEZDoAFl2N5+VjqJGMnW4HMnvWS5uOUbY6+X687w0oTX07D/azgz +9MjpV47JZw4tR4ecQ3S7yLACiD9m8QM1crbz18tu9WFwPuTN7MvLN8jkGt9DMdWbk55 oju/xXXYD5qYZO6V9v5FSJWt9KBOZ5Kb8HkaIq8EbdSam0foeEdV9qIH0UYc8w1SyF/E Kevi/X8l+7oEpr3p9BGJvNBLne2WwG4/C42JbHnPNh76yw9vpytxaQBOEXnPEuWb6+Fk wxrLYk5D+aJjgWOBbt9ao0wkkJ1EM91CqdH5sHAUKGqpMm2xewGwnREgxkg/7lL5tLC8 oVKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677899439; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=s3BTWvMS4HbVHXG0hGuYCfsLy+8T8n3QOZ6K5OLKcZQ=; b=i96JRE2SCJ705mdrEwxMr3d0dfYMvAPWxf/LXxdmKoWVWU6RaASOxssgXcNU7rj1jv ioUQUyykppJDn5wQS9ppDdR5uxTpHP3FyWgiSmr88PB21t30/omL01ydUagjcNZjUDv5 QETvvudxNDVs5O8VRm3hm/VkzIpCgqs9zTT7sTCd4k7HLHNVyic+34j82FY1Mqju1qe5 x2bG0/VmyIJ/YQA+VrSr6u3ZYhb+s10bICHbRCRgLZkIW7TFFur3P9D/IxZA8aC5ol4e GVnPWfiXURgDlMszWy83ZRRKMznXoLc0SB9+Fi1qic0AlZQFd0w/Lo3Mm+AIhr90j04t JkrA== X-Gm-Message-State: AO0yUKW+YD/mKNLmIgf+DBCZDIKEAgORuZfGMjNe5Aw2x96gdlEUlsQt gKJLYdl2B+fPZrPkoIKiaQ3ZHHtJIh60p7L+yJnPXuWVVhmnhPAy8b7M7f8xHiUiy2zJ9Thml0Q E0TyVBhz+IjWeOIaueF5ehMXZewx/8ixM4sPKwYghymym4/3JRLovr6dcpqsP5w05/aki6OA= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:903:2587:b0:19a:9f86:adab with SMTP id jb7-20020a170903258700b0019a9f86adabmr1474639plb.7.1677899439077; Fri, 03 Mar 2023 19:10:39 -0800 (PST) Date: Sat, 4 Mar 2023 03:10:29 +0000 In-Reply-To: <20230302062741.483079-1-jstultz@google.com> Mime-Version: 1.0 References: <20230302062741.483079-1-jstultz@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230304031029.3037914-1-jstultz@google.com> Subject: [PATCH v2] pstore: Revert pmsg_lock back to a normal mutex From: John Stultz To: LKML Cc: John Stultz , Wei Wang , Midas Chien , " =?utf-8?b?Q2h1bmh1aSBMaSAo5p2O5pil?= =?utf-8?b?6L6JKQ==?= " , Steven Rostedt , Kees Cook , Anton Vorontsov , "Guilherme G. Piccoli" , Tony Luck , kernel-team@android.com X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759237379141527337?= X-GMAIL-MSGID: =?utf-8?q?1759409230443953859?= This reverts commit 76d62f24db07f22ccf9bc18ca793c27d4ebef721. So while priority inversion on the pmsg_lock is an occasional problem that an rt_mutex would help with, in uses where logging is writing to pmsg heavily from multiple threads, the pmsg_lock can be heavily contended. Normal mutexes can do adaptive spinning, which keeps the contention overhead fairly low maybe adding on the order of 10s of us delay waiting, but the slowpath w/ rt_mutexes makes the blocked tasks sleep & wake. This makes matters worse when there is heavy contentention, as it just allows additional threads to run and line up to try to take the lock. It devolves to a worse case senerio where the lock acquisition and scheduling overhead dominates, and each thread is waiting on the order of ~ms to do ~us of work. Obviously, having tons of threads all contending on a single lock for logging is non-optimal, so the proper fix is probably reworking pstore pmsg to have per-cpu buffers so we don't have contention. But in the short term, lets revert the change to the rt_mutex and go back to normal mutexes to avoid a potentially major performance regression. Cc: Wei Wang Cc: Midas Chien Cc: "Chunhui Li (李春辉)" Cc: Steven Rostedt Cc: Kees Cook Cc: Anton Vorontsov Cc: "Guilherme G. Piccoli" Cc: Tony Luck Cc: kernel-team@android.com Fixes: 76d62f24db07 ("pstore: Switch pmsg_lock to an rt_mutex to avoid priority inversion") Reported-by: "Chunhui Li (李春辉)" Tested-by: Chunhui Li Signed-off-by: John Stultz --- I know Steven is working on a fix to address the rtmutex not spinning, but as the earlier version of it didn't resolve the issue for Chunhui Li, I wanted to resend this out again w/ Tested-by tags, so it is ready to go if needed. I am looking to get a local reproducer so I can help validate Steven's efforts. v2: * Fix quoting around Chunhui Li's email name (so they are actually cc'ed) * Added tested by tag --- fs/pstore/pmsg.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/fs/pstore/pmsg.c b/fs/pstore/pmsg.c index ab82e5f05346..b31c9c72d90b 100644 --- a/fs/pstore/pmsg.c +++ b/fs/pstore/pmsg.c @@ -7,10 +7,9 @@ #include #include #include -#include #include "internal.h" -static DEFINE_RT_MUTEX(pmsg_lock); +static DEFINE_MUTEX(pmsg_lock); static ssize_t write_pmsg(struct file *file, const char __user *buf, size_t count, loff_t *ppos) @@ -29,9 +28,9 @@ static ssize_t write_pmsg(struct file *file, const char __user *buf, if (!access_ok(buf, count)) return -EFAULT; - rt_mutex_lock(&pmsg_lock); + mutex_lock(&pmsg_lock); ret = psinfo->write_user(&record, buf); - rt_mutex_unlock(&pmsg_lock); + mutex_unlock(&pmsg_lock); return ret ? ret : count; }