From patchwork Thu Mar 23 04:00:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73774 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2708998wrt; Wed, 22 Mar 2023 21:07:00 -0700 (PDT) X-Google-Smtp-Source: AK7set9MSRlnVeypDTMB2d11MpY7s8uZ26x1DZkitcuFOnqRkOcvC2/+RjmqRlZc07bJZ+5yn0+u X-Received: by 2002:a17:906:26c4:b0:92b:4f8e:dddb with SMTP id u4-20020a17090626c400b0092b4f8edddbmr10600187ejc.34.1679544419978; Wed, 22 Mar 2023 21:06:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679544419; cv=none; d=google.com; s=arc-20160816; b=kbBeeSITL2Vm8j5JK5LQrBQlSI3/zhfJCL2bp2Tb9PHjGVsenxytKQnpGvr2ZVQCvX Ogv6M0xDohHbYtHRwMDFpG1D5KimzILjD3gs+tWFHaaGZBFv14Mk1sStka0vRlf5L5eZ SYTWLi9nHGnxodaTNIqUMCkVWn/PmI23m2Cs+ftMupws2HtzWo8SLM3T06mBLseyrzIM br5cpCamxvuGGZEA3e4m6e4fOYtqc3mSM4GzMn+0tt00rq6kS/tXkkrbakGQ/sHVr80P Dzr4tSQKqSkhceWVUy5GhKqxgbjiL5r0J9CPwOkgdAhkM4Mj36DgoFa4LMvvqXqUoiBq ldng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=zlRHMwQCwPxTz7URsIPFYEiZ1y/NkKG6JVNDPEffcgo=; b=Rub9bMpj7EihdBEXvC1YuhBSNyAGgHvOo+AKm3EOlzJ8qhKE59sLwDmh/58AZs6Wf+ 1E2iCvgU3BktmHrXUozOgemPrNdabSyz3W+VG23aL1uc+KrrHCmfHxONTn0lqz+slQND eO3/PzXh7U/hnq7225Kox8Qbb0JFK7vsQ92cKNzTUbMFRVLObGZzYkBr3Gakh4EL/dRN vxWeieTEOAHgQbq/kvc/bNyN+YUWlChuAWD+UB25NIYLePPkSMW9lcMCSKY3qSAP/0iZ GWhctCTWoIcpTA3nFNTCrJCYZwMqlWb3GaJ2mPf6m2AUZmJ5U5r1rmvnT/07Xbjts3Zb L2Bg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=q0IiHI38; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u11-20020a170906950b00b009332d9b2a96si9897342ejx.955.2023.03.22.21.06.36; Wed, 22 Mar 2023 21:06:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=q0IiHI38; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230238AbjCWEBA (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229507AbjCWEAw (ORCPT ); Thu, 23 Mar 2023 00:00:52 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10C941F933 for ; Wed, 22 Mar 2023 21:00:43 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id k3-20020a170902ce0300b0019ca6e66303so11693749plg.18 for ; Wed, 22 Mar 2023 21:00:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zlRHMwQCwPxTz7URsIPFYEiZ1y/NkKG6JVNDPEffcgo=; b=q0IiHI38SIToezuWJ5YCNMq/43EaxUo3i4AMEPHcC9bZCfWe3qfaYbxabY9EbcCoqm LUESQ/1v1D2aalE9SO+twkVNP5hnW+UNiSPbsUMxGnfaQG+8mDUTyeXBv18fXpGW3SWD GCVDR2eWTWBw72p7FFHYiZQYGqUWBUdEyjEquw53iNIuEWGhIQhSBNfbqBw3sfwQmtJy zq0COAvO1AYz/HqEKQJev5lhSkkjfsk/GYKhQDhBPMJgp6hmFHeDaNzuC/4om1icb3Uk IS34ZPkCnh/Dxdz7tz/xvLp+bFG8mdt7D5IhxCaGQj3LQws3RSkaD2yIkaXnXsqGxGWA GdAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544043; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zlRHMwQCwPxTz7URsIPFYEiZ1y/NkKG6JVNDPEffcgo=; b=S/qgH8MtyYFvHIVxpJ463gPJYKgqCGx2brnIwf7qBfzaw/mMYjqXFVxF4ZwYYhhnyR l3w5LoIDwEhNGQyKCwWrejI6bN4yK2Da2AIqPTeiAwtCFzg7MVcmAUab0w5abEuV92sG XjbOVkZpkFgg+zz9VVPgWLB0wYiPLW1eM/r0P6CUpGuiEi9peeM94z7JFA4Vj6+5Y6GZ y8X1ei+TDdrjUaxfeBtwvuP8TJgXeSOkRh9pTS0e+yptUGlIe+95tnpIDBh0khCnjLpv aWQFuPs6nzF+vhLNrFRchRiM3anKOl0bfQNJE2VtIVGjaFpOzIiqUYoPxZFWEYMSWCt1 umEA== X-Gm-Message-State: AO0yUKU85uguIEWZbx745PS8BMIfZcP2B6qS23CVpGiPR3ngTwzAJmt3 ovzYlBuYL+OhEJnFxWcD7QJk0g2O2VsOSfYu X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:e745:b0:1a0:4aa3:3a9a with SMTP id p5-20020a170902e74500b001a04aa33a9amr1983132plf.2.1679544043324; Wed, 22 Mar 2023 21:00:43 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:31 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-2-yosryahmed@google.com> Subject: [RFC PATCH 1/7] cgroup: rstat: only disable interrupts for the percpu lock From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761129969796955053?= X-GMAIL-MSGID: =?utf-8?q?1761129969796955053?= Currently, when sleeping is not allowed during rstat flushing, we hold the global rstat lock with interrupts disabled throughout the entire flush operation. Flushing in an O(# cgroups * # cpus) operation, and having interrupts disabled throughout is dangerous. For some contexts, we may not want to sleep, but can be interrupted (e.g. while holding a spinlock or RCU read lock). As such, do not disable interrupts throughout rstat flushing, only when holding the percpu lock. This breaks down the O(# cgroups * # cpus) duration with interrupts disabled to a series of O(# cgroups) durations. Furthermore, if a cpu spinning waiting for the global rstat lock, it doesn't need to spin with interrupts disabled anymore. If the caller of rstat flushing needs interrupts to be disabled, it's up to them to decide that, and it should be fine to hold the global rstat lock with interrupts disabled. There is currently a single context that may invoke rstat flushing with interrupts disabled, the mem_cgroup_flush_stats() call in mem_cgroup_usage(), if called from mem_cgroup_threshold(). To make it safe to hold the global rstat lock with interrupts enabled, make sure we only flush from in_task() contexts. The side effect of that we read stale stats in interrupt context, but this should be okay, as flushing in interrupt context is dangerous anyway as it is an expensive operation, so reading stale stats is safer. Signed-off-by: Yosry Ahmed --- kernel/cgroup/rstat.c | 40 +++++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 11 deletions(-) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..af11e28bd055 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -7,6 +7,7 @@ #include #include +/* This lock should only be held from task context */ static DEFINE_SPINLOCK(cgroup_rstat_lock); static DEFINE_PER_CPU(raw_spinlock_t, cgroup_rstat_cpu_lock); @@ -210,14 +211,24 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) /* if @may_sleep, play nice and yield if necessary */ if (may_sleep && (need_resched() || spin_needbreak(&cgroup_rstat_lock))) { - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock(&cgroup_rstat_lock); if (!cond_resched()) cpu_relax(); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock(&cgroup_rstat_lock); } } } +static bool should_skip_flush(void) +{ + /* + * We acquire cgroup_rstat_lock without disabling interrupts, so we + * should not try to acquire from interrupt contexts to avoid deadlocks. + * It can be expensive to flush stats in interrupt context anyway. + */ + return !in_task(); +} + /** * cgroup_rstat_flush - flush stats in @cgrp's subtree * @cgrp: target cgroup @@ -229,15 +240,18 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) * This also gets all cgroups in the subtree including @cgrp off the * ->updated_children lists. * - * This function may block. + * This function is safe to call from contexts that disable interrupts, but + * @may_sleep must be set to false, otherwise the function may block. */ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) { - might_sleep(); + if (should_skip_flush()) + return; - spin_lock_irq(&cgroup_rstat_lock); + might_sleep(); + spin_lock(&cgroup_rstat_lock); cgroup_rstat_flush_locked(cgrp, true); - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock(&cgroup_rstat_lock); } /** @@ -248,11 +262,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) */ void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) { - unsigned long flags; + if (should_skip_flush()) + return; - spin_lock_irqsave(&cgroup_rstat_lock, flags); + spin_lock(&cgroup_rstat_lock); cgroup_rstat_flush_locked(cgrp, false); - spin_unlock_irqrestore(&cgroup_rstat_lock, flags); + spin_unlock(&cgroup_rstat_lock); } /** @@ -267,8 +282,11 @@ void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) void cgroup_rstat_flush_hold(struct cgroup *cgrp) __acquires(&cgroup_rstat_lock) { + if (should_skip_flush()) + return; + might_sleep(); - spin_lock_irq(&cgroup_rstat_lock); + spin_lock(&cgroup_rstat_lock); cgroup_rstat_flush_locked(cgrp, true); } @@ -278,7 +296,7 @@ void cgroup_rstat_flush_hold(struct cgroup *cgrp) void cgroup_rstat_flush_release(void) __releases(&cgroup_rstat_lock) { - spin_unlock_irq(&cgroup_rstat_lock); + spin_unlock(&cgroup_rstat_lock); } int cgroup_rstat_init(struct cgroup *cgrp) From patchwork Thu Mar 23 04:00:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73779 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2714111wrt; Wed, 22 Mar 2023 21:27:13 -0700 (PDT) X-Google-Smtp-Source: AK7set+NpBn7jVTlqWEI3j18slZpAVwWo1u7vXkwKkLaK9TQ/H/OhnPHU9W5/gDB9xZPSGvcP8iY X-Received: by 2002:a17:906:f8cb:b0:93b:1cc5:4a1 with SMTP id lh11-20020a170906f8cb00b0093b1cc504a1mr5307918ejb.40.1679545633071; Wed, 22 Mar 2023 21:27:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679545633; cv=none; d=google.com; s=arc-20160816; b=juk6SB24I1oeXJsS3T4aIh/vTfcWC8hKOdvXtk4e8NnG46JDsbenoggmlhbhs52fJt XQj7Z+g6wjQK0aGfIMaEj7mJZ4EYHbJ4AlDbuNKmbkPgzxYv/13HHLRcCJZtyQF3H0AK aybvtpYQgEkEskEfzY8RDJ3+1fKd6MLpkwzhZxaLWQu8it83nWgGOxBRz3PTpL5lhKwU GgvCjekP4nT470OuLa4WoJR3yLdM3iZzUk5/8sPHLBW7I4wr7kY5kisDtoFNlQrJhLEw vJKliX1n0AvU7fnDX2EairIw4MUWGcO/xSBkS2xMuMLaWYgn6dax7pvZXXkrr5eljUhE vfSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=0V6KepxSWjC0U8YUMhyf5sEsW80RflfKR4tDJnFyjUM=; b=yy9K4KbHPTS5EB5HAZsmof/PcQ0SlkluTVuaLx81xlmR4I8qyGCTy4EqtlPZ4nOYE4 7WECvMJvKuX10Wp0x/oMRGMKCRyjauBjzgimYH/oqtzdlOg9ds4lY82HfM1mcWE8nYPu /mN6tN0fQ8gEWGbxPPP41W7R3KSROFL1UxjyyoIf5IxUiuZbcXrkv6fAA2lZFMqIGC7j YVgkI/7uO4FxdNc66fPHxwagJGoFOC8fQpVoruMpjYP/ILYVmMpcP41JL7cqgMF3Tc9D YJfOoQijSEwnYLN/uPotKmKaBCy7lkSqSLuY8GHOYeuHClutvHK6hcyMHyrz8gsQXghE 0LuQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=nvFwhwj7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o5-20020a17090608c500b00921d6a82dd4si17541800eje.969.2023.03.22.21.26.49; Wed, 22 Mar 2023 21:27:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=nvFwhwj7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230323AbjCWEBF (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229526AbjCWEAx (ORCPT ); Thu, 23 Mar 2023 00:00:53 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6BC31F935 for ; Wed, 22 Mar 2023 21:00:45 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id q30-20020a17090a17a100b0023d376ac2c5so321244pja.5 for ; Wed, 22 Mar 2023 21:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0V6KepxSWjC0U8YUMhyf5sEsW80RflfKR4tDJnFyjUM=; b=nvFwhwj7qGyMPQaTPrzYFw+T0XNSe66OQzM5b4pjwLTwVBRy9Yg/AbwpYFnRMIM5gG L465fbRhjctsBw/iYduTrtP7LUUTB2pfnwQVaagEQM0GyAtfd40drA654+PpJ27hrtIB BUWcp/c32KQ98SFLCZ9FSHYNM6vaide1p2wRQYwdJvG/bU3/l2g1YAjLINeXoxbcrV2W 3/G5Y/6fhAJDfUsgxXfVWzP78YqVFCMwh/+/7IKP9AVvA+7qfhsyQkLxneMASA+CreO0 VfuwrktKMl3lyTDdGpOXeKO210ebUx1dz4XnfSvMg+6i1Ne/G/0TP1oqkrL/txGwmyNx VJtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544045; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0V6KepxSWjC0U8YUMhyf5sEsW80RflfKR4tDJnFyjUM=; b=skSkR8VeGDk8/cPLOKpeLz72fuFTI2Mx2D6WBrEWdgxndtyLX5cD+OjZurGodx7iP3 srlm2HgSC03MUFYjVq/PTL2XHG9GacsKrtjPb6iYVeyrbwMksFBpzB7SiD+2K2tLbSYG nkXZcsBZyigtsEtk8Qlo2XcKX/jHj1LaFU1vgT93n7tGnl2uNkD/CwHQT6Mm0MnWAMjx apTIjIjXnmas4mM0PNKCYvOxbz+MVQi6l+LuycS+kCDEVSjTPRQdoHpnuwlY+AP8UBvu W3sH5RYvdbIzeS2Pv5xllahnNHxjl0xkoCkZnBJi+ZMj3WuIay40ueyz56atMolvUs09 FBsA== X-Gm-Message-State: AO0yUKWcEclZ5Y9ZVwAobIcGzKLmbg0FLaQrMjoQC/he5Dznj4pNcOQU 8fFnTZwJFthtse1ZB7u+RZeGywye7gi0qoCT X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:851:b0:628:30d:2d2f with SMTP id q17-20020a056a00085100b00628030d2d2fmr2879444pfk.5.1679544045305; Wed, 22 Mar 2023 21:00:45 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:32 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-3-yosryahmed@google.com> Subject: [RFC PATCH 2/7] memcg: do not disable interrupts when holding stats_flush_lock From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761131241733461819?= X-GMAIL-MSGID: =?utf-8?q?1761131241733461819?= The rstat flushing code was modified so that we do not disable interrupts when we hold the global rstat lock. Do the same for stats_flush_lock on the memcg side to avoid unnecessarily disabling interrupts throughout flushing. Since the code exclusively uses trylock to acquire this lock, it should be fine to hold from interrupt contexts or normal contexts without disabling interrupts as a deadlock cannot occur. For interrupt contexts we will return immediately without flushing anyway. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..e0e92b38fa51 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -636,15 +636,17 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * This lock can be acquired from interrupt context, but we only acquire + * using trylock so it should be fine as we cannot cause a deadlock. + */ + if (!spin_trylock(&stats_flush_lock)) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + spin_unlock(&stats_flush_lock); } void mem_cgroup_flush_stats(void) From patchwork Thu Mar 23 04:00:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73777 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2710018wrt; Wed, 22 Mar 2023 21:10:33 -0700 (PDT) X-Google-Smtp-Source: AK7set+cor7YhI2FSFvxXUPxWW1tBFjV2eOSZFYs0imyDQQ0+verbC04hOuQJkYCEdw1THhCgv5p X-Received: by 2002:a17:906:b115:b0:8b1:76ca:f228 with SMTP id u21-20020a170906b11500b008b176caf228mr9242572ejy.39.1679544633665; Wed, 22 Mar 2023 21:10:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679544633; cv=none; d=google.com; s=arc-20160816; b=LRwL6FQDNCrHKF7cIOSpmMepQGndSJGlSrTp+MdgLvg+xsoUx/SzPQjfyIstiMCVzT wgoSGDPI/hRMX/5K+f4y8INwwHxjoKxiLwmCA9+DjnpggL0Ioj0mLvJRBtVLRoqR2OtF 5OYkiVDeBxw65nqGIUAehOFTwUmbPZFEKnIJ1CBTRy8oYmiaPdC9z22aB6L0ww53RJDH Sw+/CJ38cefPUERb7jwrmoHFFGHh1c+5s70mVpuCYVu0pC1QWD0FsjsyYsogEAFBTGFb e0dBSigM9twJH6KvB8FLTzR8/cYqOWT6cGJU9sGBpy3KuKxXFuFtwUMoZTo7bVcyXTpo D4Vw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=736+92Vy7w+whzoBxwgGPc/WG8vZYtmz8qFOhZmUk9w=; b=ZNsU/t8DiCcl3BrEU+kpY6aBWpEfOMKT47+Dw+TO8WGCka9amqnzpZcBf9qeX4CTlD 2ryHbyL4xTaOtaswlCTpI1m5806KYl17L7m3sqpN84ZSxvleGwhmLs9B8fHlHzxhh8BH DMK1Jry22h+Hm1i9wO3JQOe+ywIhdhGrzwjZSq9Uu8kUZcCSnNwNoaATINVoLmAtgFv6 jTchGL7otY0SfLO3+jWzvFTaW8QlQKNI1VH23wEYO0T8ap1iT6uEIaTtS6hN3fEeAFk9 MArPEvb9z9mRP3AzfH4x9GJQA+9OvT+wyCuuIsH21CdJZfmATgAaR6+VTVB3Li9aPMi7 0YEA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GwAMe3Sd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n24-20020a17090695d800b0092bde9d2ed7si15609746ejy.988.2023.03.22.21.10.08; Wed, 22 Mar 2023 21:10:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=GwAMe3Sd; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229941AbjCWEBY (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229729AbjCWEAx (ORCPT ); Thu, 23 Mar 2023 00:00:53 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 05C9C20050 for ; Wed, 22 Mar 2023 21:00:47 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-541a39df9f4so210252057b3.20 for ; Wed, 22 Mar 2023 21:00:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544047; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=736+92Vy7w+whzoBxwgGPc/WG8vZYtmz8qFOhZmUk9w=; b=GwAMe3Sdh/YfT7TszWBGAWb/Omm+6/db9kDSH3rx0U2jTcaU6cdo97INaHv86kxNQA jhXBaZOARcF2wynwqEfwzWZoINYtyHtPiv9iRsKtX2yswvpMBD6qg4pgoYL8XYK3JdrY NUg095+EPmMHIupEKGzGCL9BJH8z0FwgmxWYVXMQd4hSYkBDyIXOQyRVbTCpsq34KRPV CyG3E5BvF3wTZXEl8tQiaUrIBU2ScbP5rypiT3jSEorEK7YISHgjHIMI4g2IycEeQTfl s5/P1y3KHUJjZSE6PJ3e2OlZHSMu+2KD/QDk8kFiQ8Hi9jpOcpijxBB3B0Ib/9iMoeNk 6zZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544047; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=736+92Vy7w+whzoBxwgGPc/WG8vZYtmz8qFOhZmUk9w=; b=08iZ/NY9CI4a8qqLeTjMorPm3RWwXESwnmgce5UQCzAJsIp9m64ayENE9LoNaE76S6 wI3p1tZPRgS0GpA7LKyw8xs1U7a7jJ8nEijHfO46HDjDwDWKeCqnKCSRlj1qsLfE/2+j 5Lvi2jwRYXZttbFA8StjwEnhpHeWheOnSlrQlzOIU3x89K00Awerf3UAjK4Mw0xnd81m xxLpSPOnnd3V4y3C6jrOs2QijWsj906IY99VOt0o8WG3nZRabCwJ5v892gs8h2TC2MVh MUGdHQJuAOSZXOkTfOAxEcFE5HarC/lCIxC9dkycpjwvUKCGHfvkLgosnYy1QWxyKcj8 EosA== X-Gm-Message-State: AAQBX9d/ajyz2d5znB4U/QE/smuGPpx6hji2tfEYD2zmQa7hHyj1/qES CaRrcItwbWIREANv1SX0gq7GEeZS+4/71MvJ X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:ad1b:0:b0:541:69bc:8626 with SMTP id l27-20020a81ad1b000000b0054169bc8626mr1137749ywh.10.1679544047097; Wed, 22 Mar 2023 21:00:47 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:33 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-4-yosryahmed@google.com> Subject: [RFC PATCH 3/7] cgroup: rstat: remove cgroup_rstat_flush_irqsafe() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761130193705522311?= X-GMAIL-MSGID: =?utf-8?q?1761130193705522311?= The naming of cgroup_rstat_flush_irqsafe() can be confusing. It can read like "irqsave", which means that it disables interrupts throughout, but it actually doesn't. It is just "safe" to call from contexts with interrupts disabled. Furthermore, this is only used today by mem_cgroup_flush_stats(), which will be changed by a later patch to optionally allow sleeping. Simplify the code and make it easier to reason about by instead passing in an explicit @may_sleep argument to cgroup_rstat_flush(), which gets passed directly to cgroup_rstat_flush_locked(). Signed-off-by: Yosry Ahmed --- block/blk-cgroup.c | 2 +- include/linux/cgroup.h | 3 +-- kernel/cgroup/cgroup.c | 4 ++-- kernel/cgroup/rstat.c | 24 +++++------------------- mm/memcontrol.c | 6 +++--- 5 files changed, 12 insertions(+), 27 deletions(-) diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index bd50b55bdb61..3fe313ce5e6b 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1043,7 +1043,7 @@ static int blkcg_print_stat(struct seq_file *sf, void *v) if (!seq_css(sf)->parent) blkcg_fill_root_iostats(); else - cgroup_rstat_flush(blkcg->css.cgroup); + cgroup_rstat_flush(blkcg->css.cgroup, true); rcu_read_lock(); hlist_for_each_entry_rcu(blkg, &blkcg->blkg_list, blkcg_node) { diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..743c345b6a3f 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -691,8 +691,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) * cgroup scalable recursive statistics. */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); -void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush(struct cgroup *cgrp, bool may_sleep); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 935e8121b21e..58df0fc379f6 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -5393,7 +5393,7 @@ static void css_release_work_fn(struct work_struct *work) if (ss) { /* css release path */ if (!list_empty(&css->rstat_css_node)) { - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(cgrp, true); list_del_rcu(&css->rstat_css_node); } @@ -5406,7 +5406,7 @@ static void css_release_work_fn(struct work_struct *work) /* cgroup release path */ TRACE_CGROUP_PATH(release, cgrp); - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(cgrp, true); spin_lock_irq(&css_set_lock); for (tcgrp = cgroup_parent(cgrp); tcgrp; diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index af11e28bd055..fe8690bb1e1c 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -243,30 +243,16 @@ static bool should_skip_flush(void) * This function is safe to call from contexts that disable interrupts, but * @may_sleep must be set to false, otherwise the function may block. */ -__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) +__bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp, bool may_sleep) { if (should_skip_flush()) return; - might_sleep(); - spin_lock(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, true); - spin_unlock(&cgroup_rstat_lock); -} - -/** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() - * @cgrp: target cgroup - * - * This function can be called from any context. - */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) -{ - if (should_skip_flush()) - return; + if (may_sleep) + might_sleep(); spin_lock(&cgroup_rstat_lock); - cgroup_rstat_flush_locked(cgrp, false); + cgroup_rstat_flush_locked(cgrp, may_sleep); spin_unlock(&cgroup_rstat_lock); } @@ -325,7 +311,7 @@ void cgroup_rstat_exit(struct cgroup *cgrp) { int cpu; - cgroup_rstat_flush(cgrp); + cgroup_rstat_flush(cgrp, true); /* sanity check */ for_each_possible_cpu(cpu) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e0e92b38fa51..72cd44f88d97 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -644,7 +644,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush(root_mem_cgroup->css.cgroup, false); atomic_set(&stats_flush_threshold, 0); spin_unlock(&stats_flush_lock); } @@ -7745,7 +7745,7 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + cgroup_rstat_flush(memcg->css.cgroup, true); pages = memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -7810,7 +7810,7 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); + cgroup_rstat_flush(css->cgroup, true); return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); } From patchwork Thu Mar 23 04:00:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73776 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2709294wrt; Wed, 22 Mar 2023 21:08:01 -0700 (PDT) X-Google-Smtp-Source: AK7set/1C7eu4CR4RrpUi0pJb4TprsHbq4vXB4KQach5/m9LylDvr5wxIuGei09QJPMJ39Z8NUN2 X-Received: by 2002:a17:907:9886:b0:932:5f7d:db33 with SMTP id ja6-20020a170907988600b009325f7ddb33mr8666538ejc.34.1679544480950; Wed, 22 Mar 2023 21:08:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679544480; cv=none; d=google.com; s=arc-20160816; b=mfQj9xob6EqUDTwYw+uctXiPM6SULY03X4pHIMpOVSRjCOUpNJJ2X4NjoLqQWYFdvw pnWxz1fZl4qFXvWF1WIwk5Xb9olYcuh9y06m8DFVyWvLmqfX26rBUCRQZipqOFXxGVTN w+DhtcpxxghW3orDC97HygOqF2qJc6hRPqq2nRZae/9j7oCKVcSHKKgaR/xIYBMfIK3c F2zXgjeLx6+L6T1R9QH41CHCuCcNG5gFXjvIWb40avItTSoKRLGVl0UDY05H9vZ8z3t7 5LcmH9zDjrsF6Mlv5Zsj0jvvFho4D1TIo2S/6TS+TFvFv5RkfG2MWuJnMTsetn17XM1i QTzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=xrG0SezuE3D1BzHk1bOIN+tOwhU+wFsyLck18RLDkiM=; b=I0fW0LYE/ltdp3QHwzxyOBhmKrcCUi2BYXyLPoK3hIG5yIuytLT2L6ou5WV2RktkFd wt9RLh6s021RhWOBIE6futLLFwlSzOpzs4SWcooHe1LNnoi7q+2GyxxorsGyjpfBsxFR zGBgQCdfYn567H1KeGr42VS2cLgS3CzLYblx7SKmBf41pWViH+oLygfeyrJQdDLsFq65 amZ3pxeK9hwvXmZ62y0ayi1dDvaLR0cEaFkI2GyGp2Jp6SBFYXLPcE8fEtuHAjtaAMP6 P0Lw/WuqYBa3uIblXZsXYTe32YXE4XaWh2jPfKumgWe2x4+x985EyC/z1hwEjI14HWeL rTxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=cgw7YjSY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n24-20020a17090695d800b0092bde9d2ed7si15610040ejy.988.2023.03.22.21.07.37; Wed, 22 Mar 2023 21:08:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=cgw7YjSY; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230350AbjCWEBN (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229889AbjCWEAx (ORCPT ); Thu, 23 Mar 2023 00:00:53 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 915CC2005E for ; Wed, 22 Mar 2023 21:00:49 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id e129-20020a251e87000000b00b56598237f5so21501786ybe.16 for ; Wed, 22 Mar 2023 21:00:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544049; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xrG0SezuE3D1BzHk1bOIN+tOwhU+wFsyLck18RLDkiM=; b=cgw7YjSYw+yZCrYg4MWNWyL1ocWVlMfhI6U/IH5SNNI/KXZEirBQTnVHw1Jk19ca7x YZMRDXY2GIXRShVJyUBfGcwrm5OQ54Bn3XOEBif3BbHD9ArewU0b2vLyFm38bp8+NRx8 4lwk3DHRAIpY5JhlVhZi+qJFnS5DTPOFtneU65+9wwMhEPN3ScjeohcSLADEGreopxgz hPSmYPd9gVS979yLkwNh9j+9QGCIU5M1yATespEadqsIAym+pG18x2O6VFUlYJCK8ejc v8yJF8c/6m0MiyUkNdlDzwM4KjC9ikuULQ0wGXuneSrTmETcMqRGnUz3Mhll67JgWKef njxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544049; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xrG0SezuE3D1BzHk1bOIN+tOwhU+wFsyLck18RLDkiM=; b=MR1O9Qxzk6nZixuEj25uHXaHsvr9cgOjAGZFSylwaaoUVWKDd6T4/1zSBu9oGWpZSz wGxg+NcT8hB2yCtU7QPQOcPdyM2n0fsIkG+H5PW/DfmkR6iQI95R19yu0KVCMdNPFCUz BHuQa6AawxQM6Urs2nsvpzhsOk6K4sqZxsy/0UuboIZmXqCaOto//bEnIVTWzAq5nkTD Opb8ecbyeLRdKzPGjPoOmLvf0MGGmtVNWj51lNkKgIyrWKUAMjcfvwX5nJ1w7BJjcBV2 HOv1G0rxc+MsUg/z/1xED3QTNMfRWfcIn5p531E2pQkX11KjEBNk6J56wvSzNAJgEgsR pbdg== X-Gm-Message-State: AAQBX9d0+OlOfmDdEvmoGvWT/FEM6qAFARYNj2wI9ehZYUZ6eE0vgM6R /xJfwHf4TiuicN3HBTqGX2o0yTMYJVaaBUlM X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a81:b149:0:b0:544:bb1e:f9cf with SMTP id p70-20020a81b149000000b00544bb1ef9cfmr1185027ywh.4.1679544048914; Wed, 22 Mar 2023 21:00:48 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:34 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-5-yosryahmed@google.com> Subject: [RFC PATCH 4/7] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761130033580349455?= X-GMAIL-MSGID: =?utf-8?q?1761130033580349455?= Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Enable choosing whether sleeping is allowed or not when flushing memcg stats, and allow sleeping in safe contexts to avoid unnecessarily performing a lot of work without sleeping. Signed-off-by: Yosry Ahmed --- include/linux/memcontrol.h | 8 ++++---- mm/memcontrol.c | 35 ++++++++++++++++++++++------------- mm/vmscan.c | 2 +- mm/workingset.c | 3 ++- 4 files changed, 29 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..0c7b286f2caf 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1036,8 +1036,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats(bool may_sleep); +void mem_cgroup_flush_stats_delayed(bool may_sleep); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1531,11 +1531,11 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_flush_stats(bool may_sleep) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_delayed(bool may_sleep) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 72cd44f88d97..39a9c7a978ae 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static void __mem_cgroup_flush_stats(bool may_sleep) { /* * This lock can be acquired from interrupt context, but we only acquire @@ -644,26 +644,26 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush(root_mem_cgroup->css.cgroup, false); + cgroup_rstat_flush(root_mem_cgroup->css.cgroup, may_sleep); atomic_set(&stats_flush_threshold, 0); spin_unlock(&stats_flush_lock); } -void mem_cgroup_flush_stats(void) +void mem_cgroup_flush_stats(bool may_sleep) { if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + __mem_cgroup_flush_stats(may_sleep); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_delayed(bool may_sleep) { if (time_after64(jiffies_64, flush_next_time)) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(may_sleep); } static void flush_memcg_stats_dwork(struct work_struct *w) { - __mem_cgroup_flush_stats(); + __mem_cgroup_flush_stats(true); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1570,7 +1570,7 @@ static void memory_stat_format(struct mem_cgroup *memcg, char *buf, int bufsize) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -3671,7 +3671,11 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * mem_cgroup_threshold() calls here from irqsafe context. + * Don't sleep. + */ + mem_cgroup_flush_stats(false); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4014,7 +4018,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v) int nid; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (stat = stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=%lu", stat->name, @@ -4090,7 +4094,7 @@ static int memcg_stat_show(struct seq_file *m, void *v) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) != ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (i = 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4594,7 +4598,12 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). + * Do not sleep. + */ + mem_cgroup_flush_stats(false); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); @@ -6596,7 +6605,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v) int i; struct mem_cgroup *memcg = mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(true); for (i = 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..59d1830d08ac 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(false); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..042eabbb43f6 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,8 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + /* Do not sleep with RCU lock held */ + mem_cgroup_flush_stats_delayed(false); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 23 04:00:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73775 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2709189wrt; Wed, 22 Mar 2023 21:07:39 -0700 (PDT) X-Google-Smtp-Source: AK7set82Nn1h1vqmKBR+PHWAugEmVXWln2V/eX2aqeMN11tZvtKPSLhH/yLtzsbnr3sK9JCyF8NE X-Received: by 2002:a17:907:98e3:b0:930:2241:3bff with SMTP id ke3-20020a17090798e300b0093022413bffmr10150696ejc.67.1679544458868; Wed, 22 Mar 2023 21:07:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679544458; cv=none; d=google.com; s=arc-20160816; b=EHlrLOeUW0qaSc1g1lARWoFXWY5w/qh2djIaygYIIeqVB8ezsbMz0jRr+Y60DIdnc9 mnyTYTzajogzNqlCqqMwMuS2CIugQDoS6bu590mNLR1CWfDFGp87UsxTcGxPmu6+plup 3lK5z1xuF8p/VINRKfqAl0gm0ZLnT8uBNtlc3DtiCxXYhJcd9/roA8q7JgwV8KcvM96Y zcYqjgjpn/uH9/FkTDXnr+5/AdwD8W9Ub8STjxJjlIhRB77e98ncj/IjGOdC3R0t1eRH mG6MwAXMLW09e+LGhi8yafvufKAZnRJIK73QuzZLUE1Y32mOdvDt5/b82L1dC3Et/X35 6uXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=9m7S+Ebo8MkLWAncbvKsOW8wVqp1O+0Jr3lM7bEbkLI=; b=WXto82C+8toN5jEndbv+1akbquxHPL6W/3J7RahDew97QR7NLxaZpkdkJ5qJ+lOVke Huc1VQtP8PpMdvrb09eFQASoPe93SjJhFChD6LgExopiLYGEciim+bf0sygNri6zWbz/ gIX7JtzbySuQAl68bnR5sOaJwLZFSmwJjoyIReIY4vIWo4sCOyXmUuVezXMFAp7QA6w7 Dt2QEeiqd4tnuKTtyBoFRdGcOBc1Y9wAQGm69DE4RhcRv8PhSaT8v8dngQHM7vfdOcZi u7/0fHt55JBg0XghrRFbahzDUAoxLTgQTL56wOmiloRtbA/nK2fXo/VWPdp+RFwdVfho 5EUw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=VdCCijFi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ne38-20020a1709077ba600b008bded90af7csi22243450ejc.531.2023.03.22.21.07.15; Wed, 22 Mar 2023 21:07:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=VdCCijFi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230340AbjCWEBK (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229955AbjCWEAx (ORCPT ); Thu, 23 Mar 2023 00:00:53 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3702E20077 for ; Wed, 22 Mar 2023 21:00:50 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id b8-20020a17090a488800b0023d1bf65c7eso321934pjh.3 for ; Wed, 22 Mar 2023 21:00:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544050; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9m7S+Ebo8MkLWAncbvKsOW8wVqp1O+0Jr3lM7bEbkLI=; b=VdCCijFiKDJ4yDz8tFMap2tBXNfAakMH2cmowf1HUrZHuzx98c171wgP+Q37bsMmOY dvm3L1704oXKLAgAkOK0GjUnYQoP54+fQujrx3ACvUW9YBMoC0EV4BPolgDqFHWKSqHY vC+M0AbP8Thw306r5f0tU0s0RnClNRrpM7iLZQZCh4urjjm1nPlLVN2cvKbhB0gks3OE JIza4tgsYELq7gTHgeHRM8vhlVtpzgxg6m7gbPXptQ7q4d0Q9K++kyZO3UiZG82ponKY DM+ht1D3A+Eg8Co2dt+ym/gky/eIx2ki3TeVSJ2PxhVMHdlX5/2KaImaACWOeUxKy2jU FXPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544050; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9m7S+Ebo8MkLWAncbvKsOW8wVqp1O+0Jr3lM7bEbkLI=; b=o0qFcF7Ja9qH1/eOABAH5KeTpTNOCvqQF0KKNRjZB7Vpuz98hQRNYHlPGFnku3x7N9 5W0kRwWAYsx1qEFlidlNjmQvsnW8qRlXrhXO+R8QZZ8ah4QczPRqbR3JgX1VuaQhlLFz eDurzePWPp2BcB7iEr+UHvSazm/UEfc3a0i39Ci0p8SMAH8FJZP0nWQv6m8Bic1CwZ2z WOchGPJV8lN/7Vt84v80zmjnkphA71dV+FnLowMJv1AOV/z/x6vryDQfNwdwoJrs1Z/f QU5cO3TrozWNh623sJyjnWJeUNF3VcoxXlkACtPk8tgp/rXcWBDjVO/yszIyGaO/BXWB 0GXg== X-Gm-Message-State: AO0yUKVYL3iaoXGRc4Ioo9kH4oIXaD8hzXN8qGUmN9C/7M+/S/hbFJ+W AHqyLKCYUIsiI2F4iS/NriH0AcrEhjmIt6Jb X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:80e6:b0:623:8a88:1bba with SMTP id ei38-20020a056a0080e600b006238a881bbamr2937696pfb.2.1679544050539; Wed, 22 Mar 2023 21:00:50 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:35 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-6-yosryahmed@google.com> Subject: [RFC PATCH 5/7] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761130010676783370?= X-GMAIL-MSGID: =?utf-8?q?1761130010676783370?= Memory reclaim should be a sleepable context. Allow sleeping when flushing memcg stats to avoid unnecessarily performing a lot of work without sleeping. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 59d1830d08ac..bae35cfb33c8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(false); + mem_cgroup_flush_stats(true); /* * Determine the scan balance between anon and file LRUs. From patchwork Thu Mar 23 04:00:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73778 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2714098wrt; Wed, 22 Mar 2023 21:27:10 -0700 (PDT) X-Google-Smtp-Source: AK7set9oO15uXnDVEWjKpVDzEUZIIG+Kn+++kwAbw4vzOokmIwdvOSXRPxy9u8sVsLHSf7PNhh4x X-Received: by 2002:a17:906:80c2:b0:870:58ae:842e with SMTP id a2-20020a17090680c200b0087058ae842emr9165377ejx.24.1679545630327; Wed, 22 Mar 2023 21:27:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679545630; cv=none; d=google.com; s=arc-20160816; b=JjcVegaf5336Y4H5Vg+8lazM860HmmwrijwrCpuM0oNueJ+Bs8OOdSX7bpMChOwlTp A+vRjhvo8EsdbSUk4Zi5Xd+VG0pvJP3k7j4283I1mxJYR/eWdsIg81pJyL0wbFE6fiqm l1SzmmMcTaSMhw0JGehLeH0jQzUAdx2kKETiDb+vFbLtG20lQ/EMt/NO5LeebBCQKdS+ fZBT42C0eqaEYhlnorsf2ol0/aRKB1Q8rgK0U6DuWNuY0ao2/qtpnXNaOPV6gGWgE/gs hbxgbBAu0M6LP2mX3dig9vXwNgEBpNAOlj4tFSNb7dBCw3vI5UzBRwjhztwAA4uN5W5J Kkjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=IoeZYZryJtZ5YKLINZigwUk+eCMrWPKZYyl5pNjfs0s=; b=DvW6z05yP1RDyGP7nq/4lgL3U4dF4JocmqCHP8QmZFKq2FhR/T/yeddYU2r/VmI2UE yB9mkpkctZg/D4zVYipwb4+Hia05pDdGIfuFfRAPd8LQWZkKQMXLEim+rIEejh0LOhDq cAsxuMdCDD2SsGNMQGLwIq8Bv2gtosyJs5yzqZNNSb7quxpxPqPRmYqE8Gm2wm0Z/PnW fKMW8M/LV01WLvgMGZIsX1knxnsgqbPBxdmn6CQF3HGi+ZYAaH8oKk0w94ZB+C1QRAy/ 3hHY/gPTms1U8iWEc1XB5tHXxBCeh8qP+cmVWXujQ9FeaNuDhhuqI0G69eltgRHyTIwx KzCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=CYUJafqL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l20-20020a170906795400b00933f92e9917si10665830ejo.371.2023.03.22.21.26.47; Wed, 22 Mar 2023 21:27:10 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=CYUJafqL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230381AbjCWEBS (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230195AbjCWEAy (ORCPT ); Thu, 23 Mar 2023 00:00:54 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D72F32CFD6 for ; Wed, 22 Mar 2023 21:00:52 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id b1-20020a17090a8c8100b002400db03706so332087pjo.0 for ; Wed, 22 Mar 2023 21:00:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IoeZYZryJtZ5YKLINZigwUk+eCMrWPKZYyl5pNjfs0s=; b=CYUJafqLeSTzFoXixoXxJfg+d8a1R4JvCUerDh64ifDjpdaxbdbWqowybN/gvJ19LK Og473fSzkkpOeMA58DC64BMRm6M0MCGDM/VGSfHLkAlVcw8rDdEYpMTrVbzG7kCaO9yo XEy99SSbQbOpczsL5Zy04pgal2f5dv+k8luu/GDpUWQB1RZl9qYQhWpR8p3p3vD98PGc ce78QmYoavhlvFbHk6YMCQ65CM/lnXEYztj/xUq2pBOa3UnU3uBgApeimhahme8eLkyC CQZkL9MM7bPQMzx+6ozclKMKwr8T8qPF+ZR71Vewcxu4v3esM5hXIWD8hj0gOLxV+qOY L9ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544052; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IoeZYZryJtZ5YKLINZigwUk+eCMrWPKZYyl5pNjfs0s=; b=0OdWIZXWG69ojLsWv+qKYE58VdO4AsnmylmhUc76hqlo4lef2r99wxipnYsAHgUIJ3 dNBWbhj9k8CVdi/VnOSX+hgIcnOHIdBpyY4L4VKuPF0+B+LBuFH2WTO3DqKw8TwnMSWo suAeGCz7YyQ9czuZBuPihOv07OEguyBSV2S93fuihiK9fr6rBjcJMNcNPpl74wlxH4WE 5PhtaU3CBBVtyoX8SaYARsKhwBcTgiXTKsflvXRRsAKOfWCwuIm3UaDucGqdbtFsxg9x I1kOWUcSdnqwTVS/YPU0M3LygLghuwHMUBvIGJ5MCz7+So2U1VgvurASD/A4Fjp8+oF7 rXKQ== X-Gm-Message-State: AO0yUKWVDqqVpuRBhtZB40AeZg3kZLCsZCxzFk1F30HVAMvpLqlWHl0H W4xBl1rK3tUA/mQ2gNxtmXiJOxtETXu9EOYl X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:ce11:b0:23f:d473:dd44 with SMTP id f17-20020a17090ace1100b0023fd473dd44mr1881490pju.3.1679544052257; Wed, 22 Mar 2023 21:00:52 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:36 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-7-yosryahmed@google.com> Subject: [RFC PATCH 6/7] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761131238755469835?= X-GMAIL-MSGID: =?utf-8?q?1761131238755469835?= In workingset_refault(), we call mem_cgroup_flush_stats_delayed() to flush stats within an RCU read section and with sleeping disallowed. Move the call to mem_cgroup_flush_stats_delayed() above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Signed-off-by: Yosry Ahmed --- A lot of code paths call into workingset_refault(), so I am not generally sure at all whether it's okay to sleep in all contexts or not. Feedback here would be very helpful. --- mm/workingset.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index 042eabbb43f6..410bc6684ea7 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,8 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_delayed(true); rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,9 +463,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - /* Do not sleep with RCU lock held */ - mem_cgroup_flush_stats_delayed(false); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 23 04:00:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 73780 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2714133wrt; Wed, 22 Mar 2023 21:27:17 -0700 (PDT) X-Google-Smtp-Source: AK7set9fzcEUuzCDpOBRz0a7Pl6eynutZl+u4/u0b3TBiPixY6gp3E28Ni92ytCZ6vugCZkJUGEm X-Received: by 2002:a17:906:5901:b0:933:4c24:101b with SMTP id h1-20020a170906590100b009334c24101bmr4459378ejq.7.1679545637155; Wed, 22 Mar 2023 21:27:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679545637; cv=none; d=google.com; s=arc-20160816; b=YonH7PWlA0MEzjDv5OUf0cXh9uIfqDUOLFSHMmug2S45e6M2c/pU1xPi64hSXlUAIn dTqK206arKBLg0J62pQMeBNArKkbpenlmz34yhfg/fKeTMwRlFErVC9UOTC70igvkSIr GLqF26SQvX+wLDVWHIhsBlOZp+iw9o7WGBqbTEiBE9SXXOjBClhuH2lXXmhVK85ZMxJU L489Q4P7ydV+ZhOWZyRxOB8Ff5HwSklak9wyhK+1LyGdd8HH56pyhBDUV5X6pyzOnYEY +YZCGVsJqqoNToc6qRAYC/onGQs1ymWRmLpPRPv0EvdjT5VxkodE1uLs/8aKt4LFkxTb jJcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=GbeROioJTSRzpGZIx65CHc5YHmhlz1VRWOdmKxovUco=; b=TticxdsH/ELVyBwrSSq0h34XIce+daYZoTo8cfAlhglRYJAEsRA8pSOsTexcywsciP d+rWI0UH2dHho5nrhXH22QCnNQkDDuNbxzKdaEv6qo1+fyexPGh6gd2CquvoFDaXSHGs HFMMwumbG4CeaEtvKJx4FJAftmQi/Eo3ATbKZXkz70ZIRJHdPKAaaypGo/hg6MuUYBdE ArQJfyxXHJ86F9duWPodi5hx1QEpE/mxDeh79+9jSe/xi+7M/7ZmlB/EJV00tcrpRqU9 dgjlC9FEqUpRqP5P/rks6O3nRNaBxBLKnnslKfiuEb+GS9yFlOHdsaKGJxqN9B9aq3fo YwwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Hsck6MLI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dn3-20020a17090794c300b0093956a5d4f8si10209640ejc.1000.2023.03.22.21.26.51; Wed, 22 Mar 2023 21:27:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Hsck6MLI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229379AbjCWEB1 (ORCPT + 99 others); Thu, 23 Mar 2023 00:01:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230209AbjCWEA5 (ORCPT ); Thu, 23 Mar 2023 00:00:57 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B32501F91E for ; Wed, 22 Mar 2023 21:00:54 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 205-20020a2503d6000000b00b7411408308so2175654ybd.1 for ; Wed, 22 Mar 2023 21:00:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679544054; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GbeROioJTSRzpGZIx65CHc5YHmhlz1VRWOdmKxovUco=; b=Hsck6MLI0adIPASRaTY3d/eUkBNtZFq5fHTozHNB8yOtl2ZkHkJQQ4GNrxQh4iSoWz JkY9CbdFxu+ujh+Pyx1oKlS53Vj0EANz1HJ/mNRwoFFcZsbYiHuLCMd4Lx5axIZ0gB5l yaad/tHsVvn1tHU57blkHy/+dS8g9HHclDAV/QT3pdal8fCNCqCcJD9lMmCv5J074fQc TEWaFK69tR0mz3hWUAlm/mUgTwQep/HE638zK7huhLNEr0Yv6vPjLeQWwk4UQCFbwI31 KRefecpL1tiBaS/Y5Rnpmz8ntCNmT0BogLXtNGk9m7i1zmm8BH4noVNiIwwFJbgEqIWE ZH9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679544054; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GbeROioJTSRzpGZIx65CHc5YHmhlz1VRWOdmKxovUco=; b=JukNIAJswgiQyw+3ObOE3Ykw+fTArfXh+rvf5H3xIIxGctQaIzhpbqizm4xLiXUr6F KecpNHxN3GleQt6T25DtsFw+n9igJwCoqq+a7KbGX6OmUnUFxOZ6wOmRc1sFD1Hp3SOA L/zZOUzcgRpnv4bua1zDN3qeuR48+Qq2BQLs6PvVPsiuT1CnH6A86mH0o1ZjEto5krLQ rtnyobGcpXuCirfl39UEVC6i0x7fXlE7MJZ7yDsvYHH2PEgT/qfnDqIIO1lss8EiwQDw 6MqAnqPFa4Lz5c8FpPRnsjRT3doQJR9DnHYua0eK1FHLsCvKDynbqO8bazALr7AVADZF pG7w== X-Gm-Message-State: AAQBX9ffCJk14nBlzOZf0MBz9jdT2Q7Z5U0QzmJKcE8e1EgMO5sAxckl kh52pqayu6KwYWiA4I1Oc8mGbMA0VNbZH6tX X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1549:b0:b73:4a25:fd36 with SMTP id r9-20020a056902154900b00b734a25fd36mr1286158ybu.2.1679544053971; Wed, 22 Mar 2023 21:00:53 -0700 (PDT) Date: Thu, 23 Mar 2023 04:00:37 +0000 In-Reply-To: <20230323040037.2389095-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230323040037.2389095-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.rc1.284.g88254d51c5-goog Message-ID: <20230323040037.2389095-8-yosryahmed@google.com> Subject: [RFC PATCH 7/7] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761131246025557687?= X-GMAIL-MSGID: =?utf-8?q?1761131246025557687?= In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 39a9c7a978ae..7afd29399409 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));