From patchwork Thu Mar 30 19:17:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77374 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp80967vqo; Thu, 30 Mar 2023 12:20:06 -0700 (PDT) X-Google-Smtp-Source: AK7set+mNAPnAdV8zQgmuwyr6FuAddHVW6Drcnqw0wIzlFtqLJz1y7RoM/w2a7ep4uaa1ZavROi1 X-Received: by 2002:a05:6a20:2d94:b0:da:4d25:8fdd with SMTP id bf20-20020a056a202d9400b000da4d258fddmr21376786pzb.38.1680204006504; Thu, 30 Mar 2023 12:20:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680204006; cv=none; d=google.com; s=arc-20160816; b=ukzN2nwdrYXmhdxXoOR0K9K7x/qrzMvAeYesj34BeAQWmaVbDMsHxm+gwQAPfZWH9H S2vMSyWjxeUAocCSeYzQIJc9AQzPOaQ5umKTpcNQu54nOCI7zo9ymvy407ihEYIwDc2g tmSXQCtwvgwuf2m1fwCGMgBhJo+RmjZIMmYKnpJXrVKPSMrRjlaOWbgGanYr1noumcHT PQVoxa9Xu/rOv6MhM8uMDncOnE/FTjkq6mvgNqMvbu41HiqL8Svi74Dgr1ofeYbUXhyv 9lXkZUvHauzHzgQGKPcTRJdkYKhbYXZV7Vt5D3Na1XMkSo3L5V79kb/RVIuuJbEehhg3 /MdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=gt8Iux1fItox7SKP+KGt/7G+kfJ457NjvMpdEWZj2kYTpeRqkk54ZX2x1Vh4bHGP8p oy3EXdcbtCvW1/it3LJG0nAmj7h3BWFqZGi4pPX3Luc+UB1oRXOlYue9nfqZDpdYiasc mxor18Sa3qV80roQtWD8P8RTIhBkwR/lqptUcesserRE3eIngrHj4D9HleegOI8IyGSY c9caAg5nkYzwfega8qp/5/wqVYvE+MumlxTLLaAfmMtS6uMlf/l5o8aHSAMrd3pbybz/ 5HxmxmgO0Bpy5hISDUVIj3rpwR/V2F5z//Mnkp/6N+aTm5q4UBIGLC2tiGbTiqHSryaY rF+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=tY0E3oyb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q65-20020a632a44000000b0050b6f305b9fsi357679pgq.606.2023.03.30.12.19.53; Thu, 30 Mar 2023 12:20:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=tY0E3oyb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232062AbjC3TSK (ORCPT + 99 others); Thu, 30 Mar 2023 15:18:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229475AbjC3TSH (ORCPT ); Thu, 30 Mar 2023 15:18:07 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59072E1A9 for ; Thu, 30 Mar 2023 12:18:06 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id g187-20020a2520c4000000b00b74680a7904so19669925ybg.15 for ; Thu, 30 Mar 2023 12:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203885; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=tY0E3oybgVZIuKKOOKBykm8/TKbf2scexLJVT3DkzJcaMo8pgAVQKd31VkYqY1QVPJ mHo+yh2Z2e/UWhIAS6NyKjgzsMFfM3Q7bRmRT9xetX5hDAZEg0lz49jvXxYxOEOA0GHz 2iOR7gvTKjdYv9QT9UofKbbkFs9vSWVQ0TKS3904q473sNNcjSfvya1m1B2ab+CL/f+M 4qtYTT2pypw3Co5Bobg3nKNc5an7wdNYyXFt6rny9eUWrmoVflUhZJQGBV9r+u9oNXbf thlMA7X4kUIdycQp/zBF4S4OoaK6pqFmZjXA4rakf85jUKtTz9rXXIDv36itoKwLwzmv vt/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203885; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ECd9n7cK9XN/T+IJ39MyPXAm65DzgHuyZCFnDGQgZIE=; b=ZxbIG5mekY3H39mY2QQYiVP1LYkFZ//Lazw3LpX5wKT4k6KvgQdVKRFH1imlXR8vPi /jTohxS+KYDihKf+kgQZzhrE/MpkJjHz3DsPrNZkofoBOG0+GxN4buOJVSrcS/7aw9Zj jT+Ex+dF2xWexJknMUw6Z1oRDdS04ieuvlwm2ZiGKL6ZXhssQaaXtxzTWN/kTZytNuEu qQVGZkLGzbqBNlg9bwwEpcAdygnnE2NVuvyj60NpGRR1CbYc0zn89GRfGfiXuvax4/OY 60vXBq3h+4vkYBPc137BiepkOL3Ovezl2H5V0qW2l7RAQ/pKrxvQda5oEq1MYe2BCrdG 4BcA== X-Gm-Message-State: AAQBX9cw1NmTKaicQqqXd1b2KP1qMo92a7cf2jOZoEGkmUvOvcYsXKw5 qcxYAusKIL9sKVkx+Ax5IXstKwfwPeWxv2EO X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:d1d8:0:b0:b6d:fc53:c5c0 with SMTP id i207-20020a25d1d8000000b00b6dfc53c5c0mr4650822ybg.1.1680203885608; Thu, 30 Mar 2023 12:18:05 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:54 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-2-yosryahmed@google.com> Subject: [PATCH v3 1/8] cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761821596665141858?= X-GMAIL-MSGID: =?utf-8?q?1761821596665141858?= cgroup_rstat_flush_irqsafe() can be a confusing name. It may read as "irqs are disabled throughout", which is what the current implementation does (currently under discussion [1]), but is not the intention. The intention is that this function is safe to call from atomic contexts. Name it as such. Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/cgroup.h | 2 +- kernel/cgroup/rstat.c | 4 ++-- mm/memcontrol.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..885f5395fcd0 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..d3252b0416b6 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -241,12 +241,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) } /** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() + * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() * @cgrp: target cgroup * * This function can be called from any context. */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) +void cgroup_rstat_flush_atomic(struct cgroup *cgrp) { unsigned long flags; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..0205e58ea430 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -642,7 +642,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); spin_unlock_irqrestore(&stats_flush_lock, flag); } From patchwork Thu Mar 30 19:17:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77375 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp80975vqo; Thu, 30 Mar 2023 12:20:07 -0700 (PDT) X-Google-Smtp-Source: AKy350ZcbMg7dcOzpS+kQCi+WN+wdLpT2G8aIkZcqBD6ytQIwH7XtwP5/eyQcwncIA8vtjarbfUS X-Received: by 2002:a17:902:e1c3:b0:1a2:910f:76ee with SMTP id t3-20020a170902e1c300b001a2910f76eemr1910346pla.64.1680204006999; Thu, 30 Mar 2023 12:20:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680204006; cv=none; d=google.com; s=arc-20160816; b=AWxea2CIGLZMu5FnFZnS7exSRFgnOEMtVOXPezs99mT/edqEc/wgUckN9NIllKXh4A PTYNFcAJsPFqJP88KErVJpijDEVCGYMZptQB5btn4XmDcIQDqpd9vKLZL0cjltVx86H1 tkxyqlNxcV1JduG2dD25qfecKyEI1rBwbwWZdIqWdFaV3ha+qxmcD90Dz48O8etmNzZQ cvCWFLuXtQQKpQ3nQ9zgxyoKmj+2lapiaORGCvqDTygxY3h8gsYXN+Wcnes3JU8mkpc5 MLixounNyamxPC2XWAFON/FtXBfpLqeuj7Btj44EGvOIDshKUtuxj5ZG6q5tTKpgjJpL 5sqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=LXlZsArwPDY7OuU2BxzeoXG1UXSoglEAfUQ5EzzMXG4=; b=ap9twfKijpjCP0WSMjj/P64bHzO6D71mTxnC4bFHpSEh/JFzu6wyQL8lALZzfZX1il AkicYKtOpYEHqeLJ9D3Ay0e+yG0uIXqbDUG9bAX7RAwiDsZYd5zP+XjM4UJkyWgyH7/C smkI24aD9cNhNeQ/L9geF4nEdcWv+eFsmPzW5+pnAyG+S8d23nJ8TQWqqkWu7RKRpqbg TvoXqy0bmXuFwrQiycvIeuvQk4F+hc/s61DoCWp4oXiZRRXuzRRtC+WmHKxg6VqEgAWe YO/dw4JabUYWoSmnsBVM/3C4iYbdnW+ciwPj3dYslVYLFDTmbWnjfRL5B2V+C4vrZWcR QarQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=QmYQM+j1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jw6-20020a170903278600b001a0a44fa57csi189077plb.389.2023.03.30.12.19.54; Thu, 30 Mar 2023 12:20:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=QmYQM+j1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232001AbjC3TSN (ORCPT + 99 others); Thu, 30 Mar 2023 15:18:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232042AbjC3TSI (ORCPT ); Thu, 30 Mar 2023 15:18:08 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDAA5EB52 for ; Thu, 30 Mar 2023 12:18:07 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id n15-20020a170902f60f00b001a273a4a685so3198465plg.15 for ; Thu, 30 Mar 2023 12:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LXlZsArwPDY7OuU2BxzeoXG1UXSoglEAfUQ5EzzMXG4=; b=QmYQM+j13cdbjBiZy3Y+dzkJUgGOVHTdr9lvDbSE859fYaJtq+8zmH02RDaKbdpvap CqejHKuoW/4IBz5mIHOf6Y8AHa6GMb/hCZ7zMABh1AePaLkgpcjAyzU7xGg8rQCDHyuT xd/GhSCAHuv9H/zwirCUR2wyGX+B6mkWwB9WVRwD19bK9qyHzg7+GCXy42vkNiRXXZdr kN4hOyhVE2VLeXDBEBRHB8fSA0RE0BkgU+jq32E4Jq+3xIhGwrpX/kGZ92PdxYBPFxyB 8OgXFYA1yB1rgGnRYxUqB9CL2/ZWFsevIT5f1UxupCOTzAHu4BNZEW/BqlkhNKD16qxi CoQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203887; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LXlZsArwPDY7OuU2BxzeoXG1UXSoglEAfUQ5EzzMXG4=; b=qvTi/aMtpKB6w2CpMin1BuoekJqIsfvRt1hUhLEkBy8Gj+ykRmaamt2vRStS1G3sSP S2v5Z32CL42p+uKEnfxbbqegA4AT7f1imV5WPj57/QJEMfTcnsD6nSJMXyptWkndOYAM HdjxR8Jf9tmnZKQ75poPq6F1+y9wXNeVSp2Gv4Ivc0jFerl8o4MHXOcMUFbrbbgGDvqR +og+K/aFJaRUpc6DBiR4DVlbP6T6XPhEgq3aqTo5CFz/Husr13Ru3zl6Ab1bWfYTrNwy CzMsVieyjHjJsRiZ88wSJtlcdwt/9HKgNEjUjQC/IEMGMEEwcKUFTkhfdgedGbtWwih1 iM2g== X-Gm-Message-State: AAQBX9fOfEGT+KHmVtFWyZOC1cxdfK6yMO8foq13BRQn5lN8mDaH7MJg ams6FKOAM/YFQ4bVuYCJZguCnpUukmDHp1/O X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:ec11:b0:237:9cbe:22ad with SMTP id l17-20020a17090aec1100b002379cbe22admr7641616pjy.5.1680203887352; Thu, 30 Mar 2023 12:18:07 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:55 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-3-yosryahmed@google.com> Subject: [PATCH v3 2/8] memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761821596799586904?= X-GMAIL-MSGID: =?utf-8?q?1761821596799586904?= mem_cgroup_flush_stats_delayed() suggests his is using a delayed_work, but this is actually sometimes flushing directly from the callsite. What it's doing is ratelimited calls. A better name would be mem_cgroup_flush_stats_ratelimited(). Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 2 +- mm/workingset.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..ac3f3b3a45e2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1535,7 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0205e58ea430..c3b6aae78901 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -653,7 +653,7 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, flush_next_time)) mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..af862c6738c3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 30 19:17:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77377 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp81516vqo; Thu, 30 Mar 2023 12:21:03 -0700 (PDT) X-Google-Smtp-Source: AKy350ZIPpUFZbVlMdxifVVyS4O5kzLAeKLsz53xJDegV3GEnFceQvUBfldqV9CcY2V2LV7OSE30 X-Received: by 2002:aa7:9f96:0:b0:62a:63e6:3282 with SMTP id z22-20020aa79f96000000b0062a63e63282mr23963949pfr.11.1680204063221; Thu, 30 Mar 2023 12:21:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680204063; cv=none; d=google.com; s=arc-20160816; b=s9h+T/X0K9BnyXToBkpmSHGBrvNE8rO7n0Y+SBVkge0bDlmR9NmSnpp7+hbN61JF/k 2//8EurSut31iRg/One7FifKozh40Zg+q8OHTvNf+coA1GtyesLR/atOk8hRuXS9Zy8v G/qlEU6nTnYSnpOaOlTa3Mpff3j2bel36HZUpXHFmfGg2yIjCDs1ss0MUIohus7t87k1 dtp2ssOxWcXL0lUuY3tQts7b/Km1y+DcX44lEehBrYCCTLBv/U8GLgyPwlgLcjDF5Yvy J+daiJS5glbTfUD50ljoH3JPq99pspbxICaK15gDQBZyO+H9ZbuaecXGdWwttWVKW3Cy I6OA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=aVus0GFA9nzxOZ5lNIH1Nx8kHcLEa/aLkX98gl9HKOI=; b=qec/KNkpxDVCPbuzuZlLJHjY+sA1vIi8qyjQ0ThuknYnpwNm4e8nanQdkTEg/mMshQ pwcWMt8x1Zj64+qVkxvHYhpXzeEtI9jxiT2nwDFwsMoGtWiqRnCZrDlmLo5x/C8KNvEo EUhq/p54WqzGPzbG947qK/ZHxJAZkIZPGZeT8O7ITrmoVQFsNj7HYGGyPWB7vli/p4LU XP0y7WgUnybo2w+uWBdK5A1UIf5bTLDYCI8C6GnaA+dQhMqz0xXAn4EC/ttsMn6At5nd 8TDWn0AGLDWel3TdGkWqA82+BiJOFWCgezxUfipqi93aJ4ndKZkuz03xjAL0pwwS80g9 R4ZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Rb2fsB8a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d2-20020a63d702000000b0051382242e41si344123pgg.415.2023.03.30.12.20.45; Thu, 30 Mar 2023 12:21:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Rb2fsB8a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231591AbjC3TSX (ORCPT + 99 others); Thu, 30 Mar 2023 15:18:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232065AbjC3TSN (ORCPT ); Thu, 30 Mar 2023 15:18:13 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5DC90FF13 for ; Thu, 30 Mar 2023 12:18:10 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-545d3b026a8so147771347b3.7 for ; Thu, 30 Mar 2023 12:18:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203889; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aVus0GFA9nzxOZ5lNIH1Nx8kHcLEa/aLkX98gl9HKOI=; b=Rb2fsB8aWLuAh/6UfhLacuvSldBOfVahXww4db+vgAZH6d8hHK5iS/Pb3yBuHu8zXu 8OHs3K+JgNd5BWKL8HmUqSNmeBMWOUfM93BgScQIJGiFyUTuS/OhqqsgB8B/zESgECWW id0La031n/vRfUHiqMH5ptV4ZUHLtuC1awDopegwUjfP8gYPSzkD9GboR6Sv+BT5F0il 4cP0p/kpaTUv9ECKBi7X6nDyHboM9b8Z+lLZ6iyDmkLsMIt+cWstg1DQo4e/ZnrIN56I iRBi8QgWvfbGxJTMXxv2zZuY7CwzGwzqEOEYe4XvyV9MRzo4MibTgBPKLkzWKa272DY9 IgGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203889; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aVus0GFA9nzxOZ5lNIH1Nx8kHcLEa/aLkX98gl9HKOI=; b=k97e73fenYqFFFRDAuV8vwTfxBgT1TPJzJa/v0Qv5ofOLwSh8ueSeT9UD4ltXIneAS AWqEIcP9JYHUmrVTCT6gudts214BwRbbtMr1WmOf/5XHtVz/l9EmKq/PU/oqAJbiYT9T pWIxFGCAkwfER8/WyheOgFljTg28BHwHL8f9Qw6bF2Scr3cvSThHYbFyhWrZjN94wZK+ 1wwa7a8xZJ3HFzucmX9WovWBRW/0aadDDWBbk8oBGCBAQ7rQ4tjhu6DFLzWbXBB+6qq5 Anrr1xg39H2+j4uGQ6fKv/Fy+JFAILGQa5H0VpQP0UIyaCcaNMJqFX3tUcsMTS4WP1Yt PfIg== X-Gm-Message-State: AAQBX9fY6fE5PzbJvVk9GsYuquUqUUeQmxd8gbAPyUSunRpTFzlrBesz gRLUcAWgnSd0vin97Y/HjD1Oo7zfGHQ5dnYo X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:10c3:b0:b79:4826:e8e3 with SMTP id w3-20020a05690210c300b00b794826e8e3mr10930499ybu.1.1680203889210; Thu, 30 Mar 2023 12:18:09 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:56 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-4-yosryahmed@google.com> Subject: [PATCH v3 3/8] memcg: do not flush stats in irq context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761821655288158885?= X-GMAIL-MSGID: =?utf-8?q?1761821655288158885?= Currently, the only context in which we can invoke an rstat flush from irq context is through mem_cgroup_usage() on the root memcg when called from memcg_check_events(). An rstat flush is an expensive operation that should not be done in irq context, so do not flush stats and use the stale stats in this case. Arguably, usage threshold events are not reliable on the root memcg anyway since its usage is ill-defined. Suggested-by: Johannes Weiner Suggested-by: Shakeel Butt Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/memcontrol.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c3b6aae78901..ff39f78f962e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3669,7 +3669,21 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * We can reach here from irq context through: + * uncharge_batch() + * |--memcg_check_events() + * |--mem_cgroup_threshold() + * |--__mem_cgroup_threshold() + * |--mem_cgroup_usage + * + * rstat flushing is an expensive operation that should not be + * done from irq context; use stale stats in this case. + * Arguably, usage threshold events are not reliable on the root + * memcg anyway since its usage is ill-defined. + */ + if (in_task()) + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) From patchwork Thu Mar 30 19:17:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77376 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp81050vqo; Thu, 30 Mar 2023 12:20:15 -0700 (PDT) X-Google-Smtp-Source: AKy350bN5n4S5XJFjYe2XqmdYmM/0mvgy16Tqt4KSaSEuqS8KcS0N/4Xp/M9EVaeVfaa33KOxy1+ X-Received: by 2002:a17:90b:33c7:b0:23f:35c8:895 with SMTP id lk7-20020a17090b33c700b0023f35c80895mr25969780pjb.32.1680204015422; Thu, 30 Mar 2023 12:20:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680204015; cv=none; d=google.com; s=arc-20160816; b=uhnqwG/j/04eb6skUkrE7EeAxB2mVQzm/mK8XfzXcQiIKdYxo+I9Xbbzt3E52wRB7A +qofyexLDkDzMuvl1USMNT50vrqSi4zmqZTjMxf0eL5g41K56krTpmQoRNrb3vN1DL5Q 7fXyoTwpVD6FKVmCgm+7dpeJKeHR45QGl/gTFSy5Q6gKhn+JkrVPRY++PXaCfrDlJo+7 E7pSYjwbI+KGK0Q15+ntLigeYVK8QO74nXys0x9UaJa06EODiFoioqP7GEOpttiMwsy4 hp/bH4Cf1cn8Spca9gJ1er806dJt5N4CHFOYv/hi4Im60zEtJtjgA8deAzcmKm8nS1xo k4Rg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=4nnUf1Bi8zb8HMhH9DtFsVEWdtAjMz82Sf34ZimV0fg=; b=0zyKGGAiRPQ/iPaMv0QNw6LWJcr1J1OdAelGX1DC9zZLx7sbySMvDyk/JH0FXhZj1a NR2AnLF6WMDhstNOKJxfKrg70SiuwTRR6zU889fbpwt9PCaWhKtS1G/NKkLLpCiPQ40z M2FsoY5ggWY6T1rYog599PK5XTB7gfnVA19w9Es451cWADlwjQPsleDzNfL3Mk4ZfwXT gP7Uw/5zjiL3oeHMS3FeSb9LDA0x0MkumqOY/LkKwReKT8yzgltkbzR8w+DrI7ODJQrB HQ2UuwUUfWN+cmJGOTqYs3eDZJ3ZprUf6wdel7PO5nmcThGsXWaXa5+jgO0Xgd6Hkebh +0ww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=X2RFp1pZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gl3-20020a17090b120300b002307345bf7bsi2612622pjb.23.2023.03.30.12.20.02; Thu, 30 Mar 2023 12:20:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=X2RFp1pZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230253AbjC3TS1 (ORCPT + 99 others); Thu, 30 Mar 2023 15:18:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231845AbjC3TST (ORCPT ); Thu, 30 Mar 2023 15:18:19 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FD7EFF35 for ; Thu, 30 Mar 2023 12:18:11 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id q9-20020a170902dac900b001a18ceff5ebso11612229plx.4 for ; Thu, 30 Mar 2023 12:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203891; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4nnUf1Bi8zb8HMhH9DtFsVEWdtAjMz82Sf34ZimV0fg=; b=X2RFp1pZWOYnRsIh7IWjsVsM+KEFc9WJSHxF/6WWA0h+C4aKaj5fmmxIF9JIbSgACd wSFJlPVZHSShfit9sIv2gEMkHiLDYI1aj0ZjNxN+KDFEUuJ5hrS0dAiMoZtgG8tE01Qw Z6IuI1YwDe0K3jJKl09PQ+88WTXCXmLj3mXjZyc3NpqNvz62t2kYidmfKl22OQzA3sQQ EAjXhE1Jh+vHizPfZUZ3LQ+IzSkqjVL6en02vxZp22lS6vBDlsf8gGPBHsHSCXP5Ac15 JluiS/D70cGu14FmTesFiFf/4cN9gJ7flhGTivB535PauxJnWfGkMyHN0wJ36oB1Mtk7 YlVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203891; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4nnUf1Bi8zb8HMhH9DtFsVEWdtAjMz82Sf34ZimV0fg=; b=Zm+lShhCPbLalzq59B8ZMMAt7Sk1N2DiS8uEa9o2MEcLdyl7aos1Ou4UviPfuaw3GB zMYozFQw0mMDnjRlOVKbn2OPfhaE/hDmHotuEKQfN4Uunh3WeyINHDanuGhT13VFLHzF /BH0vlZxZrBgOXWB6YlAIkAtujfhoTw9jPscO3c0stN7ZkhCW/6WDJpO9wGieeFGeNDr pMEUD5R5lYLcsABQ5F5LtnlIWpmvPhG5mtP5sAIeJ/EBOqBVhchVfXFySSV1P9umfGZB vWHZpF5o0scs9Z4j0UT5okgZtpAV8SVuplPXVNnpKRKsKFWk4B2xvYMD1mhlJPfjQxyX VKaA== X-Gm-Message-State: AAQBX9dATVZiU+26Z1/mTKaPo1mCCM3UwDn+Kqq3bsYQjLQKVJoJPxva Ud78J5C51DfohbZBum2mVx/RY9nC6m6sxxBY X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:2286:b0:627:e6d5:ba2d with SMTP id f6-20020a056a00228600b00627e6d5ba2dmr13290706pfe.6.1680203890966; Thu, 30 Mar 2023 12:18:10 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:57 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-5-yosryahmed@google.com> Subject: [PATCH v3 4/8] memcg: replace stats_flush_lock with an atomic From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761821605480233686?= X-GMAIL-MSGID: =?utf-8?q?1761821605480233686?= As Johannes notes in [1], stats_flush_lock is currently used to: (a) Protect updated to stats_flush_threshold. (b) Protect updates to flush_next_time. (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. However: 1. stats_flush_threshold is already an atomic 2. flush_next_time is not atomic. The writer is locked, but the reader is lockless. If the reader races with a flush, you could see this: if (time_after(jiffies, flush_next_time)) spin_trylock() flush_next_time = now + delay flush() spin_unlock() spin_trylock() flush_next_time = now + delay flush() spin_unlock() which means we already can get flushes at a higher frequency than FLUSH_TIME during races. But it isn't really a problem. The reader could also see garbled partial updates if the compiler decides to split the write, so it needs at least READ_ONCE and WRITE_ONCE protection. 3. Serializing cgroup_rstat_flush() calls against the ratelimit factors is currently broken because of the race in 2. But the race is actually harmless, all we might get is the occasional earlier flush. If there is no delta, the flush won't do much. And if there is, the flush is justified. So the lock can be removed all together. However, the lock also served the purpose of preventing a thundering herd problem for concurrent flushers, see [2]. Use an atomic instead to serve the purpose of unifying concurrent flushers. [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner Acked-by: Shakeel Butt Acked-by: Michal Hocko --- mm/memcontrol.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ff39f78f962e..65750f8b8259 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -636,15 +636,19 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * We always flush the entire tree, so concurrent flushers can just + * skip. This avoids a thundering herd problem on the rstat global lock + * from memcg flushers (e.g. reclaim, refault, etc). + */ + if (atomic_read(&stats_flush_ongoing) || + atomic_xchg(&stats_flush_ongoing, 1)) return; - flush_next_time = jiffies_64 + 2*FLUSH_TIME; + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) @@ -655,7 +659,7 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, flush_next_time)) + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) mem_cgroup_flush_stats(); } From patchwork Thu Mar 30 19:17:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77379 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp92221vqo; Thu, 30 Mar 2023 12:43:01 -0700 (PDT) X-Google-Smtp-Source: AKy350ZdXc/kn+gLstcFdqGTv1bcZp9oHdJHB+AyZNsC5IAIFTtb82yaAHEVrjsDnWktYmNxKMVg X-Received: by 2002:aa7:950d:0:b0:628:105:8ef9 with SMTP id b13-20020aa7950d000000b0062801058ef9mr24778931pfp.20.1680205380962; Thu, 30 Mar 2023 12:43:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680205380; cv=none; d=google.com; s=arc-20160816; b=JxDnPrx/anmRClC6Xk7RXxXNtPvC6J2RM0lLlyKLFQKCFhFQPzO7o43b4TcMvloubg W+0m1kq6XgyQgGtO7InbWbow4nFklr2+vuhTtlgqqZ+bseMc43zIJd+TdkEehdl22ZD9 kPMAVWtf+59HMSfQ6XzjZezCZutLmQ/IgqpYfEmkFbBSNDsOisnxW5I5SePN7DgTl6RC b9JwzhJx6hne8QD2lzbCL/XzgtssRwQUGCWSDfhXANHWVMQipTKheGRTPuSZNbzCR+57 R8+cBvTek8N64lP9/+Ar0J8whknyxfjaTMa1EbH0LkOZKClBa2XjwqzWAuyTE0y3dhHU rrbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=4K13XuzFhLt0ZIgNSFirN/NTpinGGuhSsjDOtu88ZFQ=; b=xOFG9neZvNL3AkAKD+yYr+u4k+Yu95N6hWGEgBKZwwXinzGuILU1uTkosCVNDZJnRQ MHJ+7OBTUlEegHHLuFydzV6q0IWFJA1qNtGEGnpFH+PEZ8eGCBENFZK13Us63/TrA3Bk aaHA9DcP/GmavqSJZSG9cxckSy003Y7R8ZdW+YpoZMhqX3CY1Gnw4iJAylefeVDRKuCG Lul3zQBAQPkNxDUpF5NMmHCbbyeTy8L/gy3QHATHUTEQUATVJ0AivQ14QZLIC2JKC1j1 ZveQ1EWq5hU9wmb0WUw9wKYDnQRWOXuEgyjZPqbxDE/pd6K3jFuz2piA2mgSKm6rudKK L6xw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=nHw0e79h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y5-20020aa79e05000000b00626094fcb58si520196pfq.86.2023.03.30.12.42.47; Thu, 30 Mar 2023 12:43:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=nHw0e79h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232203AbjC3TS3 (ORCPT + 99 others); Thu, 30 Mar 2023 15:18:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232105AbjC3TSU (ORCPT ); Thu, 30 Mar 2023 15:18:20 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21AE010250 for ; Thu, 30 Mar 2023 12:18:13 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id i36-20020a635424000000b0050f93a35888so5751261pgb.17 for ; Thu, 30 Mar 2023 12:18:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203893; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4K13XuzFhLt0ZIgNSFirN/NTpinGGuhSsjDOtu88ZFQ=; b=nHw0e79hZFaw3nf4LyIXmmhTYfA1FbBbf2dimkl5At5Psg3CjvNgu6oG6xad78uoN/ S2j2UYV8EipalJrwt2zNvbI6g73PyAp4x9X0MELejGCTtg493QkbzRVplqF8mo+gd+ZT v2g6y/Uq/dc/IcDqjVuG6qTAXC3lltOSedgQENPqcdYx34VNoyr/anJbPmkUtw6k7KGZ kbHnvz3Vd5/Dm/bxM/NXhZiy/XES3eJpoxargQYJd3pDWMStv8K0/vXl6hkWv2MnHLzC WyPbqSXfAwOToFWCyu1C3jP4ndGUE9RCrti/toBLaYuvsaopTHlhvDYChxTi2n0kxppd O1Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203893; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4K13XuzFhLt0ZIgNSFirN/NTpinGGuhSsjDOtu88ZFQ=; b=D54XIzHDwSr/DgwCofBLguaO+2R2pXnv7qEruVUTzJm2NBd7gkmMUtvUix1t8GVvDK IyhQMT+C7rbtPp/e7QEAq5HL/zqQcmDEKD7FwcdoEEm8NbHiq2eaa/eLxJeSYgcjaLs5 P/naZGqoW8hvT3efxb50MeJ37nhuIKkbbe+gCetT13+qJoyC/TU8nMEMX1KgpZA4SwZg p0OFBIMeqjXdf1QWXpSLxPiKSD0+gFYFjyXcxhTyX5rccj9k9UywVaqZUE7+yY6uoglS 4K2K+FtYTjCM8wvQxYgBb1fwlebi9AJZE5o/uksOTqpo1CyLjuHRw9ICYBueUcnJ0XWm sAsw== X-Gm-Message-State: AAQBX9eYfArxaf/NUSB+9bd2IVKc07vNQCE2A8A2oydzx9+vAPSe0/KK gee57ekmGyqB1roITGv1DKYFMYXmKs0JyUbI X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:1914:0:b0:50b:de95:af77 with SMTP id z20-20020a631914000000b0050bde95af77mr2125406pgl.1.1680203892860; Thu, 30 Mar 2023 12:18:12 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:58 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-6-yosryahmed@google.com> Subject: [PATCH v3 5/8] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761823037230546132?= X-GMAIL-MSGID: =?utf-8?q?1761823037230546132?= Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Flushing is an expensive operation that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically where possible. Refactor the code to make mem_cgroup_flush_stats() non-atomic (aka sleepable), and provide a separate atomic version. The atomic version is used in reclaim, refault, writeback, and in mem_cgroup_usage(). All other code paths are left to use the non-atomic version. This includes callbacks for userspace reads and the periodic flusher. Since refault is the only caller of mem_cgroup_flush_stats_ratelimited(), change it to mem_cgroup_flush_stats_atomic_ratelimited(). Reclaim and refault code paths are modified to do non-atomic flushing in separate later patches -- so it will eventually be changed back to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Michal Hocko Acked-by: Johannes Weiner Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 9 ++++++-- mm/memcontrol.c | 45 ++++++++++++++++++++++++++++++-------- mm/vmscan.c | 2 +- mm/workingset.c | 2 +- 4 files changed, 45 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ac3f3b3a45e2..b424ba3ebd09 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_flush_stats_atomic(void); +void mem_cgroup_flush_stats_atomic_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1536,11 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_flush_stats_atomic(void) +{ +} + +static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 65750f8b8259..a2ce3aa10d94 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static void do_flush_stats(bool atomic) { /* * We always flush the entire tree, so concurrent flushers can just @@ -646,26 +646,46 @@ static void __mem_cgroup_flush_stats(void) return; WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + + if (atomic) + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + else + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } +static bool should_flush_stats(void) +{ + return atomic_read(&stats_flush_threshold) > num_online_cpus(); +} + void mem_cgroup_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + if (should_flush_stats()) + do_flush_stats(false); } -void mem_cgroup_flush_stats_ratelimited(void) +void mem_cgroup_flush_stats_atomic(void) +{ + if (should_flush_stats()) + do_flush_stats(true); +} + +void mem_cgroup_flush_stats_atomic_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); } static void flush_memcg_stats_dwork(struct work_struct *w) { - __mem_cgroup_flush_stats(); + /* + * Always flush here so that flushing in latency-sensitive paths is + * as cheap as possible. + */ + do_flush_stats(false); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -3685,9 +3705,12 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) * done from irq context; use stale stats in this case. * Arguably, usage threshold events are not reliable on the root * memcg anyway since its usage is ill-defined. + * + * Additionally, other call paths through memcg_check_events() + * disable irqs, so make sure we are flushing stats atomically. */ if (in_task()) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4610,7 +4633,11 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. + */ + mem_cgroup_flush_stats_atomic(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..a9511ccb936f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index af862c6738c3..dab0c362b9e3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 30 19:17:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77384 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp94594vqo; Thu, 30 Mar 2023 12:48:16 -0700 (PDT) X-Google-Smtp-Source: AKy350YYIOZeWGQfy1ZuPALCVyLR/Z1HW7bsN9PIyaflTJCNxzxGhUGK3QyQYNCoP0IhSgyIMn+S X-Received: by 2002:aa7:c60a:0:b0:4fa:d2b1:9176 with SMTP id h10-20020aa7c60a000000b004fad2b19176mr22349224edq.22.1680205696226; Thu, 30 Mar 2023 12:48:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680205696; cv=none; d=google.com; s=arc-20160816; b=PTc7UlsFzwwfgJ6ZnI2Jb1Wqt8qHeDW0u/e8ztxPIrMRt8kxGUPZeUO3TuQGCQ/+6x DKdjcrfYQpglKPXsBSHreg2XdikDI69rNwsD/8QHi6AMtgvL4ZQ9zmpjv7uKkJQq/4I0 rzXHxMGhmsjUqf3OkX1rcAXIr/6PwWVvgl3hei4l1Muj/2IZ7B8BtXXxn/YLMCSDzx1D Lb97aG6cA7PSMWuNhbq3J2jU8jMFKO3Ix0dKzzraxGjMlgCS/1taI/QL94/6nLK/sj/M ztCx942fr+8XeK4AyP0hokUE1D3tc6j0Ugr9zsdF4nf4rexCPoA7ZrLRa33kWnGyrg78 3hLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=qt4QkPgT/iMy/ixFQJgx4HZWf/NpXrLTSeixAgNXHWAr/abT2m5K5VTRcxTE0qfT3w nGUQSxKDydu8hMQO6Ufjuf8jugei6+hroh2CugUeVVYebz/iwbcwadxjDyX7XxHCR4Ow kF96rMniTTnRtWc7qFKuv3glUSmzNNZFK17pmmokhVhTGo4Q5kpGR5W+FLv9zwRu0JCK n2b8kNJmQ11c+lUdwKQKrcltTH3YBkba469KO2QzOyJjLPbcwHOsGVIMN0BDhhCFuTl8 OfKYxtZPh0t5GUsQ4E3jS9Dx4+wpdetoHFUIvaXazlVd751IqVp/k1Z+3W1Y8wTTXnj1 1crw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=EVT1tk0h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q5-20020aa7da85000000b0050251edd654si416947eds.568.2023.03.30.12.47.52; Thu, 30 Mar 2023 12:48:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=EVT1tk0h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232211AbjC3TSd (ORCPT + 99 others); Thu, 30 Mar 2023 15:18:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232064AbjC3TSV (ORCPT ); Thu, 30 Mar 2023 15:18:21 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A554FF27 for ; Thu, 30 Mar 2023 12:18:15 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id gj9-20020a17090b108900b0024038c818a0so6245240pjb.7 for ; Thu, 30 Mar 2023 12:18:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=EVT1tk0hlY3K/RTYvcQHMKPpwQPVXOnpdSgQM68BKQtZmfmON2APgqvMOvMMFyRoss Dmd2H4VUZwWGIfs6kzYLS/yY0LBdZ4oH1VkvSRWDGcSOJqUREz+4yAA6vO0jGdRbUllx xh0AWez/MaDxogJM7rwIORKSy2FNGdepATzkqraYhiO3RCB8HnWcqe3zSlq35IxkzxtC FV1PJ6neoxu760xZAlTGk8nkupYvKz5UcVjEtp57Ggrbl9Lds+WScYdzag5BG/1gW5Yv DUw8gcccaLuVJ49JXssrnmfFrYddyvv++tF0t2hffuUaTqszDGsJ7ZLQoChqmjJGkrqP YfTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203895; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zjUVlGzaNGh8M5BDppplh7Z2E1z5drNgoNePIuD8ZZo=; b=wFKZaBWFVlwqnl1xgLnVYnQFYNV61omOeeL/TJs6n0bz0XiVh3BIn4wKUsgMoEzTix 5/dDCf9PP7rSw8aQjkO475/iXDa1FBnNNwHdTjARoF6d4AVnw2c5fdl4oLtI8Bxtbbm0 i4rqiQggSz/5aZftLaAe07+2iEUj47gLmAywSFoF3CvEUGRIoR3k0tK4b/+kS1/PrnR3 GmNIiW6ZYtHyE+jN1F6tXqe5mYx+FUXC1XVkSfZnDxKWx6Q8BpdHZcZxZROO3m+HPzU8 p2HAnx1QhRJk4rlN68eK1GLnkCsfvNaYNcZ0qRuYypK+w4mVPl41xvzwfl7Sk6F6XlNN IPoA== X-Gm-Message-State: AAQBX9cPbz6Guvz7+jsW28PrGQDpNAE1yK2PcLUG6niOBMygq+JBDHX2 X2Z+L+sA/Z4Ebn1QH1rn9pmeU5HyVXZ/i2p0 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:6b44:b0:1a2:6e4d:7832 with SMTP id g4-20020a1709026b4400b001a26e4d7832mr2680127plt.1.1680203894775; Thu, 30 Mar 2023 12:18:14 -0700 (PDT) Date: Thu, 30 Mar 2023 19:17:59 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-7-yosryahmed@google.com> Subject: [PATCH v3 6/8] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761823367946294365?= X-GMAIL-MSGID: =?utf-8?q?1761823367946294365?= In workingset_refault(), we call mem_cgroup_flush_stats_atomic_ratelimited() to read accurate stats within an RCU read section and with sleeping disallowed. Move the call above the RCU read section to make it non-atomic. Flushing is an expensive operation that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically where possible. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_atomic_ratelimited(), just make it non-atomic, and rename it to mem_cgroup_flush_stats_ratelimited(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 4 ++-- mm/workingset.c | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b424ba3ebd09..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1038,7 +1038,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, void mem_cgroup_flush_stats(void); void mem_cgroup_flush_stats_atomic(void); -void mem_cgroup_flush_stats_atomic_ratelimited(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1540,7 +1540,7 @@ static inline void mem_cgroup_flush_stats_atomic(void) { } -static inline void mem_cgroup_flush_stats_atomic_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a2ce3aa10d94..361c0bbf7283 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -673,10 +673,10 @@ void mem_cgroup_flush_stats_atomic(void) do_flush_stats(true); } -void mem_cgroup_flush_stats_atomic_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) diff --git a/mm/workingset.c b/mm/workingset.c index dab0c362b9e3..3025beee9b34 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,9 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); + rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +464,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_atomic_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Thu Mar 30 19:18:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77380 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp92591vqo; Thu, 30 Mar 2023 12:43:49 -0700 (PDT) X-Google-Smtp-Source: AKy350YKXmo292GDKb9E333mMKaF2nw7s0rRfkz1JkjXa12fauwWFYuvbA9E/kHGo2tqMrJRY/RH X-Received: by 2002:a17:90b:1d01:b0:234:656d:235a with SMTP id on1-20020a17090b1d0100b00234656d235amr25710572pjb.43.1680205428745; Thu, 30 Mar 2023 12:43:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680205428; cv=none; d=google.com; s=arc-20160816; b=VD0AlTtCVR/gRnRwU1GvXYzOhy+nZcAMLAmVLkuumhJt+uyjlOOiZzIJQMZM11ca7q hpKmG50OKdAHaWVcfWToZz2ubkaug/zhswDkxaNpQKR1hC30oXZCq/ZtIqGFcQYM90LG mz1kY92kZukPxfKyGi5uW6WHccFWHd57pI4vIbp0nsswkP72ZTGmyC81IHhz7O1jiul0 4yyBN6bVOeAKa4jAXK6UCBY4/zbRdIFGHiOxO8n4Qw88sZCdXI4bhKun7q1DUfn2hfeW 1eCuyQTfPgilj/5cxLMfaGNgAGMVxOK07za8RPLoyEE3hKp3zrWi8xLG/2FPU78FG3TH cOaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=xcUaqJ6BFfuwa7nRJSvBlAwhhxeRf5mk+29EPOCHz/Q=; b=UELL4/0YvcKBaoGXfvBxWoozuw5jECc8jX87822sSZG/MGs/kk6bKOWTGEDgm+SDrk RgfknjedTkr346JfAXGAxG/fqZGfy9Yi1Zt/6RS0WvAbSkDJvx5SQE/6pDV7V9PoSIsh gBhbabtg2r7ekR5nTPR/G7SzWxBL8S5XMnrfc/5mMdFRpNV2kwbiNtST0dUZ632b6PAA lHKnQ2/u7uf+MhjV0QMH2kUuPzf/WoMIzi2xRwMiJsc6VMWhpLi/8n8BgpB++umTtKvY //r1Au5/y7lUXL09+R3LT3398kloMu4u8FHSeEOf7D0rgBFIBEe3AaBxqamAG+Klv/YC /VfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ltNePRXu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bv70-20020a632e49000000b00513234112b7si330251pgb.899.2023.03.30.12.43.35; Thu, 30 Mar 2023 12:43:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ltNePRXu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230429AbjC3TTH (ORCPT + 99 others); Thu, 30 Mar 2023 15:19:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231634AbjC3TS1 (ORCPT ); Thu, 30 Mar 2023 15:18:27 -0400 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E21071043D for ; Thu, 30 Mar 2023 12:18:17 -0700 (PDT) Received: by mail-pf1-x44a.google.com with SMTP id m12-20020a62f20c000000b0062612a76a08so9219050pfh.2 for ; Thu, 30 Mar 2023 12:18:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203896; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xcUaqJ6BFfuwa7nRJSvBlAwhhxeRf5mk+29EPOCHz/Q=; b=ltNePRXuZIePvYweAMHUj9ZiAeC5qavTrptvs7HWUZy4ZQjHtls0D9eRHEg0pqkaqA 2rj8RSL+AK/a3iTAV1slsfA/XO8i87DoFdpL6M03j+wstTxD0gNLM1Eo1e2c6UpyLkfA 02M3+F6NXEWgNHJDDpxLzGC2sGDBHCNs+LO2yZgTc79ihpz2oe9Gkrj2oQSZRb8nP/pk DTjpgFJ4LNEzmr5tCkDpR0cOQrENPev1JFpwtylRIUOchlXAi5bgsrxJgKXBgxobydtr bGBrN9gUtdaDKVSRv2j9cJxRBN0C3BLXf8CobNS0sZxfbCzcE+BakwfvTig0jcnVMYxC dA2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203896; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xcUaqJ6BFfuwa7nRJSvBlAwhhxeRf5mk+29EPOCHz/Q=; b=XOpzbzcPi8KzoK4Jn7QASMRoVqxrSfp+OO3Ntwoh7C5jjhYQ29GMHFrXwo299xIRDe i8OLYMHVosJNvuGk/OvUWm3Z6qcE0M4PpEuEH7bVXOjRVUQVSp6H/9O+3NJ2b5PiWRwR 6FWoWGmvY16a7t9+WSjW0v/WIx1nAGUDyLaVVsQ/j5p7PStTfcInBch2rXUtt+g79mv/ gxIlgbFMJfrZmsrP+5sWd1oTvKzgieeadBI+xKhWTHJdAo5sY0F2WStX4/PLjTMaYBsf 92FaMFvGRUJuKmE9hnSJ3Pgxx2M0HVymUpEXRlr432Z4SdnrjMHAb7PsbSJMK/zeSWM9 f+cQ== X-Gm-Message-State: AAQBX9fteGM2mU03DP8zCKUrt3O5acZyqs9Uc4/wpmnz/QU347szdkqR OOVm1A/5ihcxIuep1Veosa2p8kNns3HXFEU5 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:40c1:0:b0:4fd:5105:eb93 with SMTP id n184-20020a6340c1000000b004fd5105eb93mr6667473pga.3.1680203896567; Thu, 30 Mar 2023 12:18:16 -0700 (PDT) Date: Thu, 30 Mar 2023 19:18:00 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-8-yosryahmed@google.com> Subject: [PATCH v3 7/8] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761823087236964755?= X-GMAIL-MSGID: =?utf-8?q?1761823087236964755?= Memory reclaim is a sleepable context. Flushing is an expensive operaiton that scales with the number of cpus and the number of cgroups in the system, so avoid doing it atomically unnecessarily. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9511ccb936f..9c1c5e8b24b8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); /* * Determine the scan balance between anon and file LRUs. From patchwork Thu Mar 30 19:18:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 77378 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp82099vqo; Thu, 30 Mar 2023 12:22:12 -0700 (PDT) X-Google-Smtp-Source: AKy350abZOc65Vl+j0v34uqQqtw2zzzuwb8YLJY0zx6lyuU4RtEuN2QU252tpLHHgeiRtzJCAtF9 X-Received: by 2002:a17:90b:388e:b0:23f:7d05:8762 with SMTP id mu14-20020a17090b388e00b0023f7d058762mr25928598pjb.23.1680204132127; Thu, 30 Mar 2023 12:22:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680204132; cv=none; d=google.com; s=arc-20160816; b=0GDsUi5bhQKLGD1eyUAMSRF8KtAzOJH1wv1UA7rDrD/9u/8/Xkcz/Qb85mx7gDwZss Se64cownAeD+3m9e9myaxmt6fGn4mBx5r8oRrUBaMQKXfqtex7bWH9sunCgn5UDQnqU5 LbNzQU+rhJQfNhvVJ9SJG+T5N9D6VHEiv9PWHXEvsrtsIZ79y020i+iUnNsFbUjq+0zL /iVfTQ8pso5/08fF22pQxUHEP4glPbocdPK/bm/0RfanspBTanh4YpkG0qeOzzlUQaZO IHtn8xNCeQco7GfiiB9KX8BTud9dxSVtZRoIIwLBJJEGwzv3rYmXooCDmCCDpupvNYWO XuAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=WbNC07VGnJucfMQdaVru/HWxeCTMcNM6hFkgQ/MCeFU=; b=IntlrJxtKu7BM9qPsb+hJo3IpC7p2DXO7a7T7aztAjGXl9uxy0G7o05wgQw7S7a4ig 6EDD/zBcM7X6LpG0NhBzpkIEToX02AulenQ3RBUvBXxNAETWzs0RbIep/MkbFKp+g+Qt IBXzEqSWxl7/grSpKutL/9CnZRQDuNNNSVGB4TxaMcs7uE/WJFlxFasE307mSgtn/VVl c2OgdlSdmdaX1oAlFE7o6AloXqi/b2kPio8H97Mch4qNfKRbtEJ06y6ZbH+JLE2YWzbw IRmOER+wMoOQ9/FL6ZoHzrMt3d4KrjKa7+WkrxN1gUnNck0pvRwKgFESDY6rLIkBsS2i sb6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=PUv9MA0n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ls7-20020a17090b350700b0023369af18e2si113376pjb.72.2023.03.30.12.21.58; Thu, 30 Mar 2023 12:22:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=PUv9MA0n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231876AbjC3TTJ (ORCPT + 99 others); Thu, 30 Mar 2023 15:19:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232224AbjC3TSj (ORCPT ); Thu, 30 Mar 2023 15:18:39 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A235410259 for ; Thu, 30 Mar 2023 12:18:19 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-536a4eba107so199127387b3.19 for ; Thu, 30 Mar 2023 12:18:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680203898; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WbNC07VGnJucfMQdaVru/HWxeCTMcNM6hFkgQ/MCeFU=; b=PUv9MA0n+Z8fm7ErrGvHySjECm+JmVMmb7yIewlfjR/61XxilLPkk2HWdbFMTfUOaL VEd5S3Db2lj6fZaM1EVkiIBt/SsoMNdbAydQDGkFf9C+290T1e1uwhZ4tZ9DFf8Phq0h P4OVDvENTCX+3IndK3Cvs6OkRD4foG1c4Yew7PAGOLNrfjOHkmrszhM9+d8Istzt+wK/ krUK3yorRZOqS1BM/g9ye1FRB5jRNaXtQwUxDYcXcRNeFj0C93RhkVtRFak+8MvkC/U9 MW/LMGQgSZbJ8G2s7XbIxfyCgArJtDyQ9vNCBWNE1utNTCLeMAHTcanEcYFbU2EbD5hm feog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680203898; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WbNC07VGnJucfMQdaVru/HWxeCTMcNM6hFkgQ/MCeFU=; b=tOz4v1yYLzEzlMUzAOAqarTvkyysRXrn0DkGvLvp4KsUJr4a/lEUlPuJedGq3A20tM CiuupulDq9kngdFtAzoxMM0eyxrBhGkv5ysL6aNiTZ0ZbBrbjeiCM7HiJ54hHGcZtmT8 LQnuhEkJP2+EBtDfWigvgggk7eUOErqAN59o1cjz5YSqtI68WTAL8V5e4O0OU1vJN5tl EwITSFyH/yhFxpPb4Q8ssxYUjOY+GriuZu2a6o4ULD9PU1X7DRS8O93MFVCzsufHl0Pc zjVQVD411ndepVqoywEzhs8UugxWVuC1SutfpXE7xhwOQpodwgCMVevHkpyb87UaAL9b M3ww== X-Gm-Message-State: AAQBX9et5wStuWm3CuZTeLGVkKBnQZBieK+t2HHniP39sCUteQz+HErZ RTL7dJvsFZOFA5UksfMRppNBAuWMefl/yiFq X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a25:12d4:0:b0:b6e:d788:eba4 with SMTP id 203-20020a2512d4000000b00b6ed788eba4mr12664999ybs.6.1680203898630; Thu, 30 Mar 2023 12:18:18 -0700 (PDT) Date: Thu, 30 Mar 2023 19:18:01 +0000 In-Reply-To: <20230330191801.1967435-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230330191801.1967435-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230330191801.1967435-9-yosryahmed@google.com> Subject: [PATCH v3 8/8] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed , Michal Hocko X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761821728258293069?= X-GMAIL-MSGID: =?utf-8?q?1761821728258293069?= In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko Reviewed-by: Michal Koutný --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 361c0bbf7283..a63ee2efa780 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));