From patchwork Tue Mar 28 06:16:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75846 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2004884vqo; Mon, 27 Mar 2023 23:23:01 -0700 (PDT) X-Google-Smtp-Source: AKy350aRASJW2uOWNtasTiFKxnjkqR+TfekUaaRVNL+qaffAcs7uZgG0wWPFb95BOEIB7iHLPWIV X-Received: by 2002:a17:907:cf8c:b0:93d:e132:9122 with SMTP id ux12-20020a170907cf8c00b0093de1329122mr14197940ejc.35.1679984581474; Mon, 27 Mar 2023 23:23:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984581; cv=none; d=google.com; s=arc-20160816; b=UwfhZH5nGohUSRy7UZhh1LT1wH1vOHi/O9oz6v+dxvQ7lHojaOa2A6wCATmK19ihtf tBOKRL/EcEOFxaZak/IOvP1DheBl7GJyONXC879oXaMPDRabvdxnscf00ZxyD2IMLVXp uNztzUsNwKnj8LL7vFQ/fh4OZ0QjZtv7R+6sV1SUinmlUvJnTW33tdBgnSNEifz5gjNQ m9xTvBn+h0uikwZUE3wmcc4Zx8dZhymSbMyniGrgiPPSzRIPR1l9fwmL4LbqWHIab+mP 9CFxqwNsOCuku9vsWOMKQvJWImy8FT4nXya7FBGIS/vfW3GgRDLMVBuP4SdxTGWm1L9u fk8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Cs9g98LfB/5B5fgCHg0khEoDDLXk/FbX9zQPYHFGol8=; b=fCoUzNdtuASCFNQsFcF/4mwkaDGVbm0HnEf6dz9OWL8bx/B81OIcSKZIKvYMsf5VKT lAjLhXJxkBD6ZOIBvpCgbhoo8Z3LXXyamlqikZUnqtkfD/mzb4vavkGm8nNXq9TmjIdo vjDija2kJ65/oXzOKOkYybCr1e/yQmkEuIfdO/rOlCWvB3HrGmgoxfzTgy6sD8AebdS8 G52mq8aMnOxlngT0dhDzLUnsinqrzOvLReIrIIlX2YZsSSEdkJ42Da1r2HZFl5JajrRA UooJBE0GSame9m2Mba+nux4DAMIWyJfhxMDVLof+5eg+STrVPLr5vyx6hxhH5RCZ/dwA Vy5Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=IrEj06HW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sb6-20020a1709076d8600b0093b5bc4c792si17466163ejc.749.2023.03.27.23.22.38; Mon, 27 Mar 2023 23:23:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=IrEj06HW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232464AbjC1GRN (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232452AbjC1GRB (ORCPT ); Tue, 28 Mar 2023 02:17:01 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4AE3D358A for ; Mon, 27 Mar 2023 23:16:44 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id s4-20020a170902ea0400b001a1f4137086so7101580plg.14 for ; Mon, 27 Mar 2023 23:16:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984204; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Cs9g98LfB/5B5fgCHg0khEoDDLXk/FbX9zQPYHFGol8=; b=IrEj06HWrJaiEe47kle+4mjwycO8VkK4UYyVi89hOr/BT4ba3YoJaIkMxMmDu91Azt OCiIdpmqw/gfkGEUiinzv01MOm68VDd/zLRgX2gIU8f7UybpdpXtpixpCNAld7gzowZR +gyt1jsqLawCKRD+ojRnZ6c35iauByhdICeE6tA9/AW2kV9h46wsE+qITfJzQSmauG1W ySDyn3mwIkhUXYNihBaZeA4rjBJGjlP0HSdTCoy9A7k1vdMTk4m51J+idABUhgX1bKO7 He48ZTeiLxeRfpruo08A01IDO1BpVxjEsw+pmwGtbL4byIuTDU8yveD/qiON77jThzUh Vo3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984204; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Cs9g98LfB/5B5fgCHg0khEoDDLXk/FbX9zQPYHFGol8=; b=630mZVewJORP2fNKdtiQeFxnWfCc/o8iyhBDsR3Z/f1ZbNN7k6GkrivHt0bsfR48sZ ud0+FMY0CERbrdrtdsqV0PQin+84xOeKvKsFNHH0ptwLFENZ2QCX1qKjTQEnFl31kMHC oSTlTZ1JFZYn0czx78EGFWbyPN1PmyUXo5irSDi5CSRr+bXU8WQW0R8sEuIiISKpZho1 KnP3x29679h5AsBHJUImGRbtQSC4cqC4cNqYTr5R0ntqWMSh6LzJE4ZsOOjDCCdVEwo9 +gymlT6YOXCy2FqjTrqaPbE3vprO6sXbPq0J/9XCMeFflvFAQmkK7l97+ZnmAZn+/84T eAZg== X-Gm-Message-State: AAQBX9c2wgi8/jFbXmF6WG1RKdE9pCns6GPeIZlyZ+0GtDMICPt7pC76 83yJXGjI1DOgqacC6u0d9zFdDiSCqiBX34lv X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:24c4:b0:629:f9a2:64e6 with SMTP id d4-20020a056a0024c400b00629f9a264e6mr7565323pfv.4.1679984203700; Mon, 27 Mar 2023 23:16:43 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:30 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-2-yosryahmed@google.com> Subject: [PATCH v1 1/9] cgroup: rename cgroup_rstat_flush_"irqsafe" to "atomic" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591512466038904?= X-GMAIL-MSGID: =?utf-8?q?1761591512466038904?= cgroup_rstat_flush_irqsafe() can be a confusing name. It may read as "irqs are disabled throughout", which is what the current implementation does (currently under discussion [1]), but is not the intention. The intention is that this function is safe to call from atomic contexts. Name it as such. Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/cgroup.h | 2 +- kernel/cgroup/rstat.c | 4 ++-- mm/memcontrol.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aecffdb4..885f5395fcd0 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -692,7 +692,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) */ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp); +void cgroup_rstat_flush_atomic(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); void cgroup_rstat_flush_release(void); diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 831f1f472bb8..d3252b0416b6 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -241,12 +241,12 @@ __bpf_kfunc void cgroup_rstat_flush(struct cgroup *cgrp) } /** - * cgroup_rstat_flush_irqsafe - irqsafe version of cgroup_rstat_flush() + * cgroup_rstat_flush_atomic- atomic version of cgroup_rstat_flush() * @cgrp: target cgroup * * This function can be called from any context. */ -void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp) +void cgroup_rstat_flush_atomic(struct cgroup *cgrp) { unsigned long flags; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5abffe6f8389..0205e58ea430 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -642,7 +642,7 @@ static void __mem_cgroup_flush_stats(void) return; flush_next_time = jiffies_64 + 2*FLUSH_TIME; - cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup); + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); spin_unlock_irqrestore(&stats_flush_lock, flag); } From patchwork Tue Mar 28 06:16:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75849 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2005382vqo; Mon, 27 Mar 2023 23:24:26 -0700 (PDT) X-Google-Smtp-Source: AKy350Y6JayOkDN7N7abY0l8WJmCbG1vhi/WNhrtse/u7QXEzGmt0CUSKV68ew/FQcfEzkBoju4H X-Received: by 2002:a17:906:b10a:b0:933:dfca:1716 with SMTP id u10-20020a170906b10a00b00933dfca1716mr16419827ejy.26.1679984666474; Mon, 27 Mar 2023 23:24:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984666; cv=none; d=google.com; s=arc-20160816; b=iRjq78IBWretmDqIq27d3Pgq3sY+xUtyWciDTmUElYQOIPchE7trY412/a1vcZPwm8 f6JbfO1A+AX+QubvcHnrUPocvxMwpRcOn95kFwVkXpVFhVZXje7MCuVV8H0Jh7bk+pRd 9+40uWvbQpakAdmRPIttosWJm76rAb8ZZJeC3TVVL1eeYNd5DLlvsdcxzaopAQKokQy9 bcGfqcykgP8f0aG6g+5qZmmBXkWM3pZfcNHRjsd3GugijBd3I/mstrWRjqt0krgCTWNF wCp1ySLx8a4/BMg4BKEG66EQ1nVF5mnVVUR1ASyhhVzM5zWm580sPuMZrhmorDJq8Gw8 jK4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=wKfUWPDFjTfCFAmAu+PDSXoVaLfaEZa4F2xpA6M98WI=; b=vOuI79Xda4Mwb8wy5uQxi5Jxg4TEfvl7ww9sWNPF7eb+JE5bp2TpQuEoDHgeT9e6Ue 9ChkGh3/FGvWuZCqu0V+A2rq53qYEoW2CMrezOG1ct5TEOdD+zCaFDJQ8qMHPSVtSeo1 rkF75Qiya+Z9Ei9sNbDycBA0Y8D2VFzJxG8rw0lj7n4Lt+UqRsdM8tYLrEmPwDliz//Y baqtL5q6fhYaUjqTH6PLWZx84eH6UlaMu8FhmLeaBn3SwOhAx5YcK6XqUaRpfTacTzCd y6ERTyL/T8WNhq4kmWZBbMNhdWapCJU/NVjcfwxQ912KWVlFa7jFcwHxGyF4sgM44QoI viRA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ML25Z5sJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hb30-20020a170907161e00b0093b5bf364c1si19506227ejc.987.2023.03.27.23.24.03; Mon, 27 Mar 2023 23:24:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ML25Z5sJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232588AbjC1GRX (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232400AbjC1GRD (ORCPT ); Tue, 28 Mar 2023 02:17:03 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12D3E358E for ; Mon, 27 Mar 2023 23:16:46 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id h8-20020a654688000000b0050fa9ced8e3so2943743pgr.18 for ; Mon, 27 Mar 2023 23:16:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wKfUWPDFjTfCFAmAu+PDSXoVaLfaEZa4F2xpA6M98WI=; b=ML25Z5sJR8k5t8kSQzNLjGMjEKmdcebxHFtYwKZcrTLZX2wK0gMrUstJ28IAfa/l/B dmJfAqHqeUNGDMqpjHMcUn/3rl6ZtTwoLtJc4kM6PKx7oTSnQD1aK8fu6wcMuvNpddNP JXz/aPGcRNjn7CdP1wNYEAXnpd0Uj2w+rzBilII9bcn8vsNoKap2/emH97XB9yYmQX0j zWlTQgN/LRjTLn7LSTnUMs5Zi8YiSMdP4rhUt8H9KINdPVTgc9z/VX9FnrviFmsMUsqK yY+aBb6I7/+gAXbDQAsJSzcOoGzk7Nj4yOSGJy0Va4z4/Vp/gmfRCs4LrcZySZjB+aj0 I08g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wKfUWPDFjTfCFAmAu+PDSXoVaLfaEZa4F2xpA6M98WI=; b=DdetBZHbVqzutMpzWvM5vM8MedSQ+0h/Hu12mhIi+otrmrbATRUtwdU37OsU79BInB c/gZK+zEu2TlnB/j1Ue/PBsgvBG5o0cmRLkkAlXMSUrXMwvLLYwRJjpBvKdz7UecgBmb PWQGeuM9O0dPCeN2+esHfrfk0xOrGqqz1rPXZ7GGFRvar7LuQKv+/xOR1DTZy+LDxMns 37Ci71zFrPdXMgVZGuaLxljeMecvMtXMDMOa6dfax4y53Qv9C1F9iyr1eL3V4Z51IUG8 kkWaDtYqiZXOqv1WYCt+BUAQuWGLJSfChpfo6QzpbfC8FCtWxGcoiPu/lHSaX+E/RrEX Y3KA== X-Gm-Message-State: AAQBX9dX3LxAt+JOqbW28Ha60RxPQ6xSLupxHeasyd+irqyddy3gopp/ /lGs2w9+Unmb/GpRLsNKwWCt0/WqPyld06Pt X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:902:7088:b0:19a:5953:e85c with SMTP id z8-20020a170902708800b0019a5953e85cmr5289230plk.1.1679984205498; Mon, 27 Mar 2023 23:16:45 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:31 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-3-yosryahmed@google.com> Subject: [PATCH v1 2/9] memcg: rename mem_cgroup_flush_stats_"delayed" to "ratelimited" From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591601603386076?= X-GMAIL-MSGID: =?utf-8?q?1761591601603386076?= mem_cgroup_flush_stats_delayed() suggests his is using a delayed_work, but this is actually sometimes flushing directly from the callsite. What it's doing is ratelimited calls. A better name would be mem_cgroup_flush_stats_ratelimited(). Suggested-by: Johannes Weiner Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 2 +- mm/workingset.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b6eda2ab205d..ac3f3b3a45e2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,7 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_delayed(void); +void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, int val); @@ -1535,7 +1535,7 @@ static inline void mem_cgroup_flush_stats(void) { } -static inline void mem_cgroup_flush_stats_delayed(void) +static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0205e58ea430..c3b6aae78901 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -653,7 +653,7 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } -void mem_cgroup_flush_stats_delayed(void) +void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, flush_next_time)) mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index 00c6f4d9d9be..af862c6738c3 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -462,7 +462,7 @@ void workingset_refault(struct folio *folio, void *shadow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); + mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 06:16:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75851 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2005833vqo; Mon, 27 Mar 2023 23:25:36 -0700 (PDT) X-Google-Smtp-Source: AKy350bMMRM/QyVBC8kXye77S+U7QsHh1fqFVbOIArZyVKfkZrfyufITpE4MO1XeHEd1pRFXY6NZ X-Received: by 2002:a17:907:1819:b0:931:dcd4:4ff0 with SMTP id lg25-20020a170907181900b00931dcd44ff0mr17651119ejc.75.1679984736673; Mon, 27 Mar 2023 23:25:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984736; cv=none; d=google.com; s=arc-20160816; b=DejYeI2WLbYUaYa0yVevUHL/4Bd6WwmW8q7c3++zZqoARo1VI1dxfoQnJ4RJtSxDD3 PvCGeTBNX2SwU17lb5JygGjkR5m+9uIspgidSodw6wqB2pvfg9L17SdUWFGcx1dj3A4V E5rR2mY9l6LkshxgQ4awaDbYmC/z16ysw8R/XrlkV0FhmnrrEvkYi8cn/86A+Uzh7o7p EL7pwMrXcN8i6i1e2YG7HvdbF9w2Bb3TBMEVj62rpVc30LEi7sU+vdzXmfYru5ZoWpEE EhOB1n39Nqioo3miYjxKPqiuNcvwmeWBHa3Ow7hxvdQqgVHANw25LUvliL8Ap/EKf9Hq qHmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=wILXj9LqWwom+6PQC1+rRSpOfUrkR+GzdkCDGSSBhGY=; b=QhLaoeoTq/39aLy2sghBNf/WZWzGW8Kw1WGx6Xa6y0O4Y0RpfbQi8ITF1+7UOP9o10 JW5uVqI+P3Y2itKN7LF2vyh3ARfvhuq+IA575O2c/hAnAR4ZmXrQI8C/e2jngRApfe9r l2YMfggalHtv+hEA90jYxjPVdUYaQGy9xx0Z13SrcNdVZzNtNxf6G8F6WGzooGyMX+1G KhYfmN2GS7Ekkf7CNoOoUZ+mH+EOY4K83byKYBNyztQN/paiBcTilWqCfWhtuz1pMjB/ 7PCNDtpfkQhTbfAAWcOdppoQ3y9iRc0cbBv537qfrPvcUxh0ir9ApigTpy7L5/Da7bij ELgw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=sq4WxgrX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r8-20020aa7cfc8000000b005024cdeac4dsi1963627edy.70.2023.03.27.23.25.13; Mon, 27 Mar 2023 23:25:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=sq4WxgrX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232620AbjC1GR3 (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232475AbjC1GRE (ORCPT ); Tue, 28 Mar 2023 02:17:04 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9CDF3A87 for ; Mon, 27 Mar 2023 23:16:47 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id m12-20020a6562cc000000b0050bdfabc8e2so2931954pgv.9 for ; Mon, 27 Mar 2023 23:16:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wILXj9LqWwom+6PQC1+rRSpOfUrkR+GzdkCDGSSBhGY=; b=sq4WxgrXs35T1kNaqttQxLQHBBfHPzw9iKcGHtZRmWn7KoDZlSxKdNufMOgLyKUFr+ TMiE/wqtjNlMTixUEQeY/Vrw6MeR14OOP90t7VeDCccykNrGNGWLRshNs7Tr5dzPpQxM EqNAZipGy1I27uXZW4pV5ZH3WWGxC9s0suXWrK0/Qi+Fx4TfEA0shsUjup5r4I0T4jcO otnYwmVT7RvIIuwqMJTULG/AsEi9P5/3lltCZIuxMi9242jZnrgGHko3vAN7pXgBxOSv 6bmFPpVjK2j81kXxaYSwcWty18LJNFnzIpVeFZG9+e6GZQd6MIhYJNlu+ckzRRvaCz8L Rlqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wILXj9LqWwom+6PQC1+rRSpOfUrkR+GzdkCDGSSBhGY=; b=I6vG682qT13m3sU9ngFFJrgMtnr0BOK/QOSPrxMY737/6dupNDH9MpOdxHyVGoUu3n 7hFTfumFGFHLw33WtT2FZ3FN3ivFzN0XZ/zxdAH6WS82LN2jUV6zoCYQAmbPrxtbgwO6 dwqkKcKSJnNMVIXrUqWCS49YrCqSSJb3+YhqrJvLrhGceDM4e9VMtl32vFt6np7kfaOP BL0tA+qp2rLrP6HdMriqJB+z87Y8/YF5whpYAy301M45bc99CaMjNn3AfJk1CURtv5Le Z31LvTcCvbSLlII7gdl4r0z+xMujhVzB8iL1lTqBazZwVEvUpKNxfEIZ2LwGJNQy+hYZ VV9Q== X-Gm-Message-State: AO0yUKUJXsbsIvULgpRXDd3o7Jdb/Gec+hnuJt0TyaPkhYa5pvi4GAxf h7Vey/cYtYY8BoPYG3bDD6w9L5qmNfhP0455 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:4510:b0:626:1710:9b7d with SMTP id cw16-20020a056a00451000b0062617109b7dmr8924786pfb.0.1679984207461; Mon, 27 Mar 2023 23:16:47 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:32 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-4-yosryahmed@google.com> Subject: [PATCH v1 3/9] memcg: do not flush stats in irq context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591674940798879?= X-GMAIL-MSGID: =?utf-8?q?1761591674940798879?= Currently, the only context in which we can invoke an rstat flush from irq context is through mem_cgroup_usage() on the root memcg when called from memcg_check_events(). An rstat flush is an expensive operation that should not be done in irq context, so do not flush stats and use the stale stats in this case. Arguably, usage threshold events are not reliable on the root memcg anyway since its usage is ill-defined. Suggested-by: Johannes Weiner Suggested-by: Shakeel Butt Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c3b6aae78901..ff39f78f962e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3669,7 +3669,21 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) unsigned long val; if (mem_cgroup_is_root(memcg)) { - mem_cgroup_flush_stats(); + /* + * We can reach here from irq context through: + * uncharge_batch() + * |--memcg_check_events() + * |--mem_cgroup_threshold() + * |--__mem_cgroup_threshold() + * |--mem_cgroup_usage + * + * rstat flushing is an expensive operation that should not be + * done from irq context; use stale stats in this case. + * Arguably, usage threshold events are not reliable on the root + * memcg anyway since its usage is ill-defined. + */ + if (in_task()) + mem_cgroup_flush_stats(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) From patchwork Tue Mar 28 06:16:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75843 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2004558vqo; Mon, 27 Mar 2023 23:22:08 -0700 (PDT) X-Google-Smtp-Source: AKy350YaK+kN2+yyF7evJzcOhpjtaSNp3POUBIy8ikI/lFNCiae3bxur4t9Xj4RybTCnZ+aK0027 X-Received: by 2002:aa7:dbcb:0:b0:501:fd71:7e18 with SMTP id v11-20020aa7dbcb000000b00501fd717e18mr15163123edt.2.1679984528121; Mon, 27 Mar 2023 23:22:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984528; cv=none; d=google.com; s=arc-20160816; b=KI8I5s58zlWQKO+B39IXFzINgC2mHXBrSnLrtdU31bArF8U526zZGEqg94ziAZI0Uw eHA0tlHChScqhvIAA+1MufI+TkXElQ0d5UiJn4MLgg/maKOTIm4Xyk40OynHNOuJzVGh MLob0z7cfnhBaQyp9pjrWIALdjL57TDuRMNgKI1Z4RQ+hY0tpFDyTP7isw70vnJSzPjM xsoZQTZulJdabacMgowm77FecGCMfZQd8iVzk2amYkEXTchk6ftNsexHVHXbWurbizZD HFiBBtuAOZYeOAHtG29xQ4g+EECfTZg8evlcZmtD/3Vf442b3HkQtoUeAGrYo27rXD3V 5tcQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=+vNiceMoZ2PdzrNG/xVeRzEo7O+ovCZNHcQMp7VClk4=; b=hZeFVm1TlFsRnVdJPuo8oeZU01IWal58g9MzhzTxf8NUmsZIsptjTXb/GD4izVXj82 umRPR35+T9WN0pFQODEwMMh4y/Lk+eZM8RmoRacwIydihwq8DKQWxxZhi2+23ucSz+jy mvEfAlPOuUrqNK09B9mM1hNwN3Z6UZMTe1yHGMjeOucpJ3eLApTsEhbCV07HJi0XPWeZ xo96LRLAtJAOVZ+L4Ymrlen7sWBIdqO0Zn8gOovzsrRziWaQKJIUwxIsy0zy1YhHQD4r l+J16qEDveDQR7AnFKNR/4Q+SYoELE0A8gSetQbgKEC7bRmCsRaTmFphRmNvBNXlCkRD bf0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FtVms547; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z5-20020aa7c645000000b004fc19199fd6si26956509edr.143.2023.03.27.23.21.44; Mon, 27 Mar 2023 23:22:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=FtVms547; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232625AbjC1GRe (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232502AbjC1GRF (ORCPT ); Tue, 28 Mar 2023 02:17:05 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0CC835B6 for ; Mon, 27 Mar 2023 23:16:49 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id q15-20020a63d60f000000b00502e1c551aaso3002136pgg.21 for ; Mon, 27 Mar 2023 23:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+vNiceMoZ2PdzrNG/xVeRzEo7O+ovCZNHcQMp7VClk4=; b=FtVms547YGVdf9XAqjguoo0WST2Bomcq4pMeCW3qlg0ksZV5swPoVz0GmrxFu8vfrh HBzgJptVGGKy7geKpBPvC2PsWo0HmjT2F27i9dbdVA6zCF4EYemeZRNrh89DbSPqLObX h82tbSvZ/WCeLjLtii1Meb+k3BLVneGl+TdsHqmfTXT/vAFZIC+/S3Aw9k3ElSrpUy7X 9xgxQRUTlE19S9rQfATi59HRnf65TEpEpviTmJbLID2U9/qvSayVfcGFQGEl03zvUEu5 uDhGnK3XWBy1kJngfOB04Odcd1UeEO5WA33WjKyVDhleLQy8k50zMQSY0RXklX82o1Jn CmFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+vNiceMoZ2PdzrNG/xVeRzEo7O+ovCZNHcQMp7VClk4=; b=kK8uZbg+oTUmTeBmvVzUVdy6zltfhI3DEFToy5S6M/JHn5FIpxHBwFt1upFEvHfsIT jshcRb5WjddkuxwlmFt8nKatoq1GqrBgxzeQm7k9seG+Il1wjvNL9OPNrHiy1ecmOu37 eNgK8XHO+cDCHwyY7K8sK4ST6eo5JsKGviOe7ygz3oPAdiO7xSimxXjH/O7iAt9Ldcaq 7VCtGNrSMjGsTsooUcN2OC96pDb6VSjWOBNNvAcCxdLS5JURdcLTE4vooYkLhYLBNdSL UpFGvu7JRQhNv2KcQcvnV5DZSeeqCJvwrKsj0sBzEpNKXdb0tUbhK9g26Vrgg5npWZsq Tkiw== X-Gm-Message-State: AAQBX9e8SZ8OHEX4bVAYpG/M7glCdUkbh0YI/c9MfUYyQs5L+AWkvQeQ lZp11N31zOqDrCMdqfZKnoirl8GzxYYQ/JaP X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:1a46:b0:625:c7de:48c1 with SMTP id h6-20020a056a001a4600b00625c7de48c1mr7269994pfv.4.1679984209151; Mon, 27 Mar 2023 23:16:49 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:33 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-5-yosryahmed@google.com> Subject: [PATCH v1 4/9] cgroup: rstat: add WARN_ON_ONCE() if flushing outside task context From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591456371677306?= X-GMAIL-MSGID: =?utf-8?q?1761591456371677306?= rstat flushing is too expensive to perform in irq context. The previous patch removed the only context that may invoke an rstat flush from irq context, add a WARN_ON_ONCE() to detect future violations, or those that we are not aware of. Signed-off-by: Yosry Ahmed Reviewed-by: Shakeel Butt --- kernel/cgroup/rstat.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index d3252b0416b6..c2571939139f 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -176,6 +176,8 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) { int cpu; + /* rstat flushing is too expensive for irq context */ + WARN_ON_ONCE(!in_task()); lockdep_assert_held(&cgroup_rstat_lock); for_each_possible_cpu(cpu) { From patchwork Tue Mar 28 06:16:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75841 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2003289vqo; Mon, 27 Mar 2023 23:18:52 -0700 (PDT) X-Google-Smtp-Source: AKy350bQ22rPMP4fwQYbd+xjHqzs+2MxO7ZCtjzMUsRgzFryxvEPn8B8e8w3CO3VxJ7+OkWn6w+5 X-Received: by 2002:aa7:c90d:0:b0:4fb:2060:4c20 with SMTP id b13-20020aa7c90d000000b004fb20604c20mr14222208edt.31.1679984332470; Mon, 27 Mar 2023 23:18:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984332; cv=none; d=google.com; s=arc-20160816; b=NqlTA4sbx22vbK8ivbJ35CAkrdzxGVUcj9fkIoodJKUQDlNstwO6doZe3mCQgPKQTA L6QmuXXxEXWaFuEHb5EHAYrS/bO95+CSItN/Fz1Qh4GWijGqEIrVVU9LJKAvzF9PJskW QxQshm4zeJc0DTIHYFiNAK0doMd2ufFYJzcMf/IwWRuifSsbPmOVlQbs6QfAR/0choZO EzUrpa65Xs2lbHDbn7j8f3MVPdD9sNM50h9tTXeciYSRKiWe9pBC9qrJYaLierqy6VJ2 tSqZHe08om/O9Lu9/ORiCw9HIk0DwhGSC2M+E1ExV9Ri0LNGerU9/Ozc+NAkom0gDr3Y Vmaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=IdlOdGF0YCCYf+h2+Wx+Pr2285UVOLm52gCW9Jb/OEo=; b=SLIaPTEHlH+8boAiKNMyQK53X7xNK8swBii2bGjhzYrvjSNy71mXf70MgCeiBhW88f 7SfDidj+Vm5kPacE50ara50T7pt+ognI14UmLjiwG9CwpdkmIrl9rdxA5g+NHB4LZSAY eLZl8ZA0iQtSFSJiknsCJFrt3Hn46kiDgHdSoKlR7n/Y3fNnIltoFfxxL+QKa2QLYTG+ p4UV8QsG1HuHQI+yjPcj/uRyhFgNiUuUdUxLgBPT/Hd8v/1l20pJegSZ6ho27eNnYhgD c97v0GJCqztHn8lviA+PV6HumP/D+Saw3LhdaxnmvOBrBUBRo1zIefJImVb5nTYkvgPx b7XA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=dxPzci0v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y13-20020aa7cccd000000b00502243d9e27si10641638edt.537.2023.03.27.23.18.29; Mon, 27 Mar 2023 23:18:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=dxPzci0v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232633AbjC1GRh (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232517AbjC1GRH (ORCPT ); Tue, 28 Mar 2023 02:17:07 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B1163C30 for ; Mon, 27 Mar 2023 23:16:51 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id e15-20020a65678f000000b0050f9e396342so2929581pgr.5 for ; Mon, 27 Mar 2023 23:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IdlOdGF0YCCYf+h2+Wx+Pr2285UVOLm52gCW9Jb/OEo=; b=dxPzci0vIh1xyJxlLBjWkEhAjm9YbPSdsUTO5u3B5j0mjvT8KDBeosrn5UiPm3Ksna ECFLKks3+i/JLwnVARkeiq9XCne6PDhOU+plEpcmqroZ0FIo5b3stpDp4azRz7RDEUXr fg0RX+xHWkbr/97SE9yMggMcSWIL9SDqbA2byVm27sTtHVM/RcaUTUFVwNm3hT6OyTW8 KGm1B3oJJOu2mLGqIrSENlsxO5AJatCTwMfnNnO9IVILzvLEURQL9lNGe4Pge9n4mWvC 01ODxHoNCu+DMqEmep2UjWrEts9KjVkOV7sWpi8aLYcEgH/6hlGK54jM+qhbvTAI+Y6D 5qIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IdlOdGF0YCCYf+h2+Wx+Pr2285UVOLm52gCW9Jb/OEo=; b=6nibts82X98VwQ74smC4DwkOYea89Qr8WXQ6kHzRuv7z9LW0WTz5aj7FAguM3W9yvi H1dJy7l2ejujViUp9bmczOI3cIJKjx2sDml5CpRwc5ZDdrmVVbWVEvIcUgATLFiuJ2Sm M7ydvPLMzWuO7xLeuKdarmQY1QmWYnepwd+ukMD/LxseUiy1YTuMD+1fSMgA7vrf8aXq 3DKYFJqWWUiKwwZHkcQHD7RrS3TPqvg7xF6OR4NzeDDnwHmPJBPmkVWCbUoqlzIbrbfa xezajpY5TAUlt9eCkD0cf5MTtdXxoWeYO8m/7Hy6POSUVBI04CoZsECSVATBbVCKR2WM siLA== X-Gm-Message-State: AAQBX9cXCHR+K4sGaAC18aUtKVTruq7/jKDZHM4q0gg3SbwGlRoky33e cpOsNh2oi1H4YF+9wPBqmP8Fmm3ndRUg8rwa X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:74c9:b0:23f:1105:4295 with SMTP id p9-20020a17090a74c900b0023f11054295mr4298168pjl.7.1679984210825; Mon, 27 Mar 2023 23:16:50 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:34 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-6-yosryahmed@google.com> Subject: [PATCH v1 5/9] memcg: replace stats_flush_lock with an atomic From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591251622929831?= X-GMAIL-MSGID: =?utf-8?q?1761591251622929831?= As Johannes notes in [1], stats_flush_lock is currently used to: (a) Protect updated to stats_flush_threshold. (b) Protect updates to flush_next_time. (c) Serializes calls to cgroup_rstat_flush() based on those ratelimits. However: 1. stats_flush_threshold is already an atomic 2. flush_next_time is not atomic. The writer is locked, but the reader is lockless. If the reader races with a flush, you could see this: if (time_after(jiffies, flush_next_time)) spin_trylock() flush_next_time = now + delay flush() spin_unlock() spin_trylock() flush_next_time = now + delay flush() spin_unlock() which means we already can get flushes at a higher frequency than FLUSH_TIME during races. But it isn't really a problem. The reader could also see garbled partial updates, so it needs at least READ_ONCE and WRITE_ONCE protection. 3. Serializing cgroup_rstat_flush() calls against the ratelimit factors is currently broken because of the race in 2. But the race is actually harmless, all we might get is the occasional earlier flush. If there is no delta, the flush won't do much. And if there is, the flush is justified. So the lock can be removed all together. However, the lock also served the purpose of preventing a thundering herd problem for concurrent flushers, see [2]. Use an atomic instead to serve the purpose of unifying concurrent flushers. [1]https://lore.kernel.org/lkml/20230323172732.GE739026@cmpxchg.org/ [2]https://lore.kernel.org/lkml/20210716212137.1391164-2-shakeelb@google.com/ Signed-off-by: Yosry Ahmed Acked-by: Johannes Weiner --- mm/memcontrol.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ff39f78f962e..64ff33e02c96 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -585,8 +585,8 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_tree_per_node *mctz) */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_SPINLOCK(stats_flush_lock); static DEFINE_PER_CPU(unsigned int, stats_updates); +static atomic_t stats_flush_ongoing = ATOMIC_INIT(0); static atomic_t stats_flush_threshold = ATOMIC_INIT(0); static u64 flush_next_time; @@ -636,15 +636,18 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) static void __mem_cgroup_flush_stats(void) { - unsigned long flag; - - if (!spin_trylock_irqsave(&stats_flush_lock, flag)) + /* + * We always flush the entire tree, so concurrent flushers can just + * skip. This avoids a thundering herd problem on the rstat global lock + * from memcg flushers (e.g. reclaim, refault, etc). + */ + if (atomic_xchg(&stats_flush_ongoing, 1)) return; - flush_next_time = jiffies_64 + 2*FLUSH_TIME; + WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); atomic_set(&stats_flush_threshold, 0); - spin_unlock_irqrestore(&stats_flush_lock, flag); + atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) @@ -655,7 +658,7 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, flush_next_time)) + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) mem_cgroup_flush_stats(); } From patchwork Tue Mar 28 06:16:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75853 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2006208vqo; Mon, 27 Mar 2023 23:26:43 -0700 (PDT) X-Google-Smtp-Source: AKy350akj3KoAGTtucZShEdYABq+pywVbhuQtkll3I8md8NgO2WkZaLjMiWNqz40dz0HW0nK9pcC X-Received: by 2002:a17:906:478a:b0:930:fded:5bf2 with SMTP id cw10-20020a170906478a00b00930fded5bf2mr17949743ejc.52.1679984803489; Mon, 27 Mar 2023 23:26:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984803; cv=none; d=google.com; s=arc-20160816; b=KcHkHqSDdaEosn2INXSm2PsbdBPqdMnLs7+cjYIEdmQGBLDN2SMAzQSPh5UriQ2AWX A0U5CufNtgDa3lXjyEjNoUw7aoDejai6oygPloPTyzl/kYru6v8vruGkM3MQ4uLUVipU zqCYcMN9EBR0tcSOs/UEZo8+O5FEU1gYVaq0cZLCVy3Y8VJWHivGEpu5ZiqoYsdnOxBc pdt/M0ezf5jGB7iQvnBr2xznwBMLn7pvdCBMaI08k5ULIXHUmtKNdkSjcFg4Sg+cC0xG 8mfTyckbVT+nm4kSgwZ4Vx6dEhQGcHpcg6saMFuJYuDY/MZDJk91JpF4q+U2Mxo0kKcb c56Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=9IOPAG+bqxWxKt0wdRgDKHhdCXWwpmJ9UigZO44sKyE=; b=hABUHyKo9Uif4ajWWxxp1lUq2j5sT8xd7QAxKirHfJElyDmgYHYFTLi2Aev55AtD8r J2oQyQ/KEXz1Q3EC0lWiZ2Z04TCw/NBFVLKTNjcYd4/+Vxgv/qX4JOhDqxlKu7JhMIvZ +NdIyzAYwnTtLMxqxjwsYCIUM5rhaZCHyHCBS5NJYOj02l6cIVhRxubkGAUkZov0IqLl nLXS2S+rps8pLxvd711Kt2jozYcGiDn71OY5PrKmawp8pfmGPtLdeNY05d3FiXeWo6Gn MXLqrkzF19RKTOkaupt/F+tz3wXbSkTTuFUBURF2DiVydkjPui/iZwdqFZE0rbun+AIc Xr1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ntB4QsMV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gn13-20020a1709070d0d00b00933b62b2533si17948405ejc.297.2023.03.27.23.26.19; Mon, 27 Mar 2023 23:26:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=ntB4QsMV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232663AbjC1GRo (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232548AbjC1GRR (ORCPT ); Tue, 28 Mar 2023 02:17:17 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 96FC835BC for ; Mon, 27 Mar 2023 23:16:53 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id o4-20020a056a00214400b00627ddde00f4so5321325pfk.4 for ; Mon, 27 Mar 2023 23:16:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984213; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9IOPAG+bqxWxKt0wdRgDKHhdCXWwpmJ9UigZO44sKyE=; b=ntB4QsMVT8Fqv0//N2SyNJC8TO+SF5Nd//5MMCXPuMIlVWZztrbNiDvPnYTm36PsVJ 0K9UDAdVpl/jKPNVqGESnlXBby6gGh6KP9olFVGSWL7LZjX9hTDyTd2rcdVRYaALTq3d Svdw5D9xcS6f+R64pTOq4hP3DTwLDxc7s6WmPqIXIzObjUGQ7BCksYYcDbsn8CMNCe4l XYGLULvlYDvGRGQwaFkUUkpNiN3oSuW1tCmNSMxILm7ttbjB9+T2xkgSS2wr2ydIjDg8 ulb0ThWq/NbS6X5lqaGmn6zu6IKbfwWSX3vLp0eLAeAr3Ue3Dif4Fen7adboVvdM5kPM ZMpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984213; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9IOPAG+bqxWxKt0wdRgDKHhdCXWwpmJ9UigZO44sKyE=; b=eryVWX7e9Pf5uP8030yJoHfQIJasUdIiGoPcJ/tCya3GhsrP4MvRlqq6BpZikeb6l3 IKTb6RjLimQDXVuekRWdOenrJFG7stO8Vo3vGdnYiz6c/myyiSiyOIE5cTXhPCg5dO1q m8p/roto7Vp7Q84Wgn50z7IftrU4wNH8bLzkp0rM5q2ma668h0Moy3b/xASbAcAPlOcU xx93R5NxY8VpAAdrz5t/caTJxxLAkv8x9WYeshWFA8Ic8mQ2UzFuzoGg4fF09MOwe2M5 hsBCnEB5lHwkPl8BspakSTcm7BDlgDEqUsCX0sFnN0qnOlCaWrP0ob5KYYYLkQhXE69a 0aQw== X-Gm-Message-State: AAQBX9d8rit7fglTrNWwMvT4wSOYjF2lJbrvph6pPUNQpRLBygPGAqgQ 8okyrUpmR/hFvWwZHON9fBZ83UoWoOPW9mDj X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:4a41:0:b0:507:46cb:f45b with SMTP id j1-20020a634a41000000b0050746cbf45bmr3969874pgl.1.1679984212630; Mon, 27 Mar 2023 23:16:52 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:35 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-7-yosryahmed@google.com> Subject: [PATCH v1 6/9] memcg: sleep during flushing stats in safe contexts From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591745332512481?= X-GMAIL-MSGID: =?utf-8?q?1761591745332512481?= Currently, all contexts that flush memcg stats do so with sleeping not allowed. Some of these contexts are perfectly safe to sleep in, such as reading cgroup files from userspace or the background periodic flusher. Refactor the code to make mem_cgroup_flush_stats() non-atomic (aka sleepable), and provide a separate atomic version. The atomic version is used in reclaim, refault, writeback, and in mem_cgroup_usage(). All other code paths are left to use the non-atomic version. This includes callbacks for userspace reads and the periodic flusher. Since refault is the only caller of mem_cgroup_flush_stats_ratelimited(), this function is changed to call the atomic version of mem_cgroup_flush_stats(). Reclaim and refault code paths are modified to do non-atomic flushing in separate later patches -- so mem_cgroup_flush_stats_ratelimited() will eventually become non-atomic. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt --- include/linux/memcontrol.h | 5 ++++ mm/memcontrol.c | 58 ++++++++++++++++++++++++++++++++------ mm/vmscan.c | 2 +- 3 files changed, 55 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ac3f3b3a45e2..a4bc3910a2eb 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1037,6 +1037,7 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, } void mem_cgroup_flush_stats(void); +void mem_cgroup_flush_stats_atomic(void); void mem_cgroup_flush_stats_ratelimited(void); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, @@ -1535,6 +1536,10 @@ static inline void mem_cgroup_flush_stats(void) { } +static inline void mem_cgroup_flush_stats_atomic(void) +{ +} + static inline void mem_cgroup_flush_stats_ratelimited(void) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 64ff33e02c96..57e8cbf701f3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -634,7 +634,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) } } -static void __mem_cgroup_flush_stats(void) +static bool mem_cgroup_pre_stats_flush(void) { /* * We always flush the entire tree, so concurrent flushers can just @@ -642,24 +642,57 @@ static void __mem_cgroup_flush_stats(void) * from memcg flushers (e.g. reclaim, refault, etc). */ if (atomic_xchg(&stats_flush_ongoing, 1)) - return; + return false; WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); - cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + return true; +} + +static void mem_cgroup_post_stats_flush(void) +{ atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } -void mem_cgroup_flush_stats(void) +static bool mem_cgroup_should_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) - __mem_cgroup_flush_stats(); + return atomic_read(&stats_flush_threshold) > num_online_cpus(); +} + +/* atomic functions, safe to call from any context */ +static void __mem_cgroup_flush_stats_atomic(void) +{ + if (mem_cgroup_pre_stats_flush()) { + cgroup_rstat_flush_atomic(root_mem_cgroup->css.cgroup); + mem_cgroup_post_stats_flush(); + } +} + +void mem_cgroup_flush_stats_atomic(void) +{ + if (mem_cgroup_should_flush_stats()) + __mem_cgroup_flush_stats_atomic(); } void mem_cgroup_flush_stats_ratelimited(void) { if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); +} + +/* non-atomic functions, only safe from sleepable contexts */ +static void __mem_cgroup_flush_stats(void) +{ + if (mem_cgroup_pre_stats_flush()) { + cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + mem_cgroup_post_stats_flush(); + } +} + +void mem_cgroup_flush_stats(void) +{ + if (mem_cgroup_should_flush_stats()) + __mem_cgroup_flush_stats(); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -3684,9 +3717,12 @@ static unsigned long mem_cgroup_usage(struct mem_cgroup *memcg, bool swap) * done from irq context; use stale stats in this case. * Arguably, usage threshold events are not reliable on the root * memcg anyway since its usage is ill-defined. + * + * Additionally, other call paths through memcg_check_events() + * disable irqs, so make sure we are flushing stats atomically. */ if (in_task()) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); val = memcg_page_state(memcg, NR_FILE_PAGES) + memcg_page_state(memcg, NR_ANON_MAPPED); if (swap) @@ -4609,7 +4645,11 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, unsigned long *pfilepages, struct mem_cgroup *memcg = mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + /* + * wb_writeback() takes a spinlock and calls + * wb_over_bg_thresh()->mem_cgroup_wb_stats(). Do not sleep. + */ + mem_cgroup_flush_stats_atomic(); *pdirty = memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback = memcg_page_state(memcg, NR_WRITEBACK); diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c1c5e8b24b8..a9511ccb936f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats_atomic(); /* * Determine the scan balance between anon and file LRUs. From patchwork Tue Mar 28 06:16:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75848 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2005064vqo; Mon, 27 Mar 2023 23:23:27 -0700 (PDT) X-Google-Smtp-Source: AKy350ZIwSLxgdGSM9UekpH+fbr+UrZ3pgjEJdoihWM4S47JDnmCTsSzsKidSoMsM2eHGAzMukAb X-Received: by 2002:a17:906:d966:b0:8b0:f58d:2da9 with SMTP id rp6-20020a170906d96600b008b0f58d2da9mr15183094ejb.64.1679984607406; Mon, 27 Mar 2023 23:23:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984607; cv=none; d=google.com; s=arc-20160816; b=vo9aCw3TvwMYbtTD9pf8T3D+MJuGXNsYy1GaNDHuvaeufR7wo5lFoAzIDbW6uN+9ur OnJldUXD3WZzEHijrSGEZVDQwDzc1Rzj/Tl92HbMooDLvBaNaOlschqIZvJyXNDl5PPZ L1fB9oEB3jzsaEcOvpUNDHPH4lDR4RnbkqXBvxT8bFcRa88xYnTxpbnzXidDx7B1ey+Z iczI6YbZ8ZoahFpt5BFr4SeIIfwHKRs4b4Nr3Hh4SqT3TFrik9DsKeusI8tTKemL2md6 bTvGeYLiRJhhCKfhq9dRFcN6xoA/aMSlOjIwkk4apmx1Q+x6kZuCJR3KMJEmaPlA1n7/ Xo7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=xnLycyx49ihstLgpeZp8I4sx6kPxquG1P3Dlzl3MzyVdXZVCYFvQpIhw48D36WZ0zV tlSyBeOgqLWnghrt4xEgoBa4zDd2xws6Lw+tyQCE3UrN1pPsyPUEEWwqG6i1JF1n7ixf M0V69z4QZgV819Y37ahK9jNsaZzOKQBpHs/mAMsnknXORO1+iVr7DerSiEHztgjddebS 0Bs82D/ao578fbJ2q0JH09sWQO9F41Rdau7O8QyaqwfbafChCyyctSxGeW61JtCIUqkT j780sGB4od1kwq6tlcIhCEt92wHbv7oZUCf5+8ej9JgnAK9jVSKrnm0aqgja3raz3LHX dHxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=qIxeFfAv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fy8-20020a1709069f0800b009335b145df8si7455740ejc.363.2023.03.27.23.23.04; Mon, 27 Mar 2023 23:23:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=qIxeFfAv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232700AbjC1GRu (ORCPT + 99 others); Tue, 28 Mar 2023 02:17:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232578AbjC1GRX (ORCPT ); Tue, 28 Mar 2023 02:17:23 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 259513A88 for ; Mon, 27 Mar 2023 23:16:55 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id b16-20020a17090a991000b0023f803081beso2939413pjp.3 for ; Mon, 27 Mar 2023 23:16:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=qIxeFfAv7LZhUaMY9U0TvTtrAPX4wRqNbnIeY0Yrn/RnFRGntBTlu0hfuIAsMfAy2n 1P28EpdkOB3njh6q599A6aRPcbQ+/6Ngp2xMzs6rBLHlr6vd8s4xgEvIn8RMUD6Mze8g o6DwJQ//Mohpbdf8YD2EiOxbMyBh8HdrF+/Yzlad7E0D+Fg0j+P6NKtF5T845D/HLRrp rP0WF5ztka030C2opL/oGdjy+y9vRWrHiXRiQXH/e0QgpKRtMDaGNueTHyNdIDcH2xFM Q3P8IwnLdzShQrK/qG/mnGtQghgE6XGNQRvgt1djyelN/qv5GiJdyRld4l0gXlsYj6Rl orEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984214; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0xrFtgulASYXrMO3GaE6CpeYp9Puj0hiviKaUT94oIg=; b=qQhF9DdaOiY3TQjhmFuPZBQ3i2vs1MVqzTemCNyQcqUHeaQb/+z3Tsi7ij7TSe6fGa mxGc+/JGDuFdrEjzvVma3UUhHR3YDsMmZrBKtCI5HgQ5866ly6ZwY4D+q044GFZoVYW6 iCgjFmiqJuVnxjq3UEQwOdpcDzrr/nTHgxpD84jg8fveKfn34TS8RKbpd4t/Tyw+BzRf PgNGFa3nDLgin2O5ENmlu1PlHG0S4wJ9AXzCB5hHlAV+n+bNRT1fQ80QkpllL3739d1k gCipq1eRTP7AO93nL5UXR+JGhRFDYMN7+bXpXE2UTHjIFf6jl8iHfYIb06X8HaaAiFuT tAeQ== X-Gm-Message-State: AAQBX9c7v+ZJ25GjU6L2EwzCQFd+beJE4ddoE4tyM0ETtMd/WA6vMSZe bbjKAh/SN97gH3P9Ecr9HbdLdpIuxbUUBf0v X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:115a:0:b0:50b:dfd4:b56f with SMTP id 26-20020a63115a000000b0050bdfd4b56fmr3747909pgr.5.1679984214584; Mon, 27 Mar 2023 23:16:54 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:36 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-8-yosryahmed@google.com> Subject: [PATCH v1 7/9] workingset: memcg: sleep when flushing stats in workingset_refault() From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591539560349625?= X-GMAIL-MSGID: =?utf-8?q?1761591539560349625?= In workingset_refault(), we call mem_cgroup_flush_stats_ratelimited() to flush stats within an RCU read section and with sleeping disallowed. Move the call to mem_cgroup_flush_stats_ratelimited() above the RCU read section and allow sleeping to avoid unnecessarily performing a lot of work without sleeping. Since workingset_refault() is the only caller of mem_cgroup_flush_stats_ratelimited(), just make it call the non-atomic mem_cgroup_flush_stats(). Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 12 ++++++------ mm/workingset.c | 4 ++-- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 57e8cbf701f3..0c0e74188e90 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -674,12 +674,6 @@ void mem_cgroup_flush_stats_atomic(void) __mem_cgroup_flush_stats_atomic(); } -void mem_cgroup_flush_stats_ratelimited(void) -{ - if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) - mem_cgroup_flush_stats_atomic(); -} - /* non-atomic functions, only safe from sleepable contexts */ static void __mem_cgroup_flush_stats(void) { @@ -695,6 +689,12 @@ void mem_cgroup_flush_stats(void) __mem_cgroup_flush_stats(); } +void mem_cgroup_flush_stats_ratelimited(void) +{ + if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) + mem_cgroup_flush_stats(); +} + static void flush_memcg_stats_dwork(struct work_struct *w) { __mem_cgroup_flush_stats(); diff --git a/mm/workingset.c b/mm/workingset.c index af862c6738c3..7d7ecc46521c 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -406,6 +406,8 @@ void workingset_refault(struct folio *folio, void *shadow) unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); eviction <<= bucket_order; + /* Flush stats (and potentially sleep) before holding RCU read lock */ + mem_cgroup_flush_stats_ratelimited(); rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might @@ -461,8 +463,6 @@ void workingset_refault(struct folio *folio, void *shadow) lruvec = mem_cgroup_lruvec(memcg, pgdat); mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - - mem_cgroup_flush_stats_ratelimited(); /* * Compare the distance to the existing workingset size. We * don't activate pages that couldn't stay resident even if From patchwork Tue Mar 28 06:16:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75855 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2006516vqo; Mon, 27 Mar 2023 23:27:44 -0700 (PDT) X-Google-Smtp-Source: AKy350YyEcSjiwB9NwvXj13skZxT6h8O4YT0HZtQHD3EgmcORlbnDjaRv/uWcsuf6uAYc0W3XgDm X-Received: by 2002:aa7:d584:0:b0:4fa:ba60:8188 with SMTP id r4-20020aa7d584000000b004faba608188mr14239408edq.8.1679984864153; Mon, 27 Mar 2023 23:27:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984864; cv=none; d=google.com; s=arc-20160816; b=CnB7GyTkzBqWUzHFzYoVMp7TFU0EDplzjhCtoxQvONUPNxq5ltwvEP37SIDewVv5ms pbS7DmMWdpiuWVVkGXR5pcxwkehEuUunmm+sWl6gqLpqEvRjoJDjosmig35EiroPrVuG ic5cK+JSB1+i9Kmr/05DGMf+lcvySoFY6U4LwrohTsR7JzBJKusxHaIG87Ad8HjoKHw8 gNNUIktmIspg5w7M78T+qFK1V0jYziK6pZFO1youmnDm280f69FxygSaBW9rBwRpIfCH tzRN97W5xScdwTF8O8eX5H8Trhu3p6ZP384bQAiIT1dmQlV4ufFD8tRPZL2scq/ODQqd PAWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=x7IFy8THM3GBUAgd4d/GEKRMIpmDkqCRA5AxR/3TFCM=; b=IhZAgJ/NMaP1gRPQiiiimEWpO6fvCLt7Y/YzhFpnVTYcdo/hktRACWLk+bMtyBagRD YZt4yP93Qf1+w90qtPDbV4svUSRE096xTrJJq9iqcdQ4mKrZYOX0jVumHWQxZopWP4Xv 0E24DC04cIRCxWDwo79aO7IYSV2aPzEQBLG1QvXu6ax57lLBaXiZgQJcAMIAjToSSJFO S4+Baq0BoEdIP5bQH02EkqGOzu0naIxtYE9qqt8oXbLfTjf+VGTSWJJ7C5SEN7ViKcPR dWTWk55n19cWjLUFBG1ysUsuEzI7r7CISLPx4cNGQkgd8jmWcpd8czssnniM0hB0ino5 XYCA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BGZjnL6U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c23-20020a056402121700b005021f0d5750si11842562edw.663.2023.03.27.23.27.20; Mon, 27 Mar 2023 23:27:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BGZjnL6U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232745AbjC1GSA (ORCPT + 99 others); Tue, 28 Mar 2023 02:18:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232523AbjC1GR0 (ORCPT ); Tue, 28 Mar 2023 02:17:26 -0400 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC8B930D3 for ; Mon, 27 Mar 2023 23:16:56 -0700 (PDT) Received: by mail-pj1-x1049.google.com with SMTP id j15-20020a17090a318f00b0023fe33f8825so2932533pjb.9 for ; Mon, 27 Mar 2023 23:16:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984216; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=x7IFy8THM3GBUAgd4d/GEKRMIpmDkqCRA5AxR/3TFCM=; b=BGZjnL6U27C3nu3nfjJxe/VV851RC/VP1GsLD606AgPd6FshXe6WNQk0I659Owa2wU gVJoC9Crt60CfuYM+ql+MgbEx44waD/ankuLiQjCR+6oc5mr8Ex9gHFcY/MTbuN2MESv WG+o2vmHU2skqPZDZWhEEvumwbMUFH8XB/2P7lCWIxY9j69P+XyQb7YmlC5lJf3dl83N R/+G9MzJdyv2aInyc3KBZTsCKswAeBzT3OW0sicqNfQnZRl6LhPxfTDtAjqxejZWrmX4 kekpZffHaXAsR+R6/xnXuXgj0ngR8GdA98eO9lk+Gv9Mx4rtmu3ZFp7UhRacTBWLo1ks cDtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984216; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=x7IFy8THM3GBUAgd4d/GEKRMIpmDkqCRA5AxR/3TFCM=; b=xj56OWwuWyrQgH2udRIFL0KVa9/Uisey3/s0qUL8UZflIPystXacHDXM+mYxSjAv4U xplF1onBAnHa6uSGV81RToIDlHJSa7X4qZD+TuEge/AXHknSJEdIN83JrDQVOlGA9GBw D2NeZKULvnG+IkC/vwjqbWWfN9nIMPNH85kNn6zAocLb9Or1vedIFkB10jrYaAA3GlJp LL2ADRVT04Jaehc2qYn3YXfpYC9HpRKdU4zPyCxMUomygDZ7YF9+vgAvsX72cUJcIHAO HEaTGB0ne4LDi2wpS/J/Sn8DFDtVlhLR/4882VYGctZzIXq4aRs6XT57/vTOpLLJRnqB y4Kw== X-Gm-Message-State: AAQBX9f9vgrHABIuNHp3VhSKnZLnD1oGkp5DrFadz4TGgnBazMgH0mu+ tZuuryxqLyQt3jQso/vgBGy5LufjI8p9rQ7T X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a63:c49:0:b0:502:fd71:d58c with SMTP id 9-20020a630c49000000b00502fd71d58cmr3919025pgm.9.1679984216134; Mon, 27 Mar 2023 23:16:56 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:37 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-9-yosryahmed@google.com> Subject: [PATCH v1 8/9] vmscan: memcg: sleep when flushing stats during reclaim From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591808971100278?= X-GMAIL-MSGID: =?utf-8?q?1761591808971100278?= Memory reclaim is a sleepable context. Allow sleeping when flushing memcg stats to avoid unnecessarily performing a lot of work without sleeping. This can slow down reclaim code if flushing stats is taking too long, but there is already multiple cond_resched()'s in reclaim code. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9511ccb936f..9c1c5e8b24b8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2845,7 +2845,7 @@ static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats_atomic(); + mem_cgroup_flush_stats(); /* * Determine the scan balance between anon and file LRUs. From patchwork Tue Mar 28 06:16:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 75847 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2004991vqo; Mon, 27 Mar 2023 23:23:14 -0700 (PDT) X-Google-Smtp-Source: AKy350aztlfZEJzDWHvgNsrg2X6eot8Fl4CitiubpdXTBauTg6cL7Ke0ZuedUy5Jej1ff+JJw9U4 X-Received: by 2002:a17:907:6d9b:b0:941:1cf6:dbea with SMTP id sb27-20020a1709076d9b00b009411cf6dbeamr11212594ejc.55.1679984594066; Mon, 27 Mar 2023 23:23:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679984594; cv=none; d=google.com; s=arc-20160816; b=AaODVO7zVRfUyEtH3DTHacjNr0JtKgLfx3adZLOI+8NEeq5OB/imfQpCu6KTpgrJUn QWsVmvMmUn0uCriH+PriggyyVLHQ1Hw2h8zM0zS1cyaNQodmt4m90B6Yb1eogsEAS6ZP aI22KxEiC7aatPz0/LxLIwWjWVu65j8Wd2qNY8WDwiIk0GwaFM9kFpjH2fzE0YlnN29y xL9XOAJVzXSCUmsxdHZNkJqLYCMrtJXc3Ke4Y1OV6ciTqejpL3Muw7aSdHMRZDM9eSt0 Scc+loAypnk2WaRjjY8MAg7GSABxBN5MSTUjJKGdGWtJv1vdb2FCK8zdJK4KVmBu+SwS xjVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=Vy8wz9BoJKucmcMBXvIGPOuITeThi///HtllaZbgtCo=; b=u1YCOByN7vObdsdLRKyEZwu2s3LmuGfPSs3ArcmmOB5ATOivpm4JVViFiQSHM6/Fos D2bnaTZ2aR1xFO7KfikH7ybm3cf0fr2zuNqMFS9mCugYlX2xA4TpDkN6JP8/OIFv5+ZT czxJbFGU9Zp/TOTuaTQM0A5vVWdIYOTtD6XoSm/qxPRnzOFFAVOfvpx1QovmizZ7EwsX gScbjLVhlAjzBfDayrQDQPtcCuQiUZCdVyAtuIfzLDCc725g9ykeYbGXTncjRzKRtSNL lnvIsDZcPzmyYwHTlXwgikbeiyRaOsOWTNUXAJrDaFudpYJ/uromnwdnMnxE6fbBoWJ1 72CA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=G3C3U8FV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k17-20020a1709063e1100b0092fd7346e4bsi29660526eji.342.2023.03.27.23.22.50; Mon, 27 Mar 2023 23:23:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=G3C3U8FV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232656AbjC1GSE (ORCPT + 99 others); Tue, 28 Mar 2023 02:18:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232640AbjC1GRj (ORCPT ); Tue, 28 Mar 2023 02:17:39 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE3C53ABF for ; Mon, 27 Mar 2023 23:16:58 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id q1-20020a656841000000b0050be5e5bb24so2927769pgt.3 for ; Mon, 27 Mar 2023 23:16:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1679984218; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Vy8wz9BoJKucmcMBXvIGPOuITeThi///HtllaZbgtCo=; b=G3C3U8FV5xO4vsplclhU9+4PkBH686vIIIrQFFquICu8//oGn5xL/Tmt4VNVE1HnMt hORJK4mB81itHhGs1+plTWPpHn6PXb4UvAXp2In5M38sRtIYXUydb86gW9UCcwU8NUMK AKSJULzbSmJ/Zh8xR8z20CA4EuuFhupmK6yUb7XM0dWX+orGZyC4roBLz+bFSVl/SrGz jDFcau8INkey+e5jkuXnym7vXagjhy1slym0NeO+iUEtBeJGIjBlnt29eYQhaZCdCsb9 TItPIolnMlslNIl39r9v4yrb91hLywelng+1GUeMlXmGbk+u0V1YHrp34uG87/cMCuf4 DKLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679984218; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Vy8wz9BoJKucmcMBXvIGPOuITeThi///HtllaZbgtCo=; b=f7YKdNJK+YRWbccPTYakD+plaVJW9U5hZNqzTwzXNUMy9E4A/8oQnSiw0QX2yx5IfO 9P1Y7VhNBTinivdyVh34FcPLdLj8hoYDDVZgh6K0PCE8/4la5b6X1jKzqHuVCKxEkTAY mijmbAxQdqCPHKCubQmppEnxiOBB936ThxeaDn5Cl2+6gXYhPHn2nqpg74F6CU9cOuhD Xlgyb2UaUYod6wn+ncGf+9UJzFsvTkGbDBo5Gxaf36JHpoceBZAIl7vltH7ERx/+tFi8 1SWRkaWSKZ8/6CdILzQnZcGR6ccBWt6OWWMuRHfsK6gWd7tUhZo4sj/OyXHvgS1jH0CL Kpyw== X-Gm-Message-State: AAQBX9c+eJCKx3k4JGEQiZRBZfbVJODep410pQ53qpAtC24l7UlggXj3 XhVMcXo0uW82dANDO1iUL5dTKEwBYux2BFwi X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a05:6a00:1781:b0:593:fcfb:208b with SMTP id s1-20020a056a00178100b00593fcfb208bmr7364360pfg.3.1679984218020; Mon, 27 Mar 2023 23:16:58 -0700 (PDT) Date: Tue, 28 Mar 2023 06:16:38 +0000 In-Reply-To: <20230328061638.203420-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20230328061638.203420-1-yosryahmed@google.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog Message-ID: <20230328061638.203420-10-yosryahmed@google.com> Subject: [PATCH v1 9/9] memcg: do not modify rstat tree for zero updates From: Yosry Ahmed To: Tejun Heo , Josef Bacik , Jens Axboe , Zefan Li , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , " =?utf-8?q?Michal_Koutn=C3=BD?= " Cc: Vasily Averin , cgroups@vger.kernel.org, linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, Yosry Ahmed X-Spam-Status: No, score=-7.7 required=5.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761591526008586241?= X-GMAIL-MSGID: =?utf-8?q?1761591526008586241?= In some situations, we may end up calling memcg_rstat_updated() with a value of 0, which means the stat was not actually updated. An example is if we fail to reclaim any pages in shrink_folio_list(). Do not add the cgroup to the rstat updated tree in this case, to avoid unnecessarily flushing it. Signed-off-by: Yosry Ahmed Acked-by: Shakeel Butt Acked-by: Johannes Weiner --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0c0e74188e90..828e670e721a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -618,6 +618,9 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { unsigned int x; + if (!val) + return; + cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); x = __this_cpu_add_return(stats_updates, abs(val));