From patchwork Fri Jul 7 04:46:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 116951 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp3028494vqx; Thu, 6 Jul 2023 22:07:03 -0700 (PDT) X-Google-Smtp-Source: APBJJlH9nidX3iVS4UwNQ+DlUijrcT7+8N2emhLbq1G7UpcqULPDXu6EMezCdLWHvxTeQJl8TLUs X-Received: by 2002:a17:902:d486:b0:1b1:9802:a31b with SMTP id c6-20020a170902d48600b001b19802a31bmr3378131plg.41.1688706422852; Thu, 06 Jul 2023 22:07:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688706422; cv=none; d=google.com; s=arc-20160816; b=UHxhV6jGYk/6zaOJzUNfnqxy7tDzrAk2vG9+sHKleGBrgI194TelSY7eKqYIauGO/a 9o7TchmrZkDfGw1/lqOdGvXYCim/j5y8YYUvEH0jf1fI4+452De5MlyEFAg4xeaEQ4yY hkQTHwtc7rCAI0uSFmjF64FSrx01y1QvPCuiRCDsfi0mhxtaJ1uzq8L1Oo+uwPPJUOZ+ 8+AKvlWhJytbSfMwen/wWMdC/eKI4Oe+g3bbGvsX7cyIYl3uhsUx3w7jUByLFQfuirVd Pt6d4vyLdL4/viXOjNxaphDScyztfbBmERwl0Hyeg1ZiQMiP1U5j4ycIZutZbAIEKdKs PaMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=FOcqYStjaEQ6e41x7ko6U0Q7unE1KNRPYn3wRQMvZDI=; fh=512KhH5uQWMkzdsfYlnl6sN/+Kp7rI1F537sZxMX9nk=; b=zzjM8ukBie6S6trZH8cB6kD0Dlq7vmc2Xm3zHxiBXolNxIz9W7tnItyAClqKjhrVuH HaQDFiW+EiQkt28n1xOGM+G8MCveqyIGylRs0NGEkygiKCmd91dn8SJ1+4i78mNonQ1m BqBy2FslUMPMHP7gsiXQaB4orl6uxUVtc+GHJSxqSVnL+urb33E1JWMyO3mYu6bnMjL/ +p9BCb/w6sPCwtHRW4SIB2DQVXjbyJvI0wjw0UgGwGJMEHqNvD9M45e/kY/6Inr8lyUs Tq0u9+gzwXBAhdUR9OCBLYQH3+1nJbBd5XdaMA3BtR87k7AZU8Fs+3X6tqOolmWVZbu9 Dcwg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=TYgKO3N5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e11-20020a17090301cb00b001b87070052csi3059992plh.140.2023.07.06.22.06.47; Thu, 06 Jul 2023 22:07:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=TYgKO3N5; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232108AbjGGErs (ORCPT + 99 others); Fri, 7 Jul 2023 00:47:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232369AbjGGErd (ORCPT ); Fri, 7 Jul 2023 00:47:33 -0400 Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8E592121 for ; Thu, 6 Jul 2023 21:46:58 -0700 (PDT) Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1b8c81e36c0so6356815ad.0 for ; Thu, 06 Jul 2023 21:46:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1688705218; x=1691297218; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=FOcqYStjaEQ6e41x7ko6U0Q7unE1KNRPYn3wRQMvZDI=; b=TYgKO3N5/8AF1DSrz62f/fVEd8Ub/T0GTEjqgOc/CnmNJDa/3aQ78ZZH9tlDgabtFW af5rbQLq7ZJ7TK7/GQZ+8VKbXdqs5RaFnVVxmDXz77LBVugNv3nFHLyQ02pFOKH6OqFD Adx8qDN1gFulXupcAHzjOeG+9nbm3L8Fc33n6c8UJRUj97T5YRh2GiZXbRIN5EGPnBFj cfJpW3pWCwk4sco7Dxke460vunPL47JNi53q9bQAGiUUjHVoCEEW0FS7CUWgbD6iE26w qd7KS3hDflwBTEsSrO5vyvk0LpVLRUBfxwyjegH12S5YGMor9x3yhdjKiQi8Fu2SyyyO 9KbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688705218; x=1691297218; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FOcqYStjaEQ6e41x7ko6U0Q7unE1KNRPYn3wRQMvZDI=; b=OhANxo1px+sXylXJ1cqi8sP4nFBEJVH4ivyvbrA/TdbpSnaPay9D9OZ/uaVap8cJrq ebGmxOIQz0yVmb+Ut0WWNUOVCda0S2ovyOkitHSbSbv2Z2/8B9mq9aEv4q31Mz1PwpQw 8/RX6IUGU7JH9jNjZ7hlnlGMSFmgJn3ApDuYNz6dHiY+kO7YLIhcMkE/lPkRv3CYRhTi jgPMO08EEw47C7BF3jck0/rPSPsj7B6n0ygBYoL1UOM6qFML0ExsuSsy34GzKK5AfY2b l2l3XO68L0CLmGhXPBcMgk1V+7OEtUtwcEE6vXKeVxPesXAEG+OUwFHpJsNaUBNo87OS 0RAA== X-Gm-Message-State: ABy/qLaBPJdoU18HibDftmVxb1E6GHzjop6PoePp5GMtra54qD6Mdah+ 3wEiXA8RBOSetnAbQ0CqF5cu0A== X-Received: by 2002:a17:902:7c0a:b0:1b4:65c3:ca68 with SMTP id x10-20020a1709027c0a00b001b465c3ca68mr405138pll.66.1688705217824; Thu, 06 Jul 2023 21:46:57 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id bc8-20020a170902930800b001ac95be5081sm2193423plb.307.2023.07.06.21.46.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 21:46:57 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: david@redhat.com, yosryahmed@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 1/2] memcg: Add support for zram object charge Date: Fri, 7 Jul 2023 12:46:49 +0800 Message-Id: <20230707044649.1169155-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770737026373021149?= X-GMAIL-MSGID: =?utf-8?q?1770737026373021149?= The compressed RAM is currently charged to kernel, not to any memory cgroup, which is not satisfy our usage scenario. if the memory of a task is limited by memcgroup, it will swap out the memory to zram swap device when the memory is insufficient. In that case, the memory limit will have no effect. So, it should makes sense to charge the compressed RAM to the page's memory cgroup. As we know, zram can be used in two ways, direct and indirect, this patchset can charge memory in both cases. Direct zram usage by process within a cgroup will fail to charge if there is no memory. Indirect zram usage by process within a cgroup via swap in PF_MEMALLOC context, will charge successfully. Signed-off-by: Zhongkun He --- include/linux/memcontrol.h | 10 ++++++++++ mm/memcontrol.c | 23 +++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 5818af8eca5a..716cc08a6527 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1819,6 +1819,8 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, rcu_read_unlock(); } +int obj_cgroup_charge_zram(struct obj_cgroup *objcg, size_t size); +void obj_cgroup_uncharge_zram(struct obj_cgroup *objcg, size_t size); #else static inline bool mem_cgroup_kmem_disabled(void) { @@ -1880,6 +1882,14 @@ static inline void count_objcg_event(struct obj_cgroup *objcg, { } +int obj_cgroup_charge_zram(struct obj_cgroup *objcg, size_t size) +{ + return 0; +} + +void obj_cgroup_uncharge_zram(struct obj_cgroup *objcg, size_t size) +{ +} #endif /* CONFIG_MEMCG_KMEM */ #if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e8ca4bdcb03c..e1d2c084c132 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3059,6 +3059,7 @@ struct obj_cgroup *get_obj_cgroup_from_page(struct page *page) } return objcg; } +EXPORT_SYMBOL(get_obj_cgroup_from_page); static void memcg_account_kmem(struct mem_cgroup *memcg, int nr_pages) { @@ -3409,6 +3410,28 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) refill_obj_stock(objcg, size, true); } +int obj_cgroup_charge_zram(struct obj_cgroup *objcg, size_t size) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return 0; + + /* + * Indirect zram usage in PF_MEMALLOC, charging must succeed. + * Direct zram usage, charging may failed. + */ + return obj_cgroup_charge(objcg, GFP_KERNEL, size); +} +EXPORT_SYMBOL(obj_cgroup_charge_zram); + +void obj_cgroup_uncharge_zram(struct obj_cgroup *objcg, size_t size) +{ + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return; + + obj_cgroup_uncharge(objcg, size); +} +EXPORT_SYMBOL(obj_cgroup_uncharge_zram); + #endif /* CONFIG_MEMCG_KMEM */ /* From patchwork Fri Jul 7 04:47:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 116952 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp3030472vqx; Thu, 6 Jul 2023 22:12:22 -0700 (PDT) X-Google-Smtp-Source: APBJJlHLzgUHw18wnjmCWw2hRpEBrQIsEQPnETfKKxPGSX/PNxTNISZJk/O/BBi3DbI08szlIlTC X-Received: by 2002:a05:6a20:7489:b0:12b:b9c0:aa61 with SMTP id p9-20020a056a20748900b0012bb9c0aa61mr3670128pzd.29.1688706741813; Thu, 06 Jul 2023 22:12:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688706741; cv=none; d=google.com; s=arc-20160816; b=LlHdgGrVpuRv+KaHzzwmNvY9kZ/C9TQJ86a5KP6VZpNtQwLeR307ISfE2pRSb9dNJg 3KT/0P10pOOiZmxJqQtiKacFpIOFB+BUko86/PctMwQPoBlOXprjQoZzEqAt/w9opnpH maXQ8V/m8+UyEfEDO9KDBMUUgORenjfs6MHfrqJ4P6UllVLmUjdDhBvcD1eBH8MMSZ7z UBY/eWPz9k/ITnmwRqMk3EtSQcwdnTuRHhMP2tiiUQpEEQamgG5mQ4fKRrLIs/qrvt/F Jd6VATh+O19CZiDv5VzcpeZtHjPxwXB/MBO1kPxnLjt4tM6Uv/SGnpmETPT870lTLUXJ RzQw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=OEV7S2qQa9wjcpqCPZF1/IRLU/XLUimMDUCsPYYW7kI=; fh=512KhH5uQWMkzdsfYlnl6sN/+Kp7rI1F537sZxMX9nk=; b=r19yu/zYnxwVPujLdJEzbnIHeMaidv1UAsJrBG9Kx8OolsYcL/bTckwBQPjWNe7dM0 9VEV4FWD4jU07MARl4sOwDir4tFtyvvUuFDvu5A8FbFmHOIwggy21wN+IjcOTf8V3+Lb IgtHujqqTtI+CTem9dwXMzTzFEn4sEqmhoPMigNVMarSBxTBtrFUdwp85zaSSaFeoeOR hfz1e0SV0h4gBx6qNC6ras4MYGgP+1tUEUQT7AMWoIlZF4TjqEeX0zrVQHeRD0BQWLzb o2DGCykaHwMDF8Sijuj16fnj1DvynawbbrMEpH5nfLUA97hwTb3CZRRokoyy/AP/NlCj SQxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=i9R4JehL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l18-20020a170903121200b001aafb271d13si3316554plh.235.2023.07.06.22.12.09; Thu, 06 Jul 2023 22:12:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=i9R4JehL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232201AbjGGEsH (ORCPT + 99 others); Fri, 7 Jul 2023 00:48:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232177AbjGGErz (ORCPT ); Fri, 7 Jul 2023 00:47:55 -0400 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47DD31FEE for ; Thu, 6 Jul 2023 21:47:16 -0700 (PDT) Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1b8b4748fe4so8010035ad.1 for ; Thu, 06 Jul 2023 21:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1688705236; x=1691297236; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=OEV7S2qQa9wjcpqCPZF1/IRLU/XLUimMDUCsPYYW7kI=; b=i9R4JehL62yeUI0J/4ssJQxvz0puzqqchTK/dmDx0jOYqynb15e1FVB9HaIbdAnBap I+9q1hHV+sLd0FkRPQrxa357li+iU7oRQ3ZKljZNWV3srIwdrp0BycDD7H+epgpwzfRQ YVWBEAsXX8PF0UhPux74/+tZUH3QnrxpBO4WdqdABk7EeoVbQxDpUwN7GkVD8PfbmKbi jrqdFSt13I8trb9p3hzK1fwPq4uIr0nK3BpF2FMwkbqRd5wXqzHhSxMUm3PxZ/vdQo/j 5S8xAvzry2rY9pLS05xsu74C9bbszUTiarWjDUT6mckjYUbOG3yUlgSfPTRsN+uftCxC r1qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688705236; x=1691297236; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OEV7S2qQa9wjcpqCPZF1/IRLU/XLUimMDUCsPYYW7kI=; b=HVHK+elDhCaXXD+UZIYbPOuuTave7+YJDXwk5MsLHVDUqioY2OaqwXj4yEy73h13vQ OJjCp4uOcOerBcxh7TDZTetIxOtUykIJJJtASUtSundL9i0SGXpLD86saym97AdZVNWI ZIo9U6f8JQMz7CW7mc5sdKyip29iwY6wMCWKGwBwsts3GQDFz8hyJGiAPePIcQdYgIwW csjY8DZ+cuv4iBJREguzmBG1VoHBXRqM2R3KUiwho+I6+A7MCDXeMnN73q8RaTrq5GFo VszV6lRLRh6T+JQBcM/edS0slnUzL59/uRIPpB4zAMBw/9DbIkyZyXmzW/ZZfbR9Bo9H B9Jw== X-Gm-Message-State: ABy/qLZqS7s0J2yn/V8JJ7rcvOTsQZ6ysZO1TQQFYyRiVITWr7t4I7cW rKYjTPztrDTZYdCtkrbIyGrwtg== X-Received: by 2002:a17:902:7405:b0:1b8:b827:aa8e with SMTP id g5-20020a170902740500b001b8b827aa8emr3111867pll.11.1688705235754; Thu, 06 Jul 2023 21:47:15 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id i11-20020a170902eb4b00b001b86deadfddsm2181858pli.264.2023.07.06.21.47.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 21:47:15 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: david@redhat.com, yosryahmed@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 2/2] zram: charge the compressed RAM to the page's memcgroup Date: Fri, 7 Jul 2023 12:47:07 +0800 Message-Id: <20230707044707.1169285-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770737360744088499?= X-GMAIL-MSGID: =?utf-8?q?1770737360744088499?= The compressed RAM is currently charged to kernel, not to any memory cgroup. This patch can charge the pages regardless of direct or indirect zram usage. Direct zram usage by process within a cgroup will fail to charge if there is no memory. Indirect zram usage by process within a cgroup via swap in PF_MEMALLOC context, will charge successfully. Signed-off-by: Zhongkun He --- drivers/block/zram/zram_drv.c | 43 +++++++++++++++++++++++++++++++++++ drivers/block/zram/zram_drv.h | 1 + 2 files changed, 44 insertions(+) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 5676e6dd5b16..770c7d58afec 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "zram_drv.h" @@ -135,6 +136,18 @@ static void zram_set_obj_size(struct zram *zram, zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) | size; } +static inline void zram_set_obj_cgroup(struct zram *zram, u32 index, + struct obj_cgroup *objcg) +{ + zram->table[index].objcg = objcg; +} + +static inline struct obj_cgroup *zram_get_obj_cgroup(struct zram *zram, + u32 index) +{ + return zram->table[index].objcg; +} + static inline bool zram_allocated(struct zram *zram, u32 index) { return zram_get_obj_size(zram, index) || @@ -1256,6 +1269,7 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; + struct obj_cgroup *objcg; #ifdef CONFIG_ZRAM_MEMORY_TRACKING zram->table[index].ac_time = 0; @@ -1289,6 +1303,13 @@ static void zram_free_page(struct zram *zram, size_t index) goto out; } + objcg = zram_get_obj_cgroup(zram, index); + if (objcg) { + obj_cgroup_uncharge_zram(objcg, zram_get_obj_size(zram, index)); + obj_cgroup_put(objcg); + zram_set_obj_cgroup(zram, index, NULL); + } + handle = zram_get_handle(zram, index); if (!handle) return; @@ -1419,6 +1440,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) struct zcomp_strm *zstrm; unsigned long element = 0; enum zram_pageflags flags = 0; + struct obj_cgroup *objcg = NULL; mem = kmap_atomic(page); if (page_same_filled(mem, &element)) { @@ -1494,6 +1516,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return -ENOMEM; } + objcg = get_obj_cgroup_from_page(page); + if (objcg && obj_cgroup_charge_zram(objcg, comp_len)) { + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zs_free(zram->mem_pool, handle); + obj_cgroup_put(objcg); + return -ENOMEM; + } + dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src = zstrm->buffer; @@ -1526,6 +1556,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } else { zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); + zram_set_obj_cgroup(zram, index, objcg); } zram_slot_unlock(zram, index); @@ -1581,6 +1612,8 @@ static int zram_recompress(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; + struct obj_cgroup *objcg = NULL; + unsigned int noreclaim_flag; u32 num_recomps = 0; void *src, *dst; int ret; @@ -1692,11 +1725,21 @@ static int zram_recompress(struct zram *zram, u32 index, struct page *page, zs_unmap_object(zram->mem_pool, handle_new); + /* + * Recompress will reclaim some memory, so we set the reclaim + * flag in order to charge comp_len_new successfully. + */ + noreclaim_flag = memalloc_noreclaim_save(); + objcg = zram_get_obj_cgroup(zram, index); + obj_cgroup_get(objcg); zram_free_page(zram, index); + obj_cgroup_charge_zram(objcg, GFP_KERNEL, comp_len_new); + zram_set_obj_cgroup(zram, index, objcg); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); zram_set_priority(zram, index, prio); + memalloc_noreclaim_restore(noreclaim_flag); atomic64_add(comp_len_new, &zram->stats.compr_data_size); atomic64_inc(&zram->stats.pages_stored); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index ca7a15bd4845..959d721d5474 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -72,6 +72,7 @@ struct zram_table_entry { #ifdef CONFIG_ZRAM_MEMORY_TRACKING ktime_t ac_time; #endif + struct obj_cgroup *objcg; }; struct zram_stats {