From patchwork Thu Jun 15 03:48:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 108248 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp378940vqr; Wed, 14 Jun 2023 20:59:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4DtNa+yM6SaGDp0rZ3WS2HzoX1I2k/GsinbadyClJUupz/voXraFbmDG7fkiluppV5u4d8 X-Received: by 2002:a05:6402:1108:b0:518:72f2:19c1 with SMTP id u8-20020a056402110800b0051872f219c1mr4613604edv.33.1686801553390; Wed, 14 Jun 2023 20:59:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686801553; cv=none; d=google.com; s=arc-20160816; b=W+EGsvOiga14nCfm2OmJQyec5kfIe/ob/UBIs34/yGTi/pTifkfqOuUF3O+/Mw4KMj yluE+vYGF4Q4wCJLLZabOYxebKOn9x04mgV56Hxud/hB3rOT+NtwgtBRLiBAYnlXYS/E CiubCYCC/2fhSjGZ4Wb43sIsIC9ZBnpmMRt5VQqExbMN1mv+Q8p2hq3cyVtCVH8NeovV T5Jv4rAy7dl8qEPg2AxQ+FKvRvyAw0vyaYnPh2ex6WOr1s1BRqTJffyMIKN6+QF5vMF5 eUEshB+FkTonNnyeKRLMJSHWA6IdY0acNeaD99nwRdAPtxzpgA2Db487qTbSgMaSK0Jp vXUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Vbu+qXDQAnJmw8qCM+u2edU9xMFzBPDexgHZg7upYSc=; b=RqUPJI/GLA1Ey3aURwaOWPRS0wuAZBNpBZGtV+EhlrNxTne/e97CyPApAV8ZxD0YOe EXMTB9YwHbx8363ApnPCkL/RDeW45K/XS98MUNWxaHOwtmYSKE9O7F+CabDwTeBnV/Ig DHjPbr8ybboz4P4vE0AcrWAfcnV/3hxj1ihIbQbrMulUD+gwLD4iBfYAJeujCcQSIjTd Wf+bavcU8cda958JKaI6dIbHwAevKStiU9tovjj8ZiW3PCe+nLCyTGS2VcXDkY5G8zb8 ciEFY2lj3XNIWVFS6D1FHPOR1GT04PcswLC/8xVUcocR/3WAFEQsrJKSrAYAb/GKvkfL fGdg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="QuB/9REr"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m22-20020aa7c496000000b004fd1f069ba6si9575999edq.560.2023.06.14.20.58.49; Wed, 14 Jun 2023 20:59:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b="QuB/9REr"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238078AbjFODtP (ORCPT + 99 others); Wed, 14 Jun 2023 23:49:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238023AbjFODtF (ORCPT ); Wed, 14 Jun 2023 23:49:05 -0400 Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 904BB211C for ; Wed, 14 Jun 2023 20:48:39 -0700 (PDT) Received: by mail-pg1-x535.google.com with SMTP id 41be03b00d2f7-54f75f85a17so2799973a12.0 for ; Wed, 14 Jun 2023 20:48:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1686800919; x=1689392919; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Vbu+qXDQAnJmw8qCM+u2edU9xMFzBPDexgHZg7upYSc=; b=QuB/9REryqq6Mgb8L/e4zMTLqzDoDXfDapT3W0Gx/mxECAPvMPdAXb/7xqeOhW6cOS rDSsPeeYrCYrv5l2lI+PFC6KhRdpynj5Nc5W68ZABRG0SVuM4SYMvOjbDQp9yiwdfuFg c3IlUzDHTM3Fe8qKMpcnHym1vs/Tc9pnPa/Z9VZjvYSWuOV+oc3k2SLfRMFyiHBgDHcf WE/hXQy+tuxSMe7jlfmDYO9nNIjXK8xkQy8hrUeKjS7HgAKMRd7kpWv9b+YE46IvVy/u dilXepFYsW49CWNtr+c9qK0KedUGDaNOr6gL61uSNoBrjIPF6bYFFyLINhJRBq0lk7Pf OxnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686800919; x=1689392919; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Vbu+qXDQAnJmw8qCM+u2edU9xMFzBPDexgHZg7upYSc=; b=bK4dYPrfZLqKdRmZUq/DJnaOhlYhdxjfVL+WNXldOaDxZvs/2cOnkB2xpPxZJsEnGz GP1AxdJmv8I1qBsF7z9MGI0fnevxg7iDq+JRxhk5nr1gwoYX6AMXoCXbYUZXLlNGq7Oj PNJYIleE4fAHneePEMpix22ycw9DkcJijQBbPnoMZDkWtar2fejZorFZlJzSIqUI8+gr 63l+OUKk1uxouyerpa8CsILTNTuM4LrOh5yI6Gm5fpJcXhhrdoo7OYM5+K6hxCr+yxjH yL+Dsl0X/eo6YYhkFJBSapRCfDcIcPa7w1i0XirtuBIiK0UY/8xveAgH+JfXICZre+9c Sw+A== X-Gm-Message-State: AC+VfDxo39bmN9NevON6odUb/o9YguSyNw3x86y/HcDTuHQS6XKv9FvT JEP4MqavyBiASKKoY2pBf74VdA== X-Received: by 2002:a05:6a21:29ca:b0:11d:d83b:8d80 with SMTP id tv10-20020a056a2129ca00b0011dd83b8d80mr2265679pzb.46.1686800919047; Wed, 14 Jun 2023 20:48:39 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id 19-20020a170902c11300b001b3fb1119fdsm2751713pli.297.2023.06.14.20.48.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jun 2023 20:48:38 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 1/3] zram: charge the compressed RAM to the page's memcgroup Date: Thu, 15 Jun 2023 11:48:30 +0800 Message-Id: <20230615034830.1361853-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768739625658917476?= X-GMAIL-MSGID: =?utf-8?q?1768739625658917476?= The compressed RAM is currently charged to kernel, not to any memory cgroup, which is not satisfy our usage scenario. if the memory of a task is limited by memcgroup, it will swap out the memory to zram swap device when the memory is insufficient. In that case, the memory limit will have no effect. So, it should makes sense to charge the compressed RAM to the page's memory cgroup. Signed-off-by: Zhongkun He --- drivers/block/zram/zram_drv.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f6d90f1ba5cf..03b508447473 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -33,6 +33,7 @@ #include #include #include +#include #include "zram_drv.h" @@ -1419,6 +1420,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) struct zcomp_strm *zstrm; unsigned long element = 0; enum zram_pageflags flags = 0; + struct mem_cgroup *memcg, *old_memcg; + + memcg = page_memcg(page); + old_memcg = set_active_memcg(memcg); mem = kmap_atomic(page); if (page_same_filled(mem, &element)) { @@ -1426,7 +1431,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) /* Free memory associated with this sector now. */ flags = ZRAM_SAME; atomic64_inc(&zram->stats.same_pages); - goto out; + goto out_free; } kunmap_atomic(mem); @@ -1440,7 +1445,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); - return ret; + goto out; } if (comp_len >= huge_class_size) @@ -1470,8 +1475,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); + if (IS_ERR_VALUE(handle)) { + ret = PTR_ERR((void *)handle); + goto out; + } if (comp_len != PAGE_SIZE) goto compress_again; @@ -1491,7 +1498,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (zram->limit_pages && alloced_pages > zram->limit_pages) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_free(zram->mem_pool, handle); - return -ENOMEM; + ret = -ENOMEM; + goto out; } dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); @@ -1506,7 +1514,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); atomic64_add(comp_len, &zram->stats.compr_data_size); -out: +out_free: /* * Free memory associated with this sector * before overwriting unused sectors. @@ -1531,6 +1539,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) /* Update stats */ atomic64_inc(&zram->stats.pages_stored); +out: + set_active_memcg(old_memcg); return ret; } From patchwork Thu Jun 15 03:48:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 108249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp381285vqr; Wed, 14 Jun 2023 21:04:37 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7yEGjMf3m3n4wF15x/FAOYghbQP03WpOnbebb4Ee4W+JKJgu6Le2yqNRHF45J2GGv4L7q2 X-Received: by 2002:a17:907:6da2:b0:973:7096:60c2 with SMTP id sb34-20020a1709076da200b00973709660c2mr3627833ejc.20.1686801877626; Wed, 14 Jun 2023 21:04:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686801877; cv=none; d=google.com; s=arc-20160816; b=x35HPWT8pqoe4RM/13Atgrz5zYeZ1OxOsIf33bCnnXuFUroSd7JS43Qobp/fNdrZDY ean6yRmHbY/9UrxYPDdocFA4mWkb8uRYBDCBlYdWqJOahZV1HfX062XdGQLRdLcdtZZd YqQMe4JNqbLXyXmqUFaysivk9HYmrsQnkQKYl2CLG6iH92tlu161ZmZCkP7hVNpYvzci FsDZ028m4tIj/Qb+r++8+oJqfncTliUUQMmebD6fFxXZ+tdPNOIBgb5Cg8QZkIUPpv2x Tei6gdYirTuBNBwXJFj+oRgbktXO7ky+xGEF2bEQtPBYBq2FZpegqeVVY4Y1VOJE1lM2 P0Yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=NRjYS/ADH2rjGpNFpYFXCA45Do2mTCUsIGO3/Z8z6+w=; b=yY19H53ItkCD67zB5GwCXT/oRxr620Hk9GqUYCT3WyhHABaAcg6uwiIaz8ONYA4M2W MZ2OirxGMwYvAiFpttvzaBgd86RPJEt8Yj8dQrKm7iYhmiwTXR7i7OP44D2fA8TLVBel Ei9xqjoTvBwvfYt3Nm9MzR1ru7VvgN2Jnh9uRUXQdcwYgOtKPmAP84Z3vjyQeoUxr4UT hzTbTH/RWa03jtBraSeffD6j/JAEqgrt3k4L85Lit1xJ+k3v7LbutG1E46PmdS59ebQI fW7CdkJEyQpZljauP1UNPuUmeYxrORx34ZjpSNuSHTH50/HXjjGswsUO5bXoIuqFEuE1 BFrA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=TR6OrR3f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o10-20020a1709061d4a00b00977d0f093b5si9357624ejh.556.2023.06.14.21.04.12; Wed, 14 Jun 2023 21:04:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=TR6OrR3f; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237875AbjFODu4 (ORCPT + 99 others); Wed, 14 Jun 2023 23:50:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241941AbjFODta (ORCPT ); Wed, 14 Jun 2023 23:49:30 -0400 Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1E2A2122 for ; Wed, 14 Jun 2023 20:48:59 -0700 (PDT) Received: by mail-qk1-x72a.google.com with SMTP id af79cd13be357-75e4497597cso275571585a.2 for ; Wed, 14 Jun 2023 20:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1686800939; x=1689392939; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=NRjYS/ADH2rjGpNFpYFXCA45Do2mTCUsIGO3/Z8z6+w=; b=TR6OrR3fFDYNkCl+qhYqAsxfyiD8kknZZjEleHF/oB/mIAtrHmUqxT5dh8hGkLm7x6 jETfLCXhDDQw3QbD/HPI3dKgkAtov4zpQaVmJPSO1erj3zR9xOsTFOp/lu60HJH6nsZ/ 4QyKhVFs6hTtM1Nv/n0pMN9cqjRyFBJ1twgQJkxpYVM0qUnn216Whur/qwooRbmn9AeT kof5U1V3m8rwug8zJXXUBQJk+6gYIZBF4L5DgmHRpFHd4qzoGdTpoEVL0cGg3oFV0vDF BLwKzNKm+0bR1j1KZrUOTSyLGxMdfiQRFInPi2+TrbQvj9I13/lac+ftr4xbOUYSQSGb ObmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686800939; x=1689392939; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NRjYS/ADH2rjGpNFpYFXCA45Do2mTCUsIGO3/Z8z6+w=; b=KnrhIS+HZlFvo+QDDzlo6ZhD+YArWf5DfkvlRpdxg4SCf6shY3+OF1C9LC0hq95vwi 9ZhGVCOIufDCSrQBXP55ccLMMIBvmsyGIZaJcxUHTP6k/FJE8iX6U8fkfZ65C169BR9Q Uu1C0Bi+DeJMLKyMTArCiCu+3/I1q5hfGKs8nu4RTv69nrWFTYh9HHLCL2f+mUnyifx6 vi+WSNNaznxPbdlbUIhuqrr6+CLMR/9Nmyhucs4Qa8CL8YjcRp1JK6/ogWwoUUM1SfL7 1diUIjF0s9o0J9Lg+YLoWe80Mb7WTj4UenMlSnYEVNKU9KSSLm7K6YcpyFeT4yZrZtyL enuw== X-Gm-Message-State: AC+VfDzs9yLiJshHMlGSr8qWdzVAf/4nHaw55SUqjpMEOrnI+zmpAE9a cr5XMlmCEoRSNCrdLNIKQriJkQ== X-Received: by 2002:a05:620a:28c8:b0:760:859a:d713 with SMTP id l8-20020a05620a28c800b00760859ad713mr15005467qkp.56.1686800938766; Wed, 14 Jun 2023 20:48:58 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id 21-20020aa79215000000b0064559b58eb8sm11012074pfo.154.2023.06.14.20.48.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jun 2023 20:48:58 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 2/3] memcg: export obj_cgroup* symbol to charge compressed RAM Date: Thu, 15 Jun 2023 11:48:49 +0800 Message-Id: <20230615034849.1361940-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768739965783734665?= X-GMAIL-MSGID: =?utf-8?q?1768739965783734665?= Export obj_cgroup* symbol to charge compressed memory in zsmalloc module. Signed-off-by: Zhongkun He --- mm/memcontrol.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..8ba93afacf31 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3064,6 +3064,7 @@ __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) rcu_read_unlock(); return objcg; } +EXPORT_SYMBOL(get_obj_cgroup_from_current); struct obj_cgroup *get_obj_cgroup_from_page(struct page *page) { @@ -3431,11 +3432,13 @@ int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) return ret; } +EXPORT_SYMBOL(obj_cgroup_charge); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) { refill_obj_stock(objcg, size, true); } +EXPORT_SYMBOL(obj_cgroup_uncharge); #endif /* CONFIG_MEMCG_KMEM */ From patchwork Thu Jun 15 03:49:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhongkun He X-Patchwork-Id: 108246 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp377395vqr; Wed, 14 Jun 2023 20:53:26 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4HeERp2cMkT300DoTjN3friLI8uu8i7ELorjqI0Mn9edqZNfyNR2AY7wgo1a0q6pCGmm8k X-Received: by 2002:a05:6a20:3d1d:b0:104:ffd0:232a with SMTP id y29-20020a056a203d1d00b00104ffd0232amr5021978pzi.8.1686801206606; Wed, 14 Jun 2023 20:53:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1686801206; cv=none; d=google.com; s=arc-20160816; b=PQbVg0vcXN6Cxz7w3AwjGc2O/T6CrmVLjTJBaLIHfreiiyVZz1Jv7wOoyWtP9h5uc3 sLSgYH9Tp9i1E1xnk54jkfeh8N1uiVfsOSfZy/RiVWW2SeT3aQPxy2GCNeepU9qu1w0j zDvTTYrD/1MiMioHeKGEEIUKlPJPDsqm4n59eFA+YRZ8H42LhMF+eByc59wHXdddto7o I75QE0F84xrphMhgrEybsu3INqrab4YVkJgnZiaRfZ6rvQSZ7BmSfBhe/O6ZJs388ynK RubdeDmHA4+jObEPgIVnN6vhR2/V8/uZUsdlDCGl+v2qCqHdadoJPBOJThXaNdM3zcl2 jD5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Ru9RQc/YctUd3oofmNo9B/wQzL/S7hlbQkvymAiMmdk=; b=cE64x1YTWpRIdi7+1B8KJ7vNxHYaBhq0O5DvXs1PloUyHQ860aBN0fi/raPGzaJmdv PcXeAmk7E930QQtSfmyOAQcw3gA8aYM4pCXkHoq2dhs+8qif+qzGkEUZj/CsIKp8UddL 2Kw+dZbYz6wNoQrIztfrwmwdx/WGoCgz8tt6TKfUi6grxVQnLRHmbjV5uDJbAeU1yr26 pxzrYsUfTpKwZ+3cbTT1f8GhGDjzbuMOQni0TKdKIE6+6Cj50bW/eFgNxMwhJsIS0Uha 79i8eXtSVbPRzyec69It09l8NbKt8aiUSkQSR+3W+yYWjM2Ep+8Ee6MsoF9tAa6IkGgl SEfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=RFOEvPUJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p17-20020a639511000000b005369f4111aesi11742107pgd.849.2023.06.14.20.53.13; Wed, 14 Jun 2023 20:53:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=RFOEvPUJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236229AbjFODuJ (ORCPT + 99 others); Wed, 14 Jun 2023 23:50:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238184AbjFODti (ORCPT ); Wed, 14 Jun 2023 23:49:38 -0400 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00867211C for ; Wed, 14 Jun 2023 20:49:12 -0700 (PDT) Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-662f0feafb2so265043b3a.1 for ; Wed, 14 Jun 2023 20:49:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1686800952; x=1689392952; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Ru9RQc/YctUd3oofmNo9B/wQzL/S7hlbQkvymAiMmdk=; b=RFOEvPUJE1xXVdLOmJQw4AYx9wGU3W9LSe9MDMMyjAwqI9uTfSl9XBmBC6yV1Yi4Kh H5ihk+wgHHlCnOOdlsK+YQIKHKcsa0sxfObaLcDMztLxyRgjxloUdUKjfcaCc5G09DE4 T1ZtX0M7w/f+A68B0vXL/ZEVD92E9KMiC/kc5Ymc3uTz6/eN3y0FCb3l4BVRWFuozSP1 zpZda+7WfBf2P6fjRmGIp0NBM73LgZXiCoJsxszp3yFdP3mEa7LVogULdTubmA4Z8Va2 dcEwfzIurYedtwnU1JpEKfxiGNiABEwM5APuMvNbbDATP0bRmEYfjh+gAcGsup4LFKbK uUAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686800952; x=1689392952; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Ru9RQc/YctUd3oofmNo9B/wQzL/S7hlbQkvymAiMmdk=; b=fVPWcJOatRvOAgI11NHMreL28SuxaC/f8P5wCEDdk68Qai6hYZMFdukAiSfsttfvF0 Y9sr2mOiiZfDRs1hcP1C6mP2ftAcOuZO3WZ1K6icsvgJV8egyISsXgih+b+NBctzLuHb ce7VqpMDXyKJ8sybunDM7vtr7WNK7XfuEJhdLma9LA9aqg6yCUxjRiLW1g/B9cY+N7ef ZMxa4EyAneAwTpHw3MsqRgqQ3390xzeADllvDuvyD3GohZxXibu5M/aXlV2MeV01qkv7 rBVWDu5mPs3o8EHfKVRTudZ5tnXnLYxrzH4zknNWtYNsA1JjJfBoXsvUcC7KCXG1zyBA +vow== X-Gm-Message-State: AC+VfDzcx/yq1kal2ZRa7WODHeUGBZWuDuJ6Ea1ihfuyOAFRfaQr4A21 /2Qvwj2oGHx2fouDh19PRzJrAA== X-Received: by 2002:a05:6a00:2e17:b0:650:1a64:d8d3 with SMTP id fc23-20020a056a002e1700b006501a64d8d3mr4983241pfb.14.1686800952295; Wed, 14 Jun 2023 20:49:12 -0700 (PDT) Received: from Tower.bytedance.net ([203.208.167.147]) by smtp.gmail.com with ESMTPSA id u6-20020a62ed06000000b00661694f1469sm6090369pfh.74.2023.06.14.20.49.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Jun 2023 20:49:11 -0700 (PDT) From: Zhongkun He To: minchan@kernel.org, senozhatsky@chromium.org, mhocko@suse.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Zhongkun He Subject: [RFC PATCH 3/3] zsmalloc: charge the zspage's object to obj_cgroup Date: Thu, 15 Jun 2023 11:49:05 +0800 Message-Id: <20230615034905.1362034-1-hezhongkun.hzk@bytedance.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1768739261979968731?= X-GMAIL-MSGID: =?utf-8?q?1768739261979968731?= The zspage's object is not charge to any memory cgroup.The memory of a task will have no limit if the zram swap device is large enough even though the task is limited by memory cgroup. So, it should be necessary to charge the zspage's object to obj_cgroup,just like slub. Signed-off-by: Zhongkun He --- mm/zsmalloc.c | 196 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 191 insertions(+), 5 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 02f7f414aade..a5d1c484dd8f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -62,6 +62,7 @@ #include #include #include +#include #define ZSPAGE_MAGIC 0x58 @@ -311,6 +312,7 @@ static bool ZsHugePage(struct zspage *zspage) static void migrate_lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); +static void zs_free_page_obj_cgroups(struct zspage *zspage); #ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage); @@ -973,6 +975,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, free_handles(pool, class, zspage); next = page = get_first_page(zspage); + zs_free_page_obj_cgroups(zspage); do { VM_BUG_ON_PAGE(!PageLocked(page), page); next = get_next_page(page); @@ -1420,6 +1423,170 @@ size_t zs_huge_class_size(struct zs_pool *pool) } EXPORT_SYMBOL_GPL(zs_huge_class_size); + +#ifdef CONFIG_MEMCG_KMEM + +static inline struct obj_cgroup **page_objcgs_check(struct page *page) +{ + unsigned long memcg_data = READ_ONCE(page->memcg_data); + + if (!memcg_data || !(memcg_data & MEMCG_DATA_OBJCGS)) + return NULL; + + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_KMEM, page); + + return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} + + +static void zs_free_page_obj_cgroups(struct zspage *zspage) +{ + struct obj_cgroup **objcgs; + struct page *page = zspage->first_page; + + objcgs = page_objcgs_check(page); + if (!objcgs) + return; + + kfree(objcgs); + page->memcg_data = 0; +} + +static inline size_t zs_obj_full_size(struct size_class *class) +{ + return class->size + sizeof(struct obj_cgroup *); +} + +static inline void zs_objcg_exchange(struct page *s_page, unsigned int s_index, + struct page *d_page, unsigned int d_index) +{ + struct page *sf_page, *df_page; + struct obj_cgroup **s_objcgs, **d_objcgs; + + sf_page = get_first_page(get_zspage(s_page)); + df_page = get_first_page(get_zspage(d_page)); + + s_objcgs = page_objcgs_check(sf_page); + d_objcgs = page_objcgs_check(df_page); + + if (!s_objcgs || !d_objcgs) + return; + + d_objcgs[d_index] = s_objcgs[s_index]; + s_objcgs[s_index] = NULL; +} + +void zs_alloc_obj_cgroups(struct zspage *zspage, struct size_class *class) +{ + struct page *page = zspage->first_page; + int objects = class->objs_per_zspage; + unsigned long memcg_data = 0; + void *vec; + + vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), + GFP_NOWAIT|__GFP_NOWARN| + __GFP_ZERO|__GFP_MOVABLE, + page_to_nid(page)); + + if (vec) + memcg_data = (unsigned long)vec | MEMCG_DATA_OBJCGS; + page->memcg_data = memcg_data; +} + +static inline bool zs_obj_pre_alloc_hook(struct size_class *class, + struct obj_cgroup **objcgp, gfp_t flags) +{ + struct obj_cgroup *objcg; + + *objcgp = NULL; + + objcg = get_obj_cgroup_from_current(); + if (!objcg) + return true; + + if (obj_cgroup_charge(objcg, flags, zs_obj_full_size(class))) { + obj_cgroup_put(objcg); + return false; + } + + *objcgp = objcg; + return true; +} + +static inline void zs_obj_post_alloc_hook(struct size_class *class, + struct zspage *zspage, + unsigned int idx, + struct obj_cgroup *objcg) +{ + struct obj_cgroup **objcgs = NULL; + + if (!objcg) + return; + + if (zspage) + objcgs = page_objcgs_check(zspage->first_page); + + if (!objcgs) { + obj_cgroup_uncharge(objcg, zs_obj_full_size(class)); + obj_cgroup_put(objcg); + return; + } + + objcgs[idx] = objcg; +} + +static inline void zs_obj_free_hook(int class_size, struct zspage *zspage, + unsigned int idx) +{ + struct obj_cgroup **objcgs, *objcg; + + objcgs = page_objcgs_check(zspage->first_page); + + if (WARN_ON(!objcgs)) + return; + + objcg = objcgs[idx]; + if (!objcg) + return; + + objcgs[idx] = NULL; + obj_cgroup_uncharge(objcg, class_size + sizeof(struct obj_cgroup *)); + obj_cgroup_put(objcg); +} +#else + +static void zs_free_page_obj_cgroups(struct zspage *zspage) +{ +} + +static inline void zs_objcg_exchange(struct page *s_page, unsigned int s_index, + struct page *d_page, unsigned int d_index) +{ +} + +int zs_alloc_obj_cgroups(struct zspage *zspage, struct size_class *class) +{ + return 0; +} + +static inline bool zs_obj_pre_alloc_hook(struct size_class *class, + struct obj_cgroup **objcgp, gfp_t flags) +{ + return true; +} + +static inline void zs_obj_post_alloc_hook(struct size_class *class, + struct page *page, + unsigned int idx, + struct obj_cgroup *objcg) +{ +} +static inline void zs_obj_free_hook(int class_size, struct zspage *zspage, + unsigned int idx) +{ +} +#endif + static unsigned long obj_malloc(struct zs_pool *pool, struct zspage *zspage, unsigned long handle) { @@ -1475,7 +1642,8 @@ static unsigned long obj_malloc(struct zs_pool *pool, */ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) { - unsigned long handle, obj; + unsigned long handle, obj, index; + struct obj_cgroup *objcg; struct size_class *class; int newfg; struct zspage *zspage; @@ -1491,17 +1659,23 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) size += ZS_HANDLE_SIZE; class = pool->size_class[get_size_class_index(size)]; + if (!zs_obj_pre_alloc_hook(class, &objcg, gfp)) { + cache_free_handle(pool, handle); + return (unsigned long)ERR_PTR(-ENOMEM); + } + /* pool->lock effectively protects the zpage migration */ spin_lock(&pool->lock); zspage = find_get_zspage(class); if (likely(zspage)) { + index = get_freeobj(zspage); obj = obj_malloc(pool, zspage, handle); /* Now move the zspage to another fullness group, if required */ fix_fullness_group(class, zspage); record_obj(handle, obj); class_stat_inc(class, ZS_OBJS_INUSE, 1); - goto out; + goto out_unlock; } spin_unlock(&pool->lock); @@ -1509,9 +1683,12 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) zspage = alloc_zspage(pool, class, gfp); if (!zspage) { cache_free_handle(pool, handle); - return (unsigned long)ERR_PTR(-ENOMEM); + handle = (unsigned long)ERR_PTR(-ENOMEM); + goto out; } + zs_alloc_obj_cgroups(zspage, class); + index = get_freeobj(zspage); spin_lock(&pool->lock); obj = obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); @@ -1524,7 +1701,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); -out: +out_unlock: #ifdef CONFIG_ZPOOL /* Add/move zspage to beginning of LRU */ if (!list_empty(&zspage->lru)) @@ -1533,6 +1710,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) #endif spin_unlock(&pool->lock); +out: + zs_obj_post_alloc_hook(class, zspage, index, objcg); return handle; } @@ -1573,7 +1752,7 @@ static void obj_free(int class_size, unsigned long obj, unsigned long *handle) f_page->index = 0; set_freeobj(zspage, f_objidx); } - + zs_obj_free_hook(class_size, zspage, f_objidx); kunmap_atomic(vaddr); mod_zspage_inuse(zspage, -1); } @@ -1640,6 +1819,7 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, obj_to_location(src, &s_page, &s_objidx); obj_to_location(dst, &d_page, &d_objidx); + zs_objcg_exchange(s_page, s_objidx, d_page, d_objidx); s_off = (class->size * s_objidx) & ~PAGE_MASK; d_off = (class->size * d_objidx) & ~PAGE_MASK; @@ -1960,6 +2140,12 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage, int idx = 0; page = get_first_page(zspage); +#ifdef CONFIG_MEMCG_KMEM + if (page == oldpage) { + newpage->memcg_data = page->memcg_data; + page->memcg_data = 0; + } +#endif do { if (page == oldpage) pages[idx] = newpage;