From patchwork Sat Nov 19 00:15:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 23153 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp500206wrr; Fri, 18 Nov 2022 17:20:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf7LGUy7NkylLMjGEIoyRpDdICh5Y6OMMcgUrTvYw5NC+BfJwEXKDgkeVYq2eW3wWEIjcP0J X-Received: by 2002:a17:906:dcc:b0:7ad:69eb:90c3 with SMTP id p12-20020a1709060dcc00b007ad69eb90c3mr8237165eji.285.1668820804303; Fri, 18 Nov 2022 17:20:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668820804; cv=none; d=google.com; s=arc-20160816; b=fp3M/0eJ8OcVs1TA3tYChQku372FQugI8KyBaitjIBhyvfNDgFIWjqBZz/gmo4LLWH rJJE2kifEt7Nq1cKArnoXYwVFj3PWBQkNuANzYm4ekGU74hwxdERKyybGp4zJ/0hyRVV z8CaS2jOTd3SsvmlsrLi7tETiyMQ+5WrcyCqw+ekxc/HQCr+pjHMTv1xKYGw5fWS5sYM MghH2zIxwt+x4p+FMnaqe3aS0OtKQ1YK5IJ3hQKtLL2a2Wzak/khQpjFMIRRPw78c6en SMYVEkTAKIqVw2uyWqldsq/hDejv2Xe+8GNe3HlrhJws4dfrYFtIvAfsTmXy5cUn6YqY tWyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=uhF53/xh3Hyv0X6zFlismYLwseOE8BPL7QAyLJFmpTSf8IuO/fFKgTY1stcJpxyVjh CMB3PJwaUmSRy+gCptamFjvbeNt4o6rrwfU2OhpdgeFqafwExDiSYfNhKOhFa8Zy+RVQ t1p3V+EqyaOamflKM6F5VP6pYVznXc0TKpKR0R9P3RgQn+67C/TqTb3pWtqEQNa2Z2QL kbaN3Goa9y3DBuU5y5I89SnNHMWwts0yz83gJmBTcRYcWFhXNzwkQNSNfyvmgqtLNr14 6ruBuZiP3juIp6IsfF224spdYAN4ZhmRsUJFM6FZvJvbREgcCQ44RC7erdS+omTnonKy lGGQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=NHkIkh8R; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id et19-20020a170907295300b0078db6f5f4b2si905612ejc.546.2022.11.18.17.19.40; Fri, 18 Nov 2022 17:20:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=NHkIkh8R; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229862AbiKSBSj (ORCPT + 99 others); Fri, 18 Nov 2022 20:18:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231645AbiKSBSM (ORCPT ); Fri, 18 Nov 2022 20:18:12 -0500 Received: from mail-pj1-x1030.google.com (mail-pj1-x1030.google.com [IPv6:2607:f8b0:4864:20::1030]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCF61169C19 for ; Fri, 18 Nov 2022 16:15:39 -0800 (PST) Received: by mail-pj1-x1030.google.com with SMTP id v4-20020a17090a088400b00212cb0ed97eso6500902pjc.5 for ; Fri, 18 Nov 2022 16:15:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=NHkIkh8RaR25rj7+Wlw0rbe1vzjooV8Fsu/f07L3zCG5lUFzcdTGGBtND54rMrvFn2 1nztmuMnmKeDkLPZsShRcJCn8KfZDSS2xoMxkJ+8eWbfouubZv8PETaQAHxCEEQyuRcP 7dtpmnLSJFtricXu29dRABoMD0uTm0Xv+9F3R8Z0zOZfzFHEh/jZfd89vph1ur+jnvoi qe4O0YDtzONH4Dze1Yr/u8DHfwdnbozJEOewCLH0RyGcNrdx/Q/rLMIXB5yDuzL+iHp+ RKiSsuMYn6PCIoOSsyr2vmZRFTqbXCde8fgnZhLJQEkH9ydslMP7E4Ww+Oh0mh53K7e2 lK5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=G5R/8R7YXRr8brXvoLTTrFp6NL8YOyaF9a3oOeBBVQesgcjrHbvBd06hEDdKntOA85 PqaxQkBshUbcCwNnC756/KHa+oOLas/kvw0wBgPHuUDTZMHxq18NcWuZdslxP1gVPoBP qetMNhRozlBedAVIifm5p7mvgG2cTDBjHgWkmfH/SFHRNaRmSdDPkZERo8Lo581c++Qr y+hbtPmQvlVdMNja+L8Yh9qCJkSyp8KqTZepd2UMM6QHWv6Y+24scRW4SJ/2uKz2d3ih x74wIYldEZkfHLvZTmYUy0K3H5I3LwHujwNGf8Mg7XUN+OB7iQMl5EuS86YenVzKtugr SUgw== X-Gm-Message-State: ANoB5pmD5A42mbvOeA23Wr5wq7b+bLsKiBUIQufB6fqefL7K3FxfHArz GnkuTwbRcDBE0uZXHUp5aCI= X-Received: by 2002:a17:902:9a03:b0:186:9f20:e7e2 with SMTP id v3-20020a1709029a0300b001869f20e7e2mr1578912plp.174.1668816939044; Fri, 18 Nov 2022 16:15:39 -0800 (PST) Received: from localhost (fwdproxy-prn-010.fbsv.net. [2a03:2880:ff:a::face:b00c]) by smtp.gmail.com with ESMTPSA id x22-20020a170902821600b0017f48a9e2d6sm4202332pln.292.2022.11.18.16.15.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:38 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 1/6] zswap: fix writeback lock ordering for zsmalloc Date: Fri, 18 Nov 2022 16:15:31 -0800 Message-Id: <20221119001536.2086599-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749885443750716068?= X-GMAIL-MSGID: =?utf-8?q?1749885443750716068?= From: Johannes Weiner zswap's customary lock order is tree->lock before pool->lock, because the tree->lock protects the entries' refcount, and the free callbacks in the backends acquire their respective pool locks to dispatch the backing object. zsmalloc's map callback takes the pool lock, so zswap must not grab the tree->lock while a handle is mapped. This currently only happens during writeback, which isn't implemented for zsmalloc. In preparation for it, move the tree->lock section out of the mapped entry section Signed-off-by: Johannes Weiner Signed-off-by: Nhat Pham --- mm/zswap.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) -- 2.30.2 diff --git a/mm/zswap.c b/mm/zswap.c index 2d48fd59cc7a..2d69c1d678fe 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -958,7 +958,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) }; if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_ATOMIC); + tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); if (!tmp) return -ENOMEM; } @@ -968,6 +968,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) swpentry = zhdr->swpentry; /* here */ tree = zswap_trees[swp_type(swpentry)]; offset = swp_offset(swpentry); + zpool_unmap_handle(pool, handle); /* find and ref zswap entry */ spin_lock(&tree->lock); @@ -975,20 +976,12 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) if (!entry) { /* entry was invalidated */ spin_unlock(&tree->lock); - zpool_unmap_handle(pool, handle); kfree(tmp); return 0; } spin_unlock(&tree->lock); BUG_ON(offset != entry->offset); - src = (u8 *)zhdr + sizeof(struct zswap_header); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; - zpool_unmap_handle(pool, handle); - } - /* try to allocate swap cache page */ switch (zswap_get_swap_cache_page(swpentry, &page)) { case ZSWAP_SWAPCACHE_FAIL: /* no memory or invalidate happened */ @@ -1006,6 +999,14 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + zhdr = zpool_map_handle(pool, handle, ZPOOL_MM_RO); + src = (u8 *)zhdr + sizeof(struct zswap_header); + if (!zpool_can_sleep_mapped(pool)) { + memcpy(tmp, src, entry->length); + src = tmp; + zpool_unmap_handle(pool, handle); + } + mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); @@ -1015,6 +1016,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); + else + zpool_unmap_handle(pool, handle); + BUG_ON(ret); BUG_ON(dlen != PAGE_SIZE); @@ -1045,7 +1051,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - goto end; + return ret; + +fail: + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); /* * if we get here due to ZSWAP_SWAPCACHE_EXIST @@ -1054,17 +1064,10 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) * if we free the entry in the following put * it is also okay to return !0 */ -fail: spin_lock(&tree->lock); zswap_entry_put(tree, entry); spin_unlock(&tree->lock); -end: - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, handle); - else - kfree(tmp); - return ret; } From patchwork Sat Nov 19 00:15:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 23154 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp500227wrr; Fri, 18 Nov 2022 17:20:09 -0800 (PST) X-Google-Smtp-Source: AA0mqf50e50rHq2mlEE0oZpyfzTiv0Jj9nO/XFa9F4a8/kLhKT9VwuW+RvAxaYkRR8yXyiDLWdHe X-Received: by 2002:aa7:cc12:0:b0:462:79ec:55eb with SMTP id q18-20020aa7cc12000000b0046279ec55ebmr8138725edt.151.1668820808942; Fri, 18 Nov 2022 17:20:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668820808; cv=none; d=google.com; s=arc-20160816; b=GGuvcbBfHGRpuLdVYj8pwX3IvUZ37w6MfRCvHUz3r/NrELFJJkLP3Q0XYcHy1/jdKk BqIx83owCh5c6rxR7gAUU1t8sxO7Kg6IrztqBJYXNdHv0E9OT53ksrUs3ehYBo/LWGxa uD0BKQR2xkXQ2oNBbsdTEzBJqKuCLybur54ulQPHafJl6fDmBLq029/k1iYnzCQd9OW5 /zrTQd7aZf3T14r+tgpNG+vuq7idk7A3jWDCpzuDrVwFKE6Y3Zf+Kv3c2WUEbX0kKtVM aIeObjkVLjh6jkdj4M6HAK65VlB/NTXi1t0THMEE+w8YL/QbuqQZ2rkC2S0GvzdI1/+B qgEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=W2KftAIfbSAVX07/SXuM65Rf8D45dV1bph1a5dZLbhOpXzvmLMb7Dkotv5+cCpB1uv 60howkR+lZnyG3Ete3FeV+SeeLhQW2rfQgiHUJeY2mKRkq9HzvjG6cEQKyLIDo+MIkEt etD1wDOB/+aLOlVKwVBxJ7r+FyuxS5RGOJ9pmMIdOw8we9atVNnf/FN2x7pPvZ4FphLs 7sPATCd1GGiaA3syoxbxalvcY2hEdJEfOSv8Kx0AbX8Z2hxHrvGlCh4C5ev3bjWVfuAn oczpTbkt7n4TQNT+vM5f2NIFuc9vs1ByQ8EicmiqfUWfJO+ys6/ycCXujY9isYHGlhE1 /+Pw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=hM3jHRqv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t21-20020a056402525500b00458d1c48708si4730401edd.303.2022.11.18.17.19.45; Fri, 18 Nov 2022 17:20:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=hM3jHRqv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230378AbiKSBSv (ORCPT + 99 others); Fri, 18 Nov 2022 20:18:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231723AbiKSBSM (ORCPT ); Fri, 18 Nov 2022 20:18:12 -0500 Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9174F6080 for ; Fri, 18 Nov 2022 16:15:40 -0800 (PST) Received: by mail-pg1-x530.google.com with SMTP id r18so6292716pgr.12 for ; Fri, 18 Nov 2022 16:15:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=hM3jHRqvmUOS23MLnToWW+gukgJ5URhKESdrNtGSY70GCQExpAwFM4v8mLrYILYaSZ o7BlEmhMWruCgk2DNQEhjWp6+CFLP4WnY4FSnnCDBXkEgUoBQP+CRxtiApMY0Aud0C2t EOAfqJTL/pd8k9qLlLGBPfkhUD3vI5oOREPErhNl0UTQw8UvuN6mF3hAo4ZUPUI7hNYm 48WftE7CwCRH0WCskYNEGYJBJPbP7yaZZcErJTzxF4cwXAbqBCC2WNNAA6HTXenZwdbs 9+gdXyCK1H7c40kVRKKR+p3ZqeaaIV+1WGtneXuim+vXKl900/Lh2B7/lEu4RH8ME2um zOgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=o8/s5JwoSOX+4jGEX1hZVuSH6kYo4Mes4Alb1zBZzJHizawexj8jjMM+5eCSp+sf2W wO24JgFKHLsd3ClqwHJ5yOqQJrC5qojbNRq/zgmHmhL61V2AR0xfRa3O58JSNUbvhC6V G4MOduWXHr5Q++hzASTdNBsOpUL4HVgcDfc7izP6kUtXBX1T4FGtGRBm26KdyZJk89oe BrFt+HmsOP/Scyq2Qt1hh8QDidZ+MGol5LtoKhkwoWMeslMvBiJtdi8NYRBq7B+Wie1L 5FGk/JsxNpCFB012hUHcfi8cDAasb7R4AvDMyILKqc9jtzugcQPZ102xMmyo2mqPq/jc IqUg== X-Gm-Message-State: ANoB5pmXBDGT5ejkIMFcUqHcxjoLWJU3m1cdpfx9F1arv921Vp+XRL5T ZbQEWbw/gOQFLUmEq8lIbJ4= X-Received: by 2002:a63:4d7:0:b0:46e:d8af:9ff6 with SMTP id 206-20020a6304d7000000b0046ed8af9ff6mr9030196pge.98.1668816940285; Fri, 18 Nov 2022 16:15:40 -0800 (PST) Received: from localhost (fwdproxy-prn-014.fbsv.net. [2a03:2880:ff:e::face:b00c]) by smtp.gmail.com with ESMTPSA id h13-20020a170902680d00b001837b19ebb8sm4259585plk.244.2022.11.18.16.15.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:39 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 2/6] zpool: clean out dead code Date: Fri, 18 Nov 2022 16:15:32 -0800 Message-Id: <20221119001536.2086599-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749885448569970865?= X-GMAIL-MSGID: =?utf-8?q?1749885448569970865?= From: Johannes Weiner There is a lot of provision for flexibility that isn't actually needed or used. Zswap (the only zpool user) always passes zpool_ops with an .evict method set. The backends who reclaim only do so for zswap, so they can also directly call zpool_ops without indirection or checks. Finally, there is no need to check the retries parameters and bail with -EINVAL in the reclaim function, when that's called just a few lines below with a hard-coded 8. There is no need to duplicate the evictable and sleep_mapped attrs from the driver in zpool_ops. Signed-off-by: Johannes Weiner Signed-off-by: Nhat Pham Reviewed-by: Sergey Senozhatsky --- mm/z3fold.c | 36 +++++------------------------------- mm/zbud.c | 32 +++++--------------------------- mm/zpool.c | 10 ++-------- 3 files changed, 12 insertions(+), 66 deletions(-) -- 2.30.2 diff --git a/mm/z3fold.c b/mm/z3fold.c index cf71da10d04e..a4de0c317ac7 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -68,9 +68,6 @@ * Structures *****************/ struct z3fold_pool; -struct z3fold_ops { - int (*evict)(struct z3fold_pool *pool, unsigned long handle); -}; enum buddy { HEADLESS = 0, @@ -138,8 +135,6 @@ struct z3fold_header { * @stale: list of pages marked for freeing * @pages_nr: number of z3fold pages in the pool. * @c_handle: cache for z3fold_buddy_slots allocation - * @ops: pointer to a structure of user defined operations specified at - * pool creation time. * @zpool: zpool driver * @zpool_ops: zpool operations structure with an evict callback * @compact_wq: workqueue for page layout background optimization @@ -158,7 +153,6 @@ struct z3fold_pool { struct list_head stale; atomic64_t pages_nr; struct kmem_cache *c_handle; - const struct z3fold_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; struct workqueue_struct *compact_wq; @@ -907,13 +901,11 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, * z3fold_create_pool() - create a new z3fold pool * @name: pool name * @gfp: gfp flags when allocating the z3fold pool structure - * @ops: user-defined operations for the z3fold pool * * Return: pointer to the new z3fold pool or NULL if the metadata allocation * failed. */ -static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, - const struct z3fold_ops *ops) +static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp) { struct z3fold_pool *pool = NULL; int i, cpu; @@ -949,7 +941,6 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, if (!pool->release_wq) goto out_wq; INIT_WORK(&pool->work, free_pages_work); - pool->ops = ops; return pool; out_wq: @@ -1230,10 +1221,6 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) slots.pool = (unsigned long)pool | (1 << HANDLES_NOFREE); spin_lock(&pool->lock); - if (!pool->ops || !pool->ops->evict || retries == 0) { - spin_unlock(&pool->lock); - return -EINVAL; - } for (i = 0; i < retries; i++) { if (list_empty(&pool->lru)) { spin_unlock(&pool->lock); @@ -1319,17 +1306,17 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) } /* Issue the eviction callback(s) */ if (middle_handle) { - ret = pool->ops->evict(pool, middle_handle); + ret = pool->zpool_ops->evict(pool->zpool, middle_handle); if (ret) goto next; } if (first_handle) { - ret = pool->ops->evict(pool, first_handle); + ret = pool->zpool_ops->evict(pool->zpool, first_handle); if (ret) goto next; } if (last_handle) { - ret = pool->ops->evict(pool, last_handle); + ret = pool->zpool_ops->evict(pool->zpool, last_handle); if (ret) goto next; } @@ -1593,26 +1580,13 @@ static const struct movable_operations z3fold_mops = { * zpool ****************/ -static int z3fold_zpool_evict(struct z3fold_pool *pool, unsigned long handle) -{ - if (pool->zpool && pool->zpool_ops && pool->zpool_ops->evict) - return pool->zpool_ops->evict(pool->zpool, handle); - else - return -ENOENT; -} - -static const struct z3fold_ops z3fold_zpool_ops = { - .evict = z3fold_zpool_evict -}; - static void *z3fold_zpool_create(const char *name, gfp_t gfp, const struct zpool_ops *zpool_ops, struct zpool *zpool) { struct z3fold_pool *pool; - pool = z3fold_create_pool(name, gfp, - zpool_ops ? &z3fold_zpool_ops : NULL); + pool = z3fold_create_pool(name, gfp); if (pool) { pool->zpool = zpool; pool->zpool_ops = zpool_ops; diff --git a/mm/zbud.c b/mm/zbud.c index 6348932430b8..3acd26193920 100644 --- a/mm/zbud.c +++ b/mm/zbud.c @@ -74,10 +74,6 @@ struct zbud_pool; -struct zbud_ops { - int (*evict)(struct zbud_pool *pool, unsigned long handle); -}; - /** * struct zbud_pool - stores metadata for each zbud pool * @lock: protects all pool fields and first|last_chunk fields of any @@ -90,8 +86,6 @@ struct zbud_ops { * @lru: list tracking the zbud pages in LRU order by most recently * added buddy. * @pages_nr: number of zbud pages in the pool. - * @ops: pointer to a structure of user defined operations specified at - * pool creation time. * @zpool: zpool driver * @zpool_ops: zpool operations structure with an evict callback * @@ -110,7 +104,6 @@ struct zbud_pool { }; struct list_head lru; u64 pages_nr; - const struct zbud_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; }; @@ -212,12 +205,11 @@ static int num_free_chunks(struct zbud_header *zhdr) /** * zbud_create_pool() - create a new zbud pool * @gfp: gfp flags when allocating the zbud pool structure - * @ops: user-defined operations for the zbud pool * * Return: pointer to the new zbud pool or NULL if the metadata allocation * failed. */ -static struct zbud_pool *zbud_create_pool(gfp_t gfp, const struct zbud_ops *ops) +static struct zbud_pool *zbud_create_pool(gfp_t gfp) { struct zbud_pool *pool; int i; @@ -231,7 +223,6 @@ static struct zbud_pool *zbud_create_pool(gfp_t gfp, const struct zbud_ops *ops) INIT_LIST_HEAD(&pool->buddied); INIT_LIST_HEAD(&pool->lru); pool->pages_nr = 0; - pool->ops = ops; return pool; } @@ -419,8 +410,7 @@ static int zbud_reclaim_page(struct zbud_pool *pool, unsigned int retries) unsigned long first_handle = 0, last_handle = 0; spin_lock(&pool->lock); - if (!pool->ops || !pool->ops->evict || list_empty(&pool->lru) || - retries == 0) { + if (list_empty(&pool->lru)) { spin_unlock(&pool->lock); return -EINVAL; } @@ -444,12 +434,12 @@ static int zbud_reclaim_page(struct zbud_pool *pool, unsigned int retries) /* Issue the eviction callback(s) */ if (first_handle) { - ret = pool->ops->evict(pool, first_handle); + ret = pool->zpool_ops->evict(pool->zpool, first_handle); if (ret) goto next; } if (last_handle) { - ret = pool->ops->evict(pool, last_handle); + ret = pool->zpool_ops->evict(pool->zpool, last_handle); if (ret) goto next; } @@ -524,25 +514,13 @@ static u64 zbud_get_pool_size(struct zbud_pool *pool) * zpool ****************/ -static int zbud_zpool_evict(struct zbud_pool *pool, unsigned long handle) -{ - if (pool->zpool && pool->zpool_ops && pool->zpool_ops->evict) - return pool->zpool_ops->evict(pool->zpool, handle); - else - return -ENOENT; -} - -static const struct zbud_ops zbud_zpool_ops = { - .evict = zbud_zpool_evict -}; - static void *zbud_zpool_create(const char *name, gfp_t gfp, const struct zpool_ops *zpool_ops, struct zpool *zpool) { struct zbud_pool *pool; - pool = zbud_create_pool(gfp, zpool_ops ? &zbud_zpool_ops : NULL); + pool = zbud_create_pool(gfp); if (pool) { pool->zpool = zpool; pool->zpool_ops = zpool_ops; diff --git a/mm/zpool.c b/mm/zpool.c index 68facc193496..fc3a9893e107 100644 --- a/mm/zpool.c +++ b/mm/zpool.c @@ -21,9 +21,6 @@ struct zpool { struct zpool_driver *driver; void *pool; - const struct zpool_ops *ops; - bool evictable; - bool can_sleep_mapped; }; static LIST_HEAD(drivers_head); @@ -177,9 +174,6 @@ struct zpool *zpool_create_pool(const char *type, const char *name, gfp_t gfp, zpool->driver = driver; zpool->pool = driver->create(name, gfp, ops, zpool); - zpool->ops = ops; - zpool->evictable = driver->shrink && ops && ops->evict; - zpool->can_sleep_mapped = driver->sleep_mapped; if (!zpool->pool) { pr_err("couldn't create %s pool\n", type); @@ -380,7 +374,7 @@ u64 zpool_get_total_size(struct zpool *zpool) */ bool zpool_evictable(struct zpool *zpool) { - return zpool->evictable; + return zpool->driver->shrink; } /** @@ -391,7 +385,7 @@ bool zpool_evictable(struct zpool *zpool) */ bool zpool_can_sleep_mapped(struct zpool *zpool) { - return zpool->can_sleep_mapped; + return zpool->driver->sleep_mapped; } MODULE_LICENSE("GPL"); From patchwork Sat Nov 19 00:15:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 23155 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp500381wrr; Fri, 18 Nov 2022 17:20:35 -0800 (PST) X-Google-Smtp-Source: AA0mqf5fpvzhOmwRDWSS2YYrEBW1MIOfHlakYNQQV94WjxpYdhxtYrQofaCzWvgBrAbSfCgO6PSD X-Received: by 2002:a17:906:a10c:b0:78d:8bcf:bb47 with SMTP id t12-20020a170906a10c00b0078d8bcfbb47mr8131894ejy.525.1668820835406; Fri, 18 Nov 2022 17:20:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668820835; cv=none; d=google.com; s=arc-20160816; b=cGb9a9TMghUKPvC5RC9+PzLYSZMhYqhoTt+1LlQx3AFf9QKZgvTud2KVpsobFInZwN icp2WnROnFloQnfEUnDZnsnymAQqLcgzS2JA5mutETamomOrUvpysP86ClpfzdPfRQd8 AL201OpcVP8pKI7As7DC96YOF/yEJTY1i9RtxZfb36LACwq4wHfokzvvcWqD7iw72906 SYu7x1/S2buNBk79+4ACSPIWZrI7dyPqZYaD/17hs6f+ssyxaVuXvqMqDjRQM9znpSQW 6yL4DD6CltpE2fBTieLK64jo0zcYQv6Bab2oaQlmihUyNrFj8x7PuFVDgCbfiCVAkjSB buiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=J8/ndhQjyivI0+0njQmK8+MKUOKbGzHQcXZugQWCZvs=; b=Bvnv2IBsiNh+mxOFdVKXslvzgqlMpo+4rudBnh/3ZiIozLD/b8XUediNindzUd1Sci Q6zKX39zT+X/rPiuM7JMTP4cz7zqEon4C4ijQijbQydjH+KSq2K4Tu4WTB6kdNJqbEbL 9UNZjmurFSrWYIN0AVrgVWnES+/b0Fh6BlT1EVpP2sQSRuR8MjNU+XaJ+GyXsPcGq41r bk8YTXYQAlwL9VXJR/WTD225upqPkovkHSB1ypC5XSx3KC/3ge+VZjTM+ljv8PMTgzc5 BbtHYMTZM+Z7g0j5vZ9MV6ytzq34dhx/5Hi57ih0eteDeDWdfZ2WJwLSwLlXZnD2akOS ScFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=KP2b1Q77; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y11-20020a056402440b00b0046733d8f781si4783935eda.374.2022.11.18.17.20.11; Fri, 18 Nov 2022 17:20:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=KP2b1Q77; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230371AbiKSBS5 (ORCPT + 99 others); Fri, 18 Nov 2022 20:18:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231771AbiKSBSO (ORCPT ); Fri, 18 Nov 2022 20:18:14 -0500 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58317169C25 for ; Fri, 18 Nov 2022 16:15:42 -0800 (PST) Received: by mail-pl1-x62b.google.com with SMTP id b21so5914639plc.9 for ; Fri, 18 Nov 2022 16:15:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J8/ndhQjyivI0+0njQmK8+MKUOKbGzHQcXZugQWCZvs=; b=KP2b1Q775FJAQ4rqiT+V+IJFMMQfCScUOTn3bMIkQe2UTGoJiiIRuPeVuUyZTh13ds VOiVENBNcHyl/WINQ1FSke9w1kol7BJzzDLoMuaEp83qF3gBhHwKEo3UflaSdffnHDik HUBy22fcvqm4ShpwMSUeYN2+KFqLHQjl0dRfeP+ppKy4a7NItmcAZEZrZNV3WPDKihW4 rT9kR4DdtMpsbK9KOC5iungG+6e1A/rxJz+ta2upg3HcSGyPYyAohfgfezc1lIJrLGBO WtGVkzVIjmEsQd5pqVqSaJ0EUKugMBEGFoOqlcW8mt0nQgdmXhFoluh7r5MlD9DlK6pp +Tog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J8/ndhQjyivI0+0njQmK8+MKUOKbGzHQcXZugQWCZvs=; b=P7iUejpune7/L6NdbzVyh1nM860EzPlG7TqCQSbqthvPX4X3iKWS3nfQge88RPiiVN HnXgbfRtL4AeYcqm9jK1KogjSepbchuVPPaXU8t/oiUN0Zst+UCNxYAZcGb2mQqbav+W scAcMeCGfAgbY3QRWS8bTnVPlo3/7lTKbUrzZMV7JsxdYDOIlaRfth/4SWO6t/Uduiyb e8FQ+4zIVl1UwtRZta9zWgtg2r3ImsWEU8bxBhdCTRFrk4rsqhj/SUce5E/sn5GsYqzF XD3sCe5F3LMz2YM9rFFHrOX10eEs6YWaNvUFKBYh80bh5vW8msekX+YiTqFk80DRPvO7 as4Q== X-Gm-Message-State: ANoB5pmJEYHA8bziBDRnEvp3OB1fAVUq037hhbuediGB5Q5BaiILsLQk wk7Ch7HVGggZAAZ83nG8RXE= X-Received: by 2002:a17:902:a40a:b0:176:d59e:3aa with SMTP id p10-20020a170902a40a00b00176d59e03aamr1865059plq.31.1668816941657; Fri, 18 Nov 2022 16:15:41 -0800 (PST) Received: from localhost (fwdproxy-prn-013.fbsv.net. [2a03:2880:ff:d::face:b00c]) by smtp.gmail.com with ESMTPSA id o1-20020a625a01000000b0056232682a7esm3801342pfb.2.2022.11.18.16.15.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:41 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 3/6] zsmalloc: Consolidate zs_pool's migrate_lock and size_class's locks Date: Fri, 18 Nov 2022 16:15:33 -0800 Message-Id: <20221119001536.2086599-4-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749885476406377629?= X-GMAIL-MSGID: =?utf-8?q?1749885476406377629?= Currently, zsmalloc has a hierarchy of locks, which includes a pool-level migrate_lock, and a lock for each size class. We have to obtain both locks in the hotpath in most cases anyway, except for zs_malloc. This exception will no longer exist when we introduce a LRU into the zs_pool for the new writeback functionality - we will need to obtain a pool-level lock to synchronize LRU handling even in zs_malloc. In preparation for zsmalloc writeback, consolidate these locks into a single pool-level lock, which drastically reduces the complexity of synchronization in zsmalloc. We have also benchmarked the lock consolidation to see the performance effect of this change on zram. First, we ran a synthetic FS workload on a server machine with 36 cores (same machine for all runs), using fs_mark -d ../zram1mnt -s 100000 -n 2500 -t 32 -k before and after for btrfs and ext4 on zram (FS usage is 80%). Here is the result (unit is file/second): With lock consolidation (btrfs): Average: 13520.2, Median: 13531.0, Stddev: 137.5961482019028 Without lock consolidation (btrfs): Average: 13487.2, Median: 13575.0, Stddev: 309.08283679298665 With lock consolidation (ext4): Average: 16824.4, Median: 16839.0, Stddev: 89.97388510006668 Without lock consolidation (ext4) Average: 16958.0, Median: 16986.0, Stddev: 194.7370021336469 As you can see, we observe a 0.3% regression for btrfs, and a 0.9% regression for ext4. This is a small, barely measurable difference in my opinion. For a more realistic scenario, we also tries building the kernel on zram. Here is the time it takes (in seconds): With lock consolidation (btrfs): real Average: 319.6, Median: 320.0, Stddev: 0.8944271909999159 user Average: 6894.2, Median: 6895.0, Stddev: 25.528415540334656 sys Average: 521.4, Median: 522.0, Stddev: 1.51657508881031 Without lock consolidation (btrfs): real Average: 319.8, Median: 320.0, Stddev: 0.8366600265340756 user Average: 6896.6, Median: 6899.0, Stddev: 16.04057355583023 sys Average: 520.6, Median: 521.0, Stddev: 1.140175425099138 With lock consolidation (ext4): real Average: 320.0, Median: 319.0, Stddev: 1.4142135623730951 user Average: 6896.8, Median: 6878.0, Stddev: 28.621670111997307 sys Average: 521.2, Median: 521.0, Stddev: 1.7888543819998317 Without lock consolidation (ext4) real Average: 319.6, Median: 319.0, Stddev: 0.8944271909999159 user Average: 6886.2, Median: 6887.0, Stddev: 16.93221781102523 sys Average: 520.4, Median: 520.0, Stddev: 1.140175425099138 The difference is entirely within the noise of a typical run on zram. This hardly justifies the complexity of maintaining both the pool lock and the class lock. In fact, for writeback, we would need to introduce yet another lock to prevent data races on the pool's LRU, further complicating the lock handling logic. IMHO, it is just better to collapse all of these into a single pool-level lock. Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham Acked-by: Minchan Kim Acked-by: Johannes Weiner --- mm/zsmalloc.c | 87 ++++++++++++++++++++++----------------------------- 1 file changed, 37 insertions(+), 50 deletions(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d03941cace2c..326faa751f0a 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -33,8 +33,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock - * class->lock + * pool->lock * zspage->lock */ @@ -192,7 +191,6 @@ static const int fullness_threshold_frac = 4; static size_t huge_class_size; struct size_class { - spinlock_t lock; struct list_head fullness_list[NR_ZS_FULLNESS]; /* * Size of objects stored in this class. Must be multiple @@ -247,8 +245,7 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + spinlock_t lock; }; struct zspage { @@ -355,7 +352,7 @@ static void cache_free_zspage(struct zs_pool *pool, struct zspage *zspage) kmem_cache_free(pool->zspage_cachep, zspage); } -/* class->lock(which owns the handle) synchronizes races */ +/* pool->lock(which owns the handle) synchronizes races */ static void record_obj(unsigned long handle, unsigned long obj) { *(unsigned long *)handle = obj; @@ -452,7 +449,7 @@ static __maybe_unused int is_first_page(struct page *page) return PagePrivate(page); } -/* Protected by class->lock */ +/* Protected by pool->lock */ static inline int get_zspage_inuse(struct zspage *zspage) { return zspage->inuse; @@ -597,13 +594,13 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); + spin_lock(&pool->lock); class_almost_full = zs_stat_get(class, CLASS_ALMOST_FULL); class_almost_empty = zs_stat_get(class, CLASS_ALMOST_EMPTY); obj_allocated = zs_stat_get(class, OBJ_ALLOCATED); obj_used = zs_stat_get(class, OBJ_USED); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -916,7 +913,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, get_zspage_mapping(zspage, &class_idx, &fg); - assert_spin_locked(&class->lock); + assert_spin_locked(&pool->lock); VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(fg != ZS_EMPTY); @@ -1247,19 +1244,19 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + spin_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_location(obj, &page, &obj_idx); zspage = get_zspage(page); /* - * migration cannot move any zpages in this zspage. Here, class->lock + * migration cannot move any zpages in this zspage. Here, pool->lock * is too heavy since callers would take some time until they calls * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + spin_unlock(&pool->lock); class = zspage_class(pool, zspage); off = (class->size * obj_idx) & ~PAGE_MASK; @@ -1412,8 +1409,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) size += ZS_HANDLE_SIZE; class = pool->size_class[get_size_class_index(size)]; - /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + /* pool->lock effectively protects the zpage migration */ + spin_lock(&pool->lock); zspage = find_get_zspage(class); if (likely(zspage)) { obj = obj_malloc(pool, zspage, handle); @@ -1421,12 +1418,12 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) fix_fullness_group(class, zspage); record_obj(handle, obj); class_stat_inc(class, OBJ_USED, 1); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); return handle; } - spin_unlock(&class->lock); + spin_unlock(&pool->lock); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1434,7 +1431,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + spin_lock(&pool->lock); obj = obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1447,7 +1444,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); return handle; } @@ -1491,16 +1488,14 @@ void zs_free(struct zs_pool *pool, unsigned long handle) return; /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + spin_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_page(obj, &f_page); zspage = get_zspage(f_page); class = zspage_class(pool, zspage); - spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); obj_free(class->size, obj); class_stat_dec(class, OBJ_USED, 1); @@ -1510,7 +1505,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) free_zspage(pool, class, zspage); out: - spin_unlock(&class->lock); + spin_unlock(&pool->lock); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1867,16 +1862,12 @@ static int zs_page_migrate(struct page *newpage, struct page *page, pool = zspage->pool; /* - * The pool migrate_lock protects the race between zpage migration + * The pool's lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + spin_lock(&pool->lock); class = zspage_class(pool, zspage); - /* - * the class lock protects zpage alloc/free in the zspage. - */ - spin_lock(&class->lock); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1906,10 +1897,9 @@ static int zs_page_migrate(struct page *newpage, struct page *page, replace_sub_page(class, zspage, newpage, page); /* * Since we complete the data copy and set up new zspage structure, - * it's okay to release migration_lock. + * it's okay to release the pool's lock. */ - write_unlock(&pool->migrate_lock); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); dec_zspage_isolation(zspage); migrate_write_unlock(zspage); @@ -1964,9 +1954,9 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + spin_lock(&pool->lock); list_splice_init(&class->fullness_list[ZS_EMPTY], &free_pages); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1976,9 +1966,9 @@ static void async_free_zspage(struct work_struct *work) get_zspage_mapping(zspage, &class_idx, &fullness); VM_BUG_ON(fullness != ZS_EMPTY); class = pool->size_class[class_idx]; - spin_lock(&class->lock); + spin_lock(&pool->lock); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); } }; @@ -2039,10 +2029,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, struct zspage *dst_zspage = NULL; unsigned long pages_freed = 0; - /* protect the race between zpage migration and zs_free */ - write_lock(&pool->migrate_lock); - /* protect zpage allocation/free */ - spin_lock(&class->lock); + /* + * protect the race between zpage migration and zs_free + * as well as zpage allocation/free + */ + spin_lock(&pool->lock); while ((src_zspage = isolate_zspage(class, true))) { /* protect someone accessing the zspage(i.e., zs_map_object) */ migrate_write_lock(src_zspage); @@ -2067,7 +2058,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); migrate_write_unlock(dst_zspage); dst_zspage = NULL; - if (rwlock_is_contended(&pool->migrate_lock)) + if (spin_is_contended(&pool->lock)) break; } @@ -2084,11 +2075,9 @@ static unsigned long __zs_compact(struct zs_pool *pool, pages_freed += class->pages_per_zspage; } else migrate_write_unlock(src_zspage); - spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + spin_unlock(&pool->lock); cond_resched(); - write_lock(&pool->migrate_lock); - spin_lock(&class->lock); + spin_lock(&pool->lock); } if (src_zspage) { @@ -2096,8 +2085,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, migrate_write_unlock(src_zspage); } - spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + spin_unlock(&pool->lock); return pages_freed; } @@ -2200,7 +2188,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + spin_lock_init(&pool->lock); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name) @@ -2271,7 +2259,6 @@ struct zs_pool *zs_create_pool(const char *name) class->index = i; class->pages_per_zspage = pages_per_zspage; class->objs_per_zspage = objs_per_zspage; - spin_lock_init(&class->lock); pool->size_class[i] = class; for (fullness = ZS_EMPTY; fullness < NR_ZS_FULLNESS; fullness++) From patchwork Sat Nov 19 00:15:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 23156 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp500484wrr; Fri, 18 Nov 2022 17:20:50 -0800 (PST) X-Google-Smtp-Source: AA0mqf598/6mUHMvRO5DJ/Sf1F1K777R3IoXLAyvogKKN6QWot4WTfeK01s6pQmN0xsJrhNC3+vk X-Received: by 2002:a05:6402:c1c:b0:461:bd12:52ce with SMTP id co28-20020a0564020c1c00b00461bd1252cemr8330711edb.197.1668820850651; Fri, 18 Nov 2022 17:20:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668820850; cv=none; d=google.com; s=arc-20160816; b=H/CgVNYN1LgF1JF3r1P85CSCrk1rA/IyHItHtmPk2upkZY2JHJe08NAJ6gr9rc57NY XzgEShysPm0SCNFbaXRI4W/amTJVpq4TrMX/JLdhOQGwwl2rJvettXvBBy2bKJ2E9Qjr a5RkVgL9gNS4EWNnODyZFFVrcWAOEawmwFdcB+z+ySzuPr5iQOfuF30N3TKTQR//Pb5z hHQRsECErUVNtVTSJ3zlWdmgRHjGN7KelN/d5FyjNTC8YCQNYliVvLkuoRR7AkSKcUmM Mrl/X0HuGwUuw+oEgFQkfzQ9vd8fmfOZGr8xWjZOdgxy8J6/hXyzHi7pMYSMpNTabqGj 8qPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xlNO430pFQOka1esCEqa7IqA23nNPCeXNP57iQ3X88k=; b=tva1CTAntnb+SLcEGU6Z5e28TzRpTvuT1hDirFRT5GMXa5hDp96W2joMMpuIshFsjm X8/XNjUaSMANS8nKjpoNtq7riXXt89y3vUaBbOLYFBUBzVfzN+XKstmUPkTQ4NZBSkv8 Absa0nj8O2ByCAt6WxAsqMsqFQEQ3XfJerqtgQ8WoQ0AroAyQQZtdM4uf+gF3ATzieDj DBOeuotiHM5bGZY+1m+SDg/Zn8Und8Fu2gpIuA51t/HpR8OD8LdUe4y8xG6i8hoqjQua p+vSf0Iz9/3pjjh8PJGcpe3fwWC0GoANOg9EB76yXmbuLDv1wAj33d7D5KLbrVCAoOxa KWxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=dou3yaHy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f12-20020a056402354c00b004608c0b9a8asi4424561edd.201.2022.11.18.17.20.26; Fri, 18 Nov 2022 17:20:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=dou3yaHy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230484AbiKSBTC (ORCPT + 99 others); Fri, 18 Nov 2022 20:19:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231929AbiKSBSO (ORCPT ); Fri, 18 Nov 2022 20:18:14 -0500 Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 808EEF5A19 for ; Fri, 18 Nov 2022 16:15:43 -0800 (PST) Received: by mail-pg1-x534.google.com with SMTP id 62so6287191pgb.13 for ; Fri, 18 Nov 2022 16:15:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xlNO430pFQOka1esCEqa7IqA23nNPCeXNP57iQ3X88k=; b=dou3yaHyJKMOuyzUjM+AISJf6d71pZrUfWgikbtI9j3Y6hoNgor6yN25JPdwAaQ20T bmEir2kFtNskQqAa6VxLNCDTkWDBSHlNjmNr1BjrGtRvWaoVjBjwCe2CNlGrMB9dPazV zVupTfokhJ0RXHrTqeO8XRh2Sl5a0XajITvW/0w0eEWrXHyjTEthY+XOlM1/P3Qx+FOk PojrXY03qZdpjSSaqO9sYspzBL7r770DMqA1qyL4NgGRnuKeKEWvSnSjInRtGVvk6Vgd LqD/1+XOJo0RiC3b6nmNF2vpTvmAVhB1Dh2dkK+s9F02Uq+tVt1tf0f7PGy4eLuCIXyD 42JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xlNO430pFQOka1esCEqa7IqA23nNPCeXNP57iQ3X88k=; b=hsgIwZ6S/C4tM4N5k+DjZIzzC1FBwrkYUHatDwG9I4xG39KVwOrjH1FQKxHV415u94 AhpB6eup9+99gHIOo7u2W+x825lWTOMEsAB8WD0bjNsu/7KwTQ/BTJUIDw4qxzXc+/Jq RkJLkmJ+C2CVfVNVbg55tH4ktukqkT0vMzi/BwkEaptNvlzqca0M9ID/+fpguM3IiQgd iYby4FfJ3BG5C7x/g1F5efowHtnx1RkGeicN1ibID55xuQ1DCc+YG0SXsOPTH5QJuhtL RTwPJRW6T1lbH83TCn69E9eYbooYf0Zg5DIeD+EOMyMt70iXr3fDgsQ/uacyEShbkO4c 4mrg== X-Gm-Message-State: ANoB5pm4SVO4WLwGHixUxZTlQIg6aLXa8vWu+iNzxelPESrwHCkqS5yQ BifrB2jcxwtpqiK7rxQBq9k= X-Received: by 2002:a63:d241:0:b0:439:8688:a98d with SMTP id t1-20020a63d241000000b004398688a98dmr8511296pgi.424.1668816943018; Fri, 18 Nov 2022 16:15:43 -0800 (PST) Received: from localhost (fwdproxy-prn-017.fbsv.net. [2a03:2880:ff:11::face:b00c]) by smtp.gmail.com with ESMTPSA id u139-20020a627991000000b00561cf757749sm3751245pfc.183.2022.11.18.16.15.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:42 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 4/6] zsmalloc: Add a LRU to zs_pool to keep track of zspages in LRU order Date: Fri, 18 Nov 2022 16:15:34 -0800 Message-Id: <20221119001536.2086599-5-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749885492325247480?= X-GMAIL-MSGID: =?utf-8?q?1749885492325247480?= This helps determines the coldest zspages as candidates for writeback. Signed-off-by: Nhat Pham Acked-by: Johannes Weiner Acked-by: Minchan Kim --- mm/zsmalloc.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 326faa751f0a..7dd464b5a6a5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -239,6 +239,11 @@ struct zs_pool { /* Compact classes */ struct shrinker shrinker; +#ifdef CONFIG_ZPOOL + /* List tracking the zspages in LRU order by most recently added object */ + struct list_head lru; +#endif + #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; #endif @@ -260,6 +265,12 @@ struct zspage { unsigned int freeobj; struct page *first_page; struct list_head list; /* fullness list */ + +#ifdef CONFIG_ZPOOL + /* links the zspage to the lru list in the pool */ + struct list_head lru; +#endif + struct zs_pool *pool; #ifdef CONFIG_COMPACTION rwlock_t lock; @@ -953,6 +964,9 @@ static void free_zspage(struct zs_pool *pool, struct size_class *class, } remove_zspage(class, zspage, ZS_EMPTY); +#ifdef CONFIG_ZPOOL + list_del(&zspage->lru); +#endif __free_zspage(pool, class, zspage); } @@ -998,6 +1012,10 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) off %= PAGE_SIZE; } +#ifdef CONFIG_ZPOOL + INIT_LIST_HEAD(&zspage->lru); +#endif + set_freeobj(zspage, 0); } @@ -1249,6 +1267,15 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, obj_to_location(obj, &page, &obj_idx); zspage = get_zspage(page); +#ifdef CONFIG_ZPOOL + /* Move the zspage to front of pool's LRU */ + if (mm == ZS_MM_WO) { + if (!list_empty(&zspage->lru)) + list_del(&zspage->lru); + list_add(&zspage->lru, &pool->lru); + } +#endif + /* * migration cannot move any zpages in this zspage. Here, pool->lock * is too heavy since callers would take some time until they calls @@ -1967,6 +1994,9 @@ static void async_free_zspage(struct work_struct *work) VM_BUG_ON(fullness != ZS_EMPTY); class = pool->size_class[class_idx]; spin_lock(&pool->lock); +#ifdef CONFIG_ZPOOL + list_del(&zspage->lru); +#endif __free_zspage(pool, class, zspage); spin_unlock(&pool->lock); } @@ -2278,6 +2308,10 @@ struct zs_pool *zs_create_pool(const char *name) */ zs_register_shrinker(pool); +#ifdef CONFIG_ZPOOL + INIT_LIST_HEAD(&pool->lru); +#endif + return pool; err: From patchwork Sat Nov 19 00:15:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 23157 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp500616wrr; Fri, 18 Nov 2022 17:21:14 -0800 (PST) X-Google-Smtp-Source: AA0mqf5g93UFXRnCQr0Z6OcBaEtHrIeY+2kFFwptmp4xmH34w5W0CC6hq3bX1xMIlaj4VHIXQc5x X-Received: by 2002:aa7:dad3:0:b0:458:d7b7:d912 with SMTP id x19-20020aa7dad3000000b00458d7b7d912mr8091057eds.316.1668820874625; Fri, 18 Nov 2022 17:21:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668820874; cv=none; d=google.com; s=arc-20160816; b=JHxKlSxKXktX93AkxsJh8c94W3sfCvhVoYohuHMJaexDPTapyw0eHh2oP28QP9E1ld /88RBPV0JkAhOoHrYqT3X96tNcSE2TldIrF1j06/WG7vyxPCPPosRNNZtBH1vu/rwO/c rj9ooOuMtpyftZYS4IjJ9F5mUqX9a3g1zYQFL3MaHDpdJmNCEi0BEe1hh+5g6bnZA2yG cmqI0FBiWyoaP/BdU0g0Nfsv7v3v37QvZ8c1ZbxtCHOHepZzvTPkZWVou0fIkWcibb6h B4N1/CGBrjq9WN3MfYan5l2ktwQiaO6Vx31lBbRRTzrIeGynbadUpV9LKcHdGTz2yuL0 YpTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZXVKU8ju9i6lgVo4GHLlSQvxR+3nZ3i/iGJQSsIEw+M=; b=qtqh6ffWYg6e0uqPI4QQgoD1hqGNyG+pYWCAyg32SemRv0sX2b5ozTPu3BNwPVEdV5 mqISm4SVbK5J1GfyFbTQ4wNXbJxo7Y2nw0mP9HFoIfmtzuSl2l/PKDeScdVej/Reqf6/ yT01kKusBk362eK3FphP80ibQFqUkxTMaAfeM8JvNWz2JCqPM/JYiFqd7hIFQ1v50kug I1VDJ3W4UD9UJOAxZ0AVkDW/IF80QE4uoYqDPgtb/FopYuecPlA3NmNM2jnSr9y73Z4K rXAbocrdx47uSU97wEClDaVbyWhFGFs93CiwyI3QcKzG4P6BKRtYLVUaLfCdNBb99w8L SPZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=jBhZspZh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vo15-20020a170907a80f00b0077cfec3a52fsi3967019ejc.839.2022.11.18.17.20.49; Fri, 18 Nov 2022 17:21:14 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=jBhZspZh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231925AbiKSBTG (ORCPT + 99 others); Fri, 18 Nov 2022 20:19:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229722AbiKSBSP (ORCPT ); Fri, 18 Nov 2022 20:18:15 -0500 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B41F7F41B1 for ; Fri, 18 Nov 2022 16:15:44 -0800 (PST) Received: by mail-pg1-x536.google.com with SMTP id 62so6287216pgb.13 for ; Fri, 18 Nov 2022 16:15:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZXVKU8ju9i6lgVo4GHLlSQvxR+3nZ3i/iGJQSsIEw+M=; b=jBhZspZhLzlDshrZ/lo4FyZvP0Y26uOgONnXztvGdE4w/RlHjf7VoFHEy/c1QXw9OC 3sh/pvMH4xjovvN5ZMgrFxs7uaMbEz6EfixhJHiZYxopq/6bP14oIYnEtgTOHe1bmjEt 2pH3951rnmqnaeRLCM26DP3goHH8vWmo0GFgnf81JZW/KSDg+nnAJfgZasbKgYIZsw66 ld/4s7hw9iZC3UYE461byM9rk8f9S8QzWQ4Q7tAQW9aA+fMgeZ6rfu/441Wsqap1fvsg TKwiBsOZCHowCBL25wiaNVJupPCK9Vqkzg0FSskz55asJlf6oPn+0Nu52DBiMPBxuC63 O5hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZXVKU8ju9i6lgVo4GHLlSQvxR+3nZ3i/iGJQSsIEw+M=; b=Pt0ngFeCV39pRvBCD56ZQBAECsjz9lehH52QfNJfK+EEif9jjNaji+1nOWU019kyTg /CRtnFSNbjTQAp6X72IG3WJw5YDoYkZDBl8EgKviKSSy9XogZ5JGz4f3bld7vkV+2D3z HOipMB7CoAojZMrqJZQOMI/0t4mNcs22JQte0hMxFPLR0+GuSbimzlMUClbtrN9KZxfF Hoib4dcetrlpHOzHQG5AKT9WXgG5y2ub3rmHlxGEDsiOL5KiSlQ+tOwUE7NGBEGM+jbO XvzNvRRZJ2EpEfpypL8RNa2uJiUUTewkTbqOvoPKXhTcNPjT7uc6zZwZCMVUShwz/MYD l7mQ== X-Gm-Message-State: ANoB5pmJG/siuwJCrj3wQviePE0fFy9CZH4NfYU1Qg2JwOKXe1c5QME6 34X85V8Z/8Du11U4KkmkRlA= X-Received: by 2002:a63:4701:0:b0:476:d784:1c44 with SMTP id u1-20020a634701000000b00476d7841c44mr8385602pga.431.1668816944233; Fri, 18 Nov 2022 16:15:44 -0800 (PST) Received: from localhost (fwdproxy-prn-120.fbsv.net. [2a03:2880:ff:78::face:b00c]) by smtp.gmail.com with ESMTPSA id x2-20020a626302000000b0056d3b8f530csm3827237pfb.34.2022.11.18.16.15.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:43 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 5/6] zsmalloc: Add zpool_ops field to zs_pool to store evict handlers Date: Fri, 18 Nov 2022 16:15:35 -0800 Message-Id: <20221119001536.2086599-6-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749885517196240770?= X-GMAIL-MSGID: =?utf-8?q?1749885517196240770?= This adds a new field to zs_pool to store evict handlers for writeback, analogous to the zbud allocator. Signed-off-by: Nhat Pham Acked-by: Minchan Kim Acked-by: Johannes Weiner --- mm/zsmalloc.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7dd464b5a6a5..9920f3584511 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -242,6 +242,8 @@ struct zs_pool { #ifdef CONFIG_ZPOOL /* List tracking the zspages in LRU order by most recently added object */ struct list_head lru; + struct zpool *zpool; + const struct zpool_ops *zpool_ops; #endif #ifdef CONFIG_ZSMALLOC_STAT @@ -382,7 +384,14 @@ static void *zs_zpool_create(const char *name, gfp_t gfp, * different contexts and its caller must provide a valid * gfp mask. */ - return zs_create_pool(name); + struct zs_pool *pool = zs_create_pool(name); + + if (pool) { + pool->zpool = zpool; + pool->zpool_ops = zpool_ops; + } + + return pool; } static void zs_zpool_destroy(void *pool) From patchwork Sat Nov 19 00:15:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 23158 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp501080wrr; Fri, 18 Nov 2022 17:22:38 -0800 (PST) X-Google-Smtp-Source: AA0mqf58qZ7A2FEeCPzxv588i8cTibiWdlKNjvMVTdwmdimr9vRpTbj6m+qb0BWn7jhXxLglGJ0D X-Received: by 2002:a63:fd08:0:b0:46f:ed91:6664 with SMTP id d8-20020a63fd08000000b0046fed916664mr8791556pgh.558.1668820958599; Fri, 18 Nov 2022 17:22:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668820958; cv=none; d=google.com; s=arc-20160816; b=c97ZdaNHhOV2z1rMxbSathXeSU5sc4OGJ8xwgSTU3pjPxzPFr8ZCGhsDkyaIEefzPN UmsnfNn7CCgibHZByN+8RpdhO3u92n9wVX5SdxEda6MaMOiL3FOrvcMq4YiTA9DlNNZi 1EXkQbsmskPeaeAthybBC6/Q8ldidqQ9csqlwnOGDFlRnd0zQYqy8bWzLE3e0yRBmuGI nkRRfFMI/f33/Q60mF5fNAzoDgbVgxvr388N+Ankodk4Z0HmIlC5PWh60EKu9/HiBve1 MTRXB1+L+sBPsSM31v490y9iUA7+gpGiHci/5jF2oUemAVzELFcwlZwhTNpkWbCO879L 4IjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Co9VEJF4pdxmrIDsS8H+BTo/CtOiZYr3gRWH6+3Vpr4=; b=CzTqqg3CgTY8UZe1wpEU8NpiFA4AaQLh9Ci/4h/hnU/nBtyxKbrG/ItQkW2fNILxuB gHCoZ7WYYkq5ELYPoTGr6BOABqy5U8UhyN/CFEs49GXinWUDAF+LAGSEWPEgWdqWU5Zh xUEXChAlpINbHEy59WavT/zoiYmSpJVFqhsfAS9SDrXRg+MCsGoj3YUMNUtmv2bKz299 eC8OCoofUyiNLHSbzAI4PDIMxplzZirfJC85fZcUVC67Hc7HUwe9/7Ow41KasgfbUKT9 eQUPbOYAkZkHguvL30LOOGOlKbiOzkkM+uF4O1RbuR3Au0IC5ARiW8VAxfbmgeDqNaFJ 8xDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=T0YKjRww; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y12-20020a17090322cc00b0016dc27fb9fbsi5727848plg.117.2022.11.18.17.22.25; Fri, 18 Nov 2022 17:22:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=T0YKjRww; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232560AbiKSBTN (ORCPT + 99 others); Fri, 18 Nov 2022 20:19:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229663AbiKSBSP (ORCPT ); Fri, 18 Nov 2022 20:18:15 -0500 Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 017D2F60B5 for ; Fri, 18 Nov 2022 16:15:46 -0800 (PST) Received: by mail-pl1-x62b.google.com with SMTP id y10so4738445plp.3 for ; Fri, 18 Nov 2022 16:15:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Co9VEJF4pdxmrIDsS8H+BTo/CtOiZYr3gRWH6+3Vpr4=; b=T0YKjRwwzXJG2yRWFWfCM687bYqCIIoadzwEjQFvRTsy4BBI7Q1v1cPIo3piv1EOu5 ksLLpkjjRbaCjmKa2qR9lbumsFOCXTXPEgF858ZOaWAvmuU21JTnd9/H72E1Y9xmuQmO BBbiGmtGmXmdVqKR6lULR+/UERUdy94CZEgLg5tu12N+NXZB3grZXVL8fYawPBSHrIbT 9grObhJ7iztQZ9tpmLwnTxgrlApibIBEAHtjdNYiAmLEArbEXfa+1m/hsVNTOYQTZM41 LqH8ZFFGKdde707H4Q1MHM/Q3gbdcfZbkFlQ4qTC/+S04DVQKr4lNJBwZlj/bUU9xsFE t8Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Co9VEJF4pdxmrIDsS8H+BTo/CtOiZYr3gRWH6+3Vpr4=; b=S266nQ7PeftqXXbt22Q3Bmj1LbDIg0e2GdfZk9d3EFk72Xn8tXn+UhfmEqyxrdL2uF 0YMUQZGgYkQhfanFP+cORemn3bADpPXkm3xGzopcgbl1sbhDoqY+pFFhhQk5wAAaVxRn XpnV5KG2oV7wCXV5pZyZQQrTluY9p4uPOYuR3AbhNcgdBjM7ovWXVfaB6x4wFvyV3jTN wJZAvJQZ1iCPkuOk6AJdCxz/mZyzCFYpB61gKZMBD6WASRJ9RnACOafiiYgR9LKY9DeM lj1Zl3wwf//fZXKwSJxl4OkVoBMNQr3FisFsouLrlBR55+BBSRT9dvk7W+NG7cF0MnaS 1Wcg== X-Gm-Message-State: ANoB5pkfYm+QJ7rC/lNYmjTDV1hvNXfS6CgQgil2sb+62nFM/IJPCAyJ 81rYpd+04Akw7Qy2lY8skQA= X-Received: by 2002:a17:903:2055:b0:188:bed6:3fbe with SMTP id q21-20020a170903205500b00188bed63fbemr1812139pla.38.1668816945445; Fri, 18 Nov 2022 16:15:45 -0800 (PST) Received: from localhost (fwdproxy-prn-018.fbsv.net. [2a03:2880:ff:12::face:b00c]) by smtp.gmail.com with ESMTPSA id o14-20020a170902d4ce00b00186acb14c4asm4418285plg.67.2022.11.18.16.15.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:45 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 6/6] zsmalloc: Implement writeback mechanism for zsmalloc Date: Fri, 18 Nov 2022 16:15:36 -0800 Message-Id: <20221119001536.2086599-7-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749885605454331752?= X-GMAIL-MSGID: =?utf-8?q?1749885605454331752?= This commit adds the writeback mechanism for zsmalloc, analogous to the zbud allocator. Zsmalloc will attempt to determine the coldest zspage (i.e least recently used) in the pool, and attempt to write back all the stored compressed objects via the pool's evict handler. Signed-off-by: Nhat Pham Acked-by: Johannes Weiner Acked-by: Minchan Kim --- mm/zsmalloc.c | 193 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 182 insertions(+), 11 deletions(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9920f3584511..3fba04e10227 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -271,12 +271,13 @@ struct zspage { #ifdef CONFIG_ZPOOL /* links the zspage to the lru list in the pool */ struct list_head lru; + bool under_reclaim; + /* list of unfreed handles whose objects have been reclaimed */ + unsigned long *deferred_handles; #endif struct zs_pool *pool; -#ifdef CONFIG_COMPACTION rwlock_t lock; -#endif }; struct mapping_area { @@ -297,10 +298,11 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -#ifdef CONFIG_COMPACTION static void migrate_lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); + +#ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage); static void migrate_write_lock_nested(struct zspage *zspage); static void migrate_write_unlock(struct zspage *zspage); @@ -308,9 +310,6 @@ static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage); #else -static void migrate_lock_init(struct zspage *zspage) {} -static void migrate_read_lock(struct zspage *zspage) {} -static void migrate_read_unlock(struct zspage *zspage) {} static void migrate_write_lock(struct zspage *zspage) {} static void migrate_write_lock_nested(struct zspage *zspage) {} static void migrate_write_unlock(struct zspage *zspage) {} @@ -413,6 +412,27 @@ static void zs_zpool_free(void *pool, unsigned long handle) zs_free(pool, handle); } +static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries); + +static int zs_zpool_shrink(void *pool, unsigned int pages, + unsigned int *reclaimed) +{ + unsigned int total = 0; + int ret = -EINVAL; + + while (total < pages) { + ret = zs_reclaim_page(pool, 8); + if (ret < 0) + break; + total++; + } + + if (reclaimed) + *reclaimed = total; + + return ret; +} + static void *zs_zpool_map(void *pool, unsigned long handle, enum zpool_mapmode mm) { @@ -451,6 +471,7 @@ static struct zpool_driver zs_zpool_driver = { .malloc_support_movable = true, .malloc = zs_zpool_malloc, .free = zs_zpool_free, + .shrink = zs_zpool_shrink, .map = zs_zpool_map, .unmap = zs_zpool_unmap, .total_size = zs_zpool_total_size, @@ -924,6 +945,25 @@ static int trylock_zspage(struct zspage *zspage) return 0; } +#ifdef CONFIG_ZPOOL +/* + * Free all the deferred handles whose objects are freed in zs_free. + */ +static void free_handles(struct zs_pool *pool, struct zspage *zspage) +{ + unsigned long handle = (unsigned long)zspage->deferred_handles; + + while (handle) { + unsigned long nxt_handle = handle_to_obj(handle); + + cache_free_handle(pool, handle); + handle = nxt_handle; + } +} +#else +static inline void free_handles(struct zs_pool *pool, struct zspage *zspage) {} +#endif + static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { @@ -938,6 +978,9 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(fg != ZS_EMPTY); + /* Free all deferred handles from zs_free */ + free_handles(pool, zspage); + next = page = get_first_page(zspage); do { VM_BUG_ON_PAGE(!PageLocked(page), page); @@ -1023,6 +1066,8 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) #ifdef CONFIG_ZPOOL INIT_LIST_HEAD(&zspage->lru); + zspage->under_reclaim = false; + zspage->deferred_handles = NULL; #endif set_freeobj(zspage, 0); @@ -1535,12 +1580,26 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_free(class->size, obj); class_stat_dec(class, OBJ_USED, 1); + +#ifdef CONFIG_ZPOOL + if (zspage->under_reclaim) { + /* + * Reclaim needs the handles during writeback. It'll free + * them along with the zspage when it's done with them. + * + * Record current deferred handle at the memory location + * whose address is given by handle. + */ + record_obj(handle, (unsigned long)zspage->deferred_handles); + zspage->deferred_handles = (unsigned long *)handle; + spin_unlock(&pool->lock); + return; + } +#endif fullness = fix_fullness_group(class, zspage); - if (fullness != ZS_EMPTY) - goto out; + if (fullness == ZS_EMPTY) + free_zspage(pool, class, zspage); - free_zspage(pool, class, zspage); -out: spin_unlock(&pool->lock); cache_free_handle(pool, handle); } @@ -1740,7 +1799,7 @@ static enum fullness_group putback_zspage(struct size_class *class, return fullness; } -#ifdef CONFIG_COMPACTION +#if defined(CONFIG_ZPOOL) || defined(CONFIG_COMPACTION) /* * To prevent zspage destroy during migration, zspage freeing should * hold locks of all pages in the zspage. @@ -1782,6 +1841,24 @@ static void lock_zspage(struct zspage *zspage) } migrate_read_unlock(zspage); } +#endif /* defined(CONFIG_ZPOOL) || defined(CONFIG_COMPACTION) */ + +#ifdef CONFIG_ZPOOL +/* + * Unlocks all the pages of the zspage. + * + * pool->lock must be held before this function is called + * to prevent the underlying pages from migrating. + */ +static void unlock_zspage(struct zspage *zspage) +{ + struct page *page = get_first_page(zspage); + + do { + unlock_page(page); + } while ((page = get_next_page(page)) != NULL); +} +#endif /* CONFIG_ZPOOL */ static void migrate_lock_init(struct zspage *zspage) { @@ -1798,6 +1875,7 @@ static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) read_unlock(&zspage->lock); } +#ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage) { write_lock(&zspage->lock); @@ -2362,6 +2440,99 @@ void zs_destroy_pool(struct zs_pool *pool) } EXPORT_SYMBOL_GPL(zs_destroy_pool); +#ifdef CONFIG_ZPOOL +static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) +{ + int i, obj_idx, ret = 0; + unsigned long handle; + struct zspage *zspage; + struct page *page; + enum fullness_group fullness; + + /* Lock LRU and fullness list */ + spin_lock(&pool->lock); + if (list_empty(&pool->lru)) { + spin_unlock(&pool->lock); + return -EINVAL; + } + + for (i = 0; i < retries; i++) { + struct size_class *class; + + zspage = list_last_entry(&pool->lru, struct zspage, lru); + list_del(&zspage->lru); + + /* zs_free may free objects, but not the zspage and handles */ + zspage->under_reclaim = true; + + class = zspage_class(pool, zspage); + fullness = get_fullness_group(class, zspage); + + /* Lock out object allocations and object compaction */ + remove_zspage(class, zspage, fullness); + + spin_unlock(&pool->lock); + + /* Lock backing pages into place */ + lock_zspage(zspage); + + obj_idx = 0; + page = zspage->first_page; + while (1) { + handle = find_alloced_obj(class, page, &obj_idx); + if (!handle) { + page = get_next_page(page); + if (!page) + break; + obj_idx = 0; + continue; + } + + /* + * This will write the object and call zs_free. + * + * zs_free will free the object, but the + * under_reclaim flag prevents it from freeing + * the zspage altogether. This is necessary so + * that we can continue working with the + * zspage potentially after the last object + * has been freed. + */ + ret = pool->zpool_ops->evict(pool->zpool, handle); + if (ret) + goto next; + + obj_idx++; + } + +next: + /* For freeing the zspage, or putting it back in the pool and LRU list. */ + spin_lock(&pool->lock); + zspage->under_reclaim = false; + + if (!get_zspage_inuse(zspage)) { + /* + * Fullness went stale as zs_free() won't touch it + * while the page is removed from the pool. Fix it + * up for the check in __free_zspage(). + */ + zspage->fullness = ZS_EMPTY; + + __free_zspage(pool, class, zspage); + spin_unlock(&pool->lock); + return 0; + } + + putback_zspage(class, zspage); + list_add(&zspage->lru, &pool->lru); + unlock_zspage(zspage); + } + + spin_unlock(&pool->lock); + return -EAGAIN; +} +#endif /* CONFIG_ZPOOL */ + static int __init zs_init(void) { int ret;