From patchwork Fri Nov 18 18:24:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 22443 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp354954wrr; Fri, 18 Nov 2022 10:51:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf6yQLBKQI7PwqX5GPTiuLnXouow8Z2UbssAvY0hvGa4uS8e6LzuZc8+FrTVfXEJ35Wl9k1L X-Received: by 2002:a17:90a:d145:b0:213:f465:14e7 with SMTP id t5-20020a17090ad14500b00213f46514e7mr9065611pjw.194.1668797463838; Fri, 18 Nov 2022 10:51:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668797463; cv=none; d=google.com; s=arc-20160816; b=mEOHkrxeUBK7+vdsNhjV09wuKuzQdn6hjTGQkK6m6FY+/ld/Zn+8zDoBzXYW+LWmO0 YlYKJm7Q+SqAZR3vnnofiPRcST2LjrQkXjw9TJKo1aa43uFSdkAYbVwgNd/ADxWW9IG4 txKLrW/fA/Oh7w4OTI7qItGS3QtKq7ZGQKE17Ewwy8hwcq6Ofh8tbgQh4H3uD7gcdfp2 bxWSGU/lrsTN7V58qMiZW2pm2dSc15AYxRrP46f21imMlUykrwheYv3pJc9kAtHWbLfZ LVYyNrHje6QHW2riLNhhwi1z10EsuqfMm40N2ElYfntDMxrJVXe7YLLSJlTN09HzZDsl E/Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=seCx/RGiIZJYTLLSqpo5kVJ+zNp/jMiuXGfW8uFDv9ee3hvoGlaACSfOUzmZw5B5x9 fNRZtWr4YAv5UbhqlnsncsJ6s3BBjFob1YCNG8bu/5JSyGw4FOQW7bEonwZS/DG2rZCD lm0p3t5Y9qXRi/CsiHBhdkpb4LLT+SsIqgnTLcjAnOhJepe1KgLKd8EUYK0sAmORGM+4 Yxq4RIrcqrbzD2rIwjJsi5hBCB0ued2qcUKZOoPxUxxT4b/1GiLvsEK7utt5wjiZDWmj TwwzEg340/zMyAXBJFra//14pe/J2Urk6//L72dZVBh0wq7XfMSmw5GAJx1/Iv1rGfFa Jw6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=nkZTR8Kb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y10-20020a17090322ca00b00188ad330242si4685228plg.131.2022.11.18.10.50.40; Fri, 18 Nov 2022 10:51:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=nkZTR8Kb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242703AbiKRSZ3 (ORCPT + 99 others); Fri, 18 Nov 2022 13:25:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242698AbiKRSZI (ORCPT ); Fri, 18 Nov 2022 13:25:08 -0500 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4307D970B0 for ; Fri, 18 Nov 2022 10:24:13 -0800 (PST) Received: by mail-pj1-x1033.google.com with SMTP id a22-20020a17090a6d9600b0021896eb5554so82617pjk.1 for ; Fri, 18 Nov 2022 10:24:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=nkZTR8Kb+B0gk7H1xIfb6LfPuKa1/cZSjTASVzpInLhhWrDd52VMjYvyLSxibmHHod XS8PyxlBFQKCwIY7VnVGxHp2GBZlNfH/mDQ4f5E1mblACckXhlpqRDGQD3VKGXrVaLNy bo1/KFKE5Sqo1Aa3JN4cuKvISQluxEyKaDUihk3A+AHs589MG38F3CTjzVYiv3aBbekT x8m49ihtYKr1AJtwmsvWsD+DhRasBVPL6r5zFur0+0F9Yfp9UH+kmP1I9x8jXDu4nyYc GLJJzZhXs5fchmAsncztYOnoY5qanzae0S0voCitkq4P4TSqCe1NvZpSE82KV0vudWgM mn+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=c1f6xfRh4ILjJyj+7am8PKdNMI1o1FSokd0M8TobAIIG0m7tQ9mjt9n4OPT5QFYDiA 7n+PRKuG9OcGwWUzrtMse8BgTmn3bgZNytr8l4mQywL7xnScvQ+mOsFxRCOS1+NKvqjV PU8USuEljYMFWWqJy9f/G6IF5qgQuDd91LbVEhW0NjrcHx5ys7DAUTreL2/nbc9fLEbZ mrgCuDaUxLtX9Ccp6tCHFE6vOe35kCzjIdyiQ+fng6YMUL6axRdbBImvpDh456V4GXO6 U9l7P0xxpI48yqObHrEdR7jSRmhIGJDZ/nRFghma16ASGwrfGx/6or1FFvQ0PmkqtOjH fGmA== X-Gm-Message-State: ANoB5pl825jZ2ME37Lz7BNX4lUHD0vZhpBqSrhgnjXYs4xPRVHP9InKV BEgDBHFch+Jp51u12td5nl9gdR49RSPcmA== X-Received: by 2002:a17:902:cacd:b0:17f:7d9a:4952 with SMTP id y13-20020a170902cacd00b0017f7d9a4952mr612738pld.117.1668795850563; Fri, 18 Nov 2022 10:24:10 -0800 (PST) Received: from localhost (fwdproxy-prn-023.fbsv.net. [2a03:2880:ff:17::face:b00c]) by smtp.gmail.com with ESMTPSA id c8-20020a170902b68800b00188b5d25438sm4058927pls.35.2022.11.18.10.24.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 10:24:10 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v5 1/6] zswap: fix writeback lock ordering for zsmalloc Date: Fri, 18 Nov 2022 10:24:02 -0800 Message-Id: <20221118182407.82548-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221118182407.82548-1-nphamcs@gmail.com> References: <20221118182407.82548-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749860969561993173?= X-GMAIL-MSGID: =?utf-8?q?1749860969561993173?= From: Johannes Weiner zswap's customary lock order is tree->lock before pool->lock, because the tree->lock protects the entries' refcount, and the free callbacks in the backends acquire their respective pool locks to dispatch the backing object. zsmalloc's map callback takes the pool lock, so zswap must not grab the tree->lock while a handle is mapped. This currently only happens during writeback, which isn't implemented for zsmalloc. In preparation for it, move the tree->lock section out of the mapped entry section Signed-off-by: Johannes Weiner Signed-off-by: Nhat Pham --- mm/zswap.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) -- 2.30.2 diff --git a/mm/zswap.c b/mm/zswap.c index 2d48fd59cc7a..2d69c1d678fe 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -958,7 +958,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) }; if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_ATOMIC); + tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); if (!tmp) return -ENOMEM; } @@ -968,6 +968,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) swpentry = zhdr->swpentry; /* here */ tree = zswap_trees[swp_type(swpentry)]; offset = swp_offset(swpentry); + zpool_unmap_handle(pool, handle); /* find and ref zswap entry */ spin_lock(&tree->lock); @@ -975,20 +976,12 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) if (!entry) { /* entry was invalidated */ spin_unlock(&tree->lock); - zpool_unmap_handle(pool, handle); kfree(tmp); return 0; } spin_unlock(&tree->lock); BUG_ON(offset != entry->offset); - src = (u8 *)zhdr + sizeof(struct zswap_header); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; - zpool_unmap_handle(pool, handle); - } - /* try to allocate swap cache page */ switch (zswap_get_swap_cache_page(swpentry, &page)) { case ZSWAP_SWAPCACHE_FAIL: /* no memory or invalidate happened */ @@ -1006,6 +999,14 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + zhdr = zpool_map_handle(pool, handle, ZPOOL_MM_RO); + src = (u8 *)zhdr + sizeof(struct zswap_header); + if (!zpool_can_sleep_mapped(pool)) { + memcpy(tmp, src, entry->length); + src = tmp; + zpool_unmap_handle(pool, handle); + } + mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); @@ -1015,6 +1016,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); + else + zpool_unmap_handle(pool, handle); + BUG_ON(ret); BUG_ON(dlen != PAGE_SIZE); @@ -1045,7 +1051,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - goto end; + return ret; + +fail: + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); /* * if we get here due to ZSWAP_SWAPCACHE_EXIST @@ -1054,17 +1064,10 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) * if we free the entry in the following put * it is also okay to return !0 */ -fail: spin_lock(&tree->lock); zswap_entry_put(tree, entry); spin_unlock(&tree->lock); -end: - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, handle); - else - kfree(tmp); - return ret; } From patchwork Fri Nov 18 18:24:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 22436 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp347555wrr; Fri, 18 Nov 2022 10:31:52 -0800 (PST) X-Google-Smtp-Source: AA0mqf5reBPJGllmnmu5E7LN5tWQbpBbEKMZcZumf3WVmzEu+UgURbTEE8I8/pzeC2Ovsebd6iYP X-Received: by 2002:a17:90a:c381:b0:212:bfc3:3271 with SMTP id h1-20020a17090ac38100b00212bfc33271mr8778587pjt.99.1668796312538; Fri, 18 Nov 2022 10:31:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668796312; cv=none; d=google.com; s=arc-20160816; b=b4ab0Bc378ihu68+L8Z3+ifx7jN7D8F7oAqwTe5ud1DkXLByNamZVlGHwuEnRhV98J dgA3uQ+CfR0VAI3XEfY1w0M4h1LxTLRjVagPGF1F3ZuS1z6T8y0xU+qtAZMlSpOmCqg4 FsBI96dPM/+5bGuCIftmmEosXh0A/99/WybevminelNL+9Ud0VndtEaEM/EdhL3EZkxi +DRipF1UbqPyCbjstLyb9f36tJvOMBZOxuAcXAQIJETG3OFeEWrDyW+u1cG2QFByqxuj TRoEBd8dJc4elQidkeEf1L6S4fmlHaGYIs8gzToS+4c8WRd/JHTZ6zPlrsYowrs0e6+9 HNXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=TmLMX68DtcyIbd+imh5axuhDGWjTd1ckFtRQexsVIBjyWRzoBAm68DzUJcNqM5U/jh ArB1TMSIoIMW5tl3t8Rq+lqVCJ34f2CmjYihvGhF+SP96x+ZdIqeBKgYOKCDdM+wG0nV +mGN7Yeg6oOXN7xiAZIcGpZ6WgYODS0l45qfmSSbs1Gfe8WdPJX5o7whsqNKGmwBWjSx ccI4/Slk/ePBXexJjssCrc9mpuc71q4E1HYRkexGcqXC/GFULn8JO8tcFXC8LW20DiZC 47trZua2BaHeQMFlRB+BUUd6L94rm73rzIM03CYU+/aXNLzb9uVEtDvWAqzfevuf3rq1 btHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ipvaN5Yb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c13-20020a056a00248d00b0054764d871b5si4714413pfv.230.2022.11.18.10.31.37; Fri, 18 Nov 2022 10:31:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ipvaN5Yb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242613AbiKRSZc (ORCPT + 99 others); Fri, 18 Nov 2022 13:25:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242666AbiKRSZJ (ORCPT ); Fri, 18 Nov 2022 13:25:09 -0500 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D465976CE for ; Fri, 18 Nov 2022 10:24:13 -0800 (PST) Received: by mail-pj1-x1033.google.com with SMTP id r61-20020a17090a43c300b00212f4e9cccdso9018193pjg.5 for ; Fri, 18 Nov 2022 10:24:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=ipvaN5YbaSj+UDPfbLsTFRb2ykvjMYQonjmXEHnGZ++qTvTA9Sxy0PQhXv8Cbdl7IP YfY5dXl96V64R4AQhICiv6XAF3KpRLfwPuuxumhVc1MXCUOEXtBUte6mnBErGhjI76Pa cpVnUMtRc8/UUBCuFADxBkE22TsJIoctj1XF8ysTFTm0Qty6JX8jCJdBlIPJpPA6gHXY V8YU6DKu29DKxDKP/27lxjI7xYa1rVpcTHUWhVRfBEgLpG+AF16uuQzZ1CbPHoI1Q6BU 5ENt0Vt5wlbJs+1jK3fmnmK7ifa8TG/HzHygopMxJaDmYKaOmie4oLteMOI/wcC/mK5b wpAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=sZKQH3yyqbLOfsQEt9UgkSnfR33SQUiMFZb5ffLPy069dkMXK8spUsSD/1+RTQs5bG mguXdpYnCfdcrFoeRtQsK8fg9ivTDWkQaC/WvDLSo38w7GAofJ7Q+cPD/wHHuJJkd2J3 MLNi/Vayecr8lOZCMq5Q1jDvn6Mcj/XwfEVygolTmxS8lQy8fmiGGmtphd3pXnCOWfc3 g6nNP/nyWga5GkYBfloxYIXDxFPD5lRA2zS53FRsV9P2r6Mmzjg0ruEErMjNB+6tt6yO WnuEMSbN/1oFAe7aVZ31QOtjtAV1IlOG4EASpzJg7Hs0dLFs9elLk88zVknGUIeUbo13 FuhA== X-Gm-Message-State: ANoB5plSm08UBId8aSObt0ClkGTWC1bh5qPQ3cJ5bcgxhGoTUUAsjhX4 AwQPvbuBGE9TNmpr2a3h+cM= X-Received: by 2002:a17:90a:bd86:b0:213:8cf1:4d9d with SMTP id z6-20020a17090abd8600b002138cf14d9dmr9198419pjr.5.1668795852078; Fri, 18 Nov 2022 10:24:12 -0800 (PST) Received: from localhost (fwdproxy-prn-117.fbsv.net. [2a03:2880:ff:75::face:b00c]) by smtp.gmail.com with ESMTPSA id x10-20020a170902a38a00b001869b988d93sm3998202pla.187.2022.11.18.10.24.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 10:24:11 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v5 2/6] zpool: clean out dead code Date: Fri, 18 Nov 2022 10:24:03 -0800 Message-Id: <20221118182407.82548-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221118182407.82548-1-nphamcs@gmail.com> References: <20221118182407.82548-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749859761972116829?= X-GMAIL-MSGID: =?utf-8?q?1749859761972116829?= From: Johannes Weiner There is a lot of provision for flexibility that isn't actually needed or used. Zswap (the only zpool user) always passes zpool_ops with an .evict method set. The backends who reclaim only do so for zswap, so they can also directly call zpool_ops without indirection or checks. Finally, there is no need to check the retries parameters and bail with -EINVAL in the reclaim function, when that's called just a few lines below with a hard-coded 8. There is no need to duplicate the evictable and sleep_mapped attrs from the driver in zpool_ops. Signed-off-by: Johannes Weiner Signed-off-by: Nhat Pham --- mm/z3fold.c | 36 +++++------------------------------- mm/zbud.c | 32 +++++--------------------------- mm/zpool.c | 10 ++-------- 3 files changed, 12 insertions(+), 66 deletions(-) -- 2.30.2 diff --git a/mm/z3fold.c b/mm/z3fold.c index cf71da10d04e..a4de0c317ac7 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -68,9 +68,6 @@ * Structures *****************/ struct z3fold_pool; -struct z3fold_ops { - int (*evict)(struct z3fold_pool *pool, unsigned long handle); -}; enum buddy { HEADLESS = 0, @@ -138,8 +135,6 @@ struct z3fold_header { * @stale: list of pages marked for freeing * @pages_nr: number of z3fold pages in the pool. * @c_handle: cache for z3fold_buddy_slots allocation - * @ops: pointer to a structure of user defined operations specified at - * pool creation time. * @zpool: zpool driver * @zpool_ops: zpool operations structure with an evict callback * @compact_wq: workqueue for page layout background optimization @@ -158,7 +153,6 @@ struct z3fold_pool { struct list_head stale; atomic64_t pages_nr; struct kmem_cache *c_handle; - const struct z3fold_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; struct workqueue_struct *compact_wq; @@ -907,13 +901,11 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, * z3fold_create_pool() - create a new z3fold pool * @name: pool name * @gfp: gfp flags when allocating the z3fold pool structure - * @ops: user-defined operations for the z3fold pool * * Return: pointer to the new z3fold pool or NULL if the metadata allocation * failed. */ -static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, - const struct z3fold_ops *ops) +static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp) { struct z3fold_pool *pool = NULL; int i, cpu; @@ -949,7 +941,6 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, if (!pool->release_wq) goto out_wq; INIT_WORK(&pool->work, free_pages_work); - pool->ops = ops; return pool; out_wq: @@ -1230,10 +1221,6 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) slots.pool = (unsigned long)pool | (1 << HANDLES_NOFREE); spin_lock(&pool->lock); - if (!pool->ops || !pool->ops->evict || retries == 0) { - spin_unlock(&pool->lock); - return -EINVAL; - } for (i = 0; i < retries; i++) { if (list_empty(&pool->lru)) { spin_unlock(&pool->lock); @@ -1319,17 +1306,17 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) } /* Issue the eviction callback(s) */ if (middle_handle) { - ret = pool->ops->evict(pool, middle_handle); + ret = pool->zpool_ops->evict(pool->zpool, middle_handle); if (ret) goto next; } if (first_handle) { - ret = pool->ops->evict(pool, first_handle); + ret = pool->zpool_ops->evict(pool->zpool, first_handle); if (ret) goto next; } if (last_handle) { - ret = pool->ops->evict(pool, last_handle); + ret = pool->zpool_ops->evict(pool->zpool, last_handle); if (ret) goto next; } @@ -1593,26 +1580,13 @@ static const struct movable_operations z3fold_mops = { * zpool ****************/ -static int z3fold_zpool_evict(struct z3fold_pool *pool, unsigned long handle) -{ - if (pool->zpool && pool->zpool_ops && pool->zpool_ops->evict) - return pool->zpool_ops->evict(pool->zpool, handle); - else - return -ENOENT; -} - -static const struct z3fold_ops z3fold_zpool_ops = { - .evict = z3fold_zpool_evict -}; - static void *z3fold_zpool_create(const char *name, gfp_t gfp, const struct zpool_ops *zpool_ops, struct zpool *zpool) { struct z3fold_pool *pool; - pool = z3fold_create_pool(name, gfp, - zpool_ops ? &z3fold_zpool_ops : NULL); + pool = z3fold_create_pool(name, gfp); if (pool) { pool->zpool = zpool; pool->zpool_ops = zpool_ops; diff --git a/mm/zbud.c b/mm/zbud.c index 6348932430b8..3acd26193920 100644 --- a/mm/zbud.c +++ b/mm/zbud.c @@ -74,10 +74,6 @@ struct zbud_pool; -struct zbud_ops { - int (*evict)(struct zbud_pool *pool, unsigned long handle); -}; - /** * struct zbud_pool - stores metadata for each zbud pool * @lock: protects all pool fields and first|last_chunk fields of any @@ -90,8 +86,6 @@ struct zbud_ops { * @lru: list tracking the zbud pages in LRU order by most recently * added buddy. * @pages_nr: number of zbud pages in the pool. - * @ops: pointer to a structure of user defined operations specified at - * pool creation time. * @zpool: zpool driver * @zpool_ops: zpool operations structure with an evict callback * @@ -110,7 +104,6 @@ struct zbud_pool { }; struct list_head lru; u64 pages_nr; - const struct zbud_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; }; @@ -212,12 +205,11 @@ static int num_free_chunks(struct zbud_header *zhdr) /** * zbud_create_pool() - create a new zbud pool * @gfp: gfp flags when allocating the zbud pool structure - * @ops: user-defined operations for the zbud pool * * Return: pointer to the new zbud pool or NULL if the metadata allocation * failed. */ -static struct zbud_pool *zbud_create_pool(gfp_t gfp, const struct zbud_ops *ops) +static struct zbud_pool *zbud_create_pool(gfp_t gfp) { struct zbud_pool *pool; int i; @@ -231,7 +223,6 @@ static struct zbud_pool *zbud_create_pool(gfp_t gfp, const struct zbud_ops *ops) INIT_LIST_HEAD(&pool->buddied); INIT_LIST_HEAD(&pool->lru); pool->pages_nr = 0; - pool->ops = ops; return pool; } @@ -419,8 +410,7 @@ static int zbud_reclaim_page(struct zbud_pool *pool, unsigned int retries) unsigned long first_handle = 0, last_handle = 0; spin_lock(&pool->lock); - if (!pool->ops || !pool->ops->evict || list_empty(&pool->lru) || - retries == 0) { + if (list_empty(&pool->lru)) { spin_unlock(&pool->lock); return -EINVAL; } @@ -444,12 +434,12 @@ static int zbud_reclaim_page(struct zbud_pool *pool, unsigned int retries) /* Issue the eviction callback(s) */ if (first_handle) { - ret = pool->ops->evict(pool, first_handle); + ret = pool->zpool_ops->evict(pool->zpool, first_handle); if (ret) goto next; } if (last_handle) { - ret = pool->ops->evict(pool, last_handle); + ret = pool->zpool_ops->evict(pool->zpool, last_handle); if (ret) goto next; } @@ -524,25 +514,13 @@ static u64 zbud_get_pool_size(struct zbud_pool *pool) * zpool ****************/ -static int zbud_zpool_evict(struct zbud_pool *pool, unsigned long handle) -{ - if (pool->zpool && pool->zpool_ops && pool->zpool_ops->evict) - return pool->zpool_ops->evict(pool->zpool, handle); - else - return -ENOENT; -} - -static const struct zbud_ops zbud_zpool_ops = { - .evict = zbud_zpool_evict -}; - static void *zbud_zpool_create(const char *name, gfp_t gfp, const struct zpool_ops *zpool_ops, struct zpool *zpool) { struct zbud_pool *pool; - pool = zbud_create_pool(gfp, zpool_ops ? &zbud_zpool_ops : NULL); + pool = zbud_create_pool(gfp); if (pool) { pool->zpool = zpool; pool->zpool_ops = zpool_ops; diff --git a/mm/zpool.c b/mm/zpool.c index 68facc193496..fc3a9893e107 100644 --- a/mm/zpool.c +++ b/mm/zpool.c @@ -21,9 +21,6 @@ struct zpool { struct zpool_driver *driver; void *pool; - const struct zpool_ops *ops; - bool evictable; - bool can_sleep_mapped; }; static LIST_HEAD(drivers_head); @@ -177,9 +174,6 @@ struct zpool *zpool_create_pool(const char *type, const char *name, gfp_t gfp, zpool->driver = driver; zpool->pool = driver->create(name, gfp, ops, zpool); - zpool->ops = ops; - zpool->evictable = driver->shrink && ops && ops->evict; - zpool->can_sleep_mapped = driver->sleep_mapped; if (!zpool->pool) { pr_err("couldn't create %s pool\n", type); @@ -380,7 +374,7 @@ u64 zpool_get_total_size(struct zpool *zpool) */ bool zpool_evictable(struct zpool *zpool) { - return zpool->evictable; + return zpool->driver->shrink; } /** @@ -391,7 +385,7 @@ bool zpool_evictable(struct zpool *zpool) */ bool zpool_can_sleep_mapped(struct zpool *zpool) { - return zpool->can_sleep_mapped; + return zpool->driver->sleep_mapped; } MODULE_LICENSE("GPL"); From patchwork Fri Nov 18 18:24:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 22444 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp355187wrr; Fri, 18 Nov 2022 10:51:43 -0800 (PST) X-Google-Smtp-Source: AA0mqf6+SnpwaDv9PlBP93zaY3L2RZ8LHwK5Ux4aZexzuiP9reWeAk6Lk/0qcgwtiOIequHwFgMO X-Received: by 2002:aa7:dd4b:0:b0:467:65a2:f635 with SMTP id o11-20020aa7dd4b000000b0046765a2f635mr7461098edw.106.1668797503329; Fri, 18 Nov 2022 10:51:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668797503; cv=none; d=google.com; s=arc-20160816; b=k20yH653LOhWYZVo+Wz+igAb8tQke0k9vf8PD4haO1BLtRKHC/X+nRbFb1FBe29H5V UOvNc/Sota+/3g3hTyZ+kCuiH6qfg8f57gbf7IT94C95HR8DoV+SH1Lve6tXp+kyFbz7 aPtHe3OH+UkZSW1p0MkoGiPZOWLWwnuMx5K8KHysFIxUsEMU6zX8R3mPrXQzDt+4M6xJ sfT5Qd6sMliT+dv9kRfS2asDXRClcUd7T+qsHQGllTJ6yTufVQLqcek9SVjDHQ00gSfy /V4SfJoW0d+onzUXUkOrdcZwhnQArTpPqXPis141Yr+qAuJEULGcRwnxZhYz49c47iMU T+rQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=J8/ndhQjyivI0+0njQmK8+MKUOKbGzHQcXZugQWCZvs=; b=KYKuE5PZo+AmL0j+9B6zI4QuKF6hr6/VWMFNNz7laxF9Ll/91P4zt/lFQ9wTLinLeG rl0KfgT8OSWHIwZBerz5zyjYviLcrWn8vHxBAsOv98mSJnU6KQO8FtoSbfcjkEky0Jh/ jsKmwCUtq/rJzzNCUdDQSdKfY0ZUGdBj9tSznVU5Ie6FG2/4bBLb+dbIR0PtWmZ5V29o 1yfhiFCuu6DLfgdFAt1kT+XioOFsUDpNdZAqZgUQ0g1iLXnApQJhAdxnkmL/+utp9U+j 5vhjmcVvKKy6Z0FCmgx6aRUMCtMIVutk94bwO1x5J4zlXxERCVthZD0j3/IikB7d5STT Wtew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=OycvcL2C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g17-20020a056402321100b004622c1b8597si3670771eda.407.2022.11.18.10.51.19; Fri, 18 Nov 2022 10:51:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=OycvcL2C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242315AbiKRSZg (ORCPT + 99 others); Fri, 18 Nov 2022 13:25:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242700AbiKRSZK (ORCPT ); Fri, 18 Nov 2022 13:25:10 -0500 Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09336976F7 for ; Fri, 18 Nov 2022 10:24:14 -0800 (PST) Received: by mail-pg1-x52f.google.com with SMTP id b62so5738388pgc.0 for ; Fri, 18 Nov 2022 10:24:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J8/ndhQjyivI0+0njQmK8+MKUOKbGzHQcXZugQWCZvs=; b=OycvcL2Cah2wGuDRenHaTYBXBsw34G9q8VY31KN1qLC3WdBMNaIlbxDYgF+4W82ZTP 09PMEgG/POl9U0Oken3hXY4GwP6yYg1qXlnThldeQLyc67Hn64GpUo1f3T8YM9znLQNl VhEVxePFrXGw0g3zejEjJ/6OhNi6DwOQSxRb/PEppOzKONuwNFak6D+OEbVXTC2O7IIY Wcu04fxOdRaUQbOpvZ45ojq7H+1jERgqu9BsNkLC86YGhkwFtW+GnJS/CHQT0k1vQVv3 7SodAoGzSVo3OS1ROgfyD+XG4IFU1s/b5AxP1fRB7nVBXKkxyw72Xr83880Sw7E4VHIh grBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J8/ndhQjyivI0+0njQmK8+MKUOKbGzHQcXZugQWCZvs=; b=BiZcVI6xc6JzN9VEeBaP4x0wkz+IBxHbVq4jakfbjO0ThrWOxRCuBrBERwWM65vORt EFEaEQBwEI8VKZ0+rO94EOX2v+oAxiBo2dD+X5FZv6wiBzlQxuc2FDI21AsXAnWKIpRK 6P1RrqzS9uabQ7sDsAlWHSughb1mDUvSBxt3of7h+JJbdMon+eD6yM96JzgKF6XQ/BAz PxWSyaiMa6QiDxnW1PZNbEW3y85LjDvq+lOrbohrk52dO6RZLn6GiJd3eiYqDh24kbVo hjW5i61EquqZ0tjfPi3Ge6yoajB2lo6urTRP5Ynxt54afwPUaWJqn5hr4S5lI6jJpJNo AXeg== X-Gm-Message-State: ANoB5plDIv4ykf+y3DJQ6fKG27fRBkdMH1HD68qB7+DwYopNXCshIgCh SKnU8Ef4nzShusbXL3VmHxE= X-Received: by 2002:a63:4746:0:b0:46e:beb0:9d2c with SMTP id w6-20020a634746000000b0046ebeb09d2cmr7882178pgk.117.1668795853446; Fri, 18 Nov 2022 10:24:13 -0800 (PST) Received: from localhost (fwdproxy-prn-014.fbsv.net. [2a03:2880:ff:e::face:b00c]) by smtp.gmail.com with ESMTPSA id x187-20020a6231c4000000b0056bc30e618dsm3511653pfx.38.2022.11.18.10.24.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 10:24:13 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v5 3/6] zsmalloc: Consolidate zs_pool's migrate_lock and size_class's locks Date: Fri, 18 Nov 2022 10:24:04 -0800 Message-Id: <20221118182407.82548-4-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221118182407.82548-1-nphamcs@gmail.com> References: <20221118182407.82548-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749861010988648633?= X-GMAIL-MSGID: =?utf-8?q?1749861010988648633?= Currently, zsmalloc has a hierarchy of locks, which includes a pool-level migrate_lock, and a lock for each size class. We have to obtain both locks in the hotpath in most cases anyway, except for zs_malloc. This exception will no longer exist when we introduce a LRU into the zs_pool for the new writeback functionality - we will need to obtain a pool-level lock to synchronize LRU handling even in zs_malloc. In preparation for zsmalloc writeback, consolidate these locks into a single pool-level lock, which drastically reduces the complexity of synchronization in zsmalloc. We have also benchmarked the lock consolidation to see the performance effect of this change on zram. First, we ran a synthetic FS workload on a server machine with 36 cores (same machine for all runs), using fs_mark -d ../zram1mnt -s 100000 -n 2500 -t 32 -k before and after for btrfs and ext4 on zram (FS usage is 80%). Here is the result (unit is file/second): With lock consolidation (btrfs): Average: 13520.2, Median: 13531.0, Stddev: 137.5961482019028 Without lock consolidation (btrfs): Average: 13487.2, Median: 13575.0, Stddev: 309.08283679298665 With lock consolidation (ext4): Average: 16824.4, Median: 16839.0, Stddev: 89.97388510006668 Without lock consolidation (ext4) Average: 16958.0, Median: 16986.0, Stddev: 194.7370021336469 As you can see, we observe a 0.3% regression for btrfs, and a 0.9% regression for ext4. This is a small, barely measurable difference in my opinion. For a more realistic scenario, we also tries building the kernel on zram. Here is the time it takes (in seconds): With lock consolidation (btrfs): real Average: 319.6, Median: 320.0, Stddev: 0.8944271909999159 user Average: 6894.2, Median: 6895.0, Stddev: 25.528415540334656 sys Average: 521.4, Median: 522.0, Stddev: 1.51657508881031 Without lock consolidation (btrfs): real Average: 319.8, Median: 320.0, Stddev: 0.8366600265340756 user Average: 6896.6, Median: 6899.0, Stddev: 16.04057355583023 sys Average: 520.6, Median: 521.0, Stddev: 1.140175425099138 With lock consolidation (ext4): real Average: 320.0, Median: 319.0, Stddev: 1.4142135623730951 user Average: 6896.8, Median: 6878.0, Stddev: 28.621670111997307 sys Average: 521.2, Median: 521.0, Stddev: 1.7888543819998317 Without lock consolidation (ext4) real Average: 319.6, Median: 319.0, Stddev: 0.8944271909999159 user Average: 6886.2, Median: 6887.0, Stddev: 16.93221781102523 sys Average: 520.4, Median: 520.0, Stddev: 1.140175425099138 The difference is entirely within the noise of a typical run on zram. This hardly justifies the complexity of maintaining both the pool lock and the class lock. In fact, for writeback, we would need to introduce yet another lock to prevent data races on the pool's LRU, further complicating the lock handling logic. IMHO, it is just better to collapse all of these into a single pool-level lock. Suggested-by: Johannes Weiner Signed-off-by: Nhat Pham Acked-by: Minchan Kim Acked-by: Johannes Weiner --- mm/zsmalloc.c | 87 ++++++++++++++++++++++----------------------------- 1 file changed, 37 insertions(+), 50 deletions(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index d03941cace2c..326faa751f0a 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -33,8 +33,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock - * class->lock + * pool->lock * zspage->lock */ @@ -192,7 +191,6 @@ static const int fullness_threshold_frac = 4; static size_t huge_class_size; struct size_class { - spinlock_t lock; struct list_head fullness_list[NR_ZS_FULLNESS]; /* * Size of objects stored in this class. Must be multiple @@ -247,8 +245,7 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + spinlock_t lock; }; struct zspage { @@ -355,7 +352,7 @@ static void cache_free_zspage(struct zs_pool *pool, struct zspage *zspage) kmem_cache_free(pool->zspage_cachep, zspage); } -/* class->lock(which owns the handle) synchronizes races */ +/* pool->lock(which owns the handle) synchronizes races */ static void record_obj(unsigned long handle, unsigned long obj) { *(unsigned long *)handle = obj; @@ -452,7 +449,7 @@ static __maybe_unused int is_first_page(struct page *page) return PagePrivate(page); } -/* Protected by class->lock */ +/* Protected by pool->lock */ static inline int get_zspage_inuse(struct zspage *zspage) { return zspage->inuse; @@ -597,13 +594,13 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); + spin_lock(&pool->lock); class_almost_full = zs_stat_get(class, CLASS_ALMOST_FULL); class_almost_empty = zs_stat_get(class, CLASS_ALMOST_EMPTY); obj_allocated = zs_stat_get(class, OBJ_ALLOCATED); obj_used = zs_stat_get(class, OBJ_USED); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -916,7 +913,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, get_zspage_mapping(zspage, &class_idx, &fg); - assert_spin_locked(&class->lock); + assert_spin_locked(&pool->lock); VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(fg != ZS_EMPTY); @@ -1247,19 +1244,19 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + spin_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_location(obj, &page, &obj_idx); zspage = get_zspage(page); /* - * migration cannot move any zpages in this zspage. Here, class->lock + * migration cannot move any zpages in this zspage. Here, pool->lock * is too heavy since callers would take some time until they calls * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + spin_unlock(&pool->lock); class = zspage_class(pool, zspage); off = (class->size * obj_idx) & ~PAGE_MASK; @@ -1412,8 +1409,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) size += ZS_HANDLE_SIZE; class = pool->size_class[get_size_class_index(size)]; - /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + /* pool->lock effectively protects the zpage migration */ + spin_lock(&pool->lock); zspage = find_get_zspage(class); if (likely(zspage)) { obj = obj_malloc(pool, zspage, handle); @@ -1421,12 +1418,12 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) fix_fullness_group(class, zspage); record_obj(handle, obj); class_stat_inc(class, OBJ_USED, 1); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); return handle; } - spin_unlock(&class->lock); + spin_unlock(&pool->lock); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1434,7 +1431,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + spin_lock(&pool->lock); obj = obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1447,7 +1444,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); return handle; } @@ -1491,16 +1488,14 @@ void zs_free(struct zs_pool *pool, unsigned long handle) return; /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + spin_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_page(obj, &f_page); zspage = get_zspage(f_page); class = zspage_class(pool, zspage); - spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); obj_free(class->size, obj); class_stat_dec(class, OBJ_USED, 1); @@ -1510,7 +1505,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) free_zspage(pool, class, zspage); out: - spin_unlock(&class->lock); + spin_unlock(&pool->lock); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1867,16 +1862,12 @@ static int zs_page_migrate(struct page *newpage, struct page *page, pool = zspage->pool; /* - * The pool migrate_lock protects the race between zpage migration + * The pool's lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + spin_lock(&pool->lock); class = zspage_class(pool, zspage); - /* - * the class lock protects zpage alloc/free in the zspage. - */ - spin_lock(&class->lock); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1906,10 +1897,9 @@ static int zs_page_migrate(struct page *newpage, struct page *page, replace_sub_page(class, zspage, newpage, page); /* * Since we complete the data copy and set up new zspage structure, - * it's okay to release migration_lock. + * it's okay to release the pool's lock. */ - write_unlock(&pool->migrate_lock); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); dec_zspage_isolation(zspage); migrate_write_unlock(zspage); @@ -1964,9 +1954,9 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + spin_lock(&pool->lock); list_splice_init(&class->fullness_list[ZS_EMPTY], &free_pages); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1976,9 +1966,9 @@ static void async_free_zspage(struct work_struct *work) get_zspage_mapping(zspage, &class_idx, &fullness); VM_BUG_ON(fullness != ZS_EMPTY); class = pool->size_class[class_idx]; - spin_lock(&class->lock); + spin_lock(&pool->lock); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + spin_unlock(&pool->lock); } }; @@ -2039,10 +2029,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, struct zspage *dst_zspage = NULL; unsigned long pages_freed = 0; - /* protect the race between zpage migration and zs_free */ - write_lock(&pool->migrate_lock); - /* protect zpage allocation/free */ - spin_lock(&class->lock); + /* + * protect the race between zpage migration and zs_free + * as well as zpage allocation/free + */ + spin_lock(&pool->lock); while ((src_zspage = isolate_zspage(class, true))) { /* protect someone accessing the zspage(i.e., zs_map_object) */ migrate_write_lock(src_zspage); @@ -2067,7 +2058,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); migrate_write_unlock(dst_zspage); dst_zspage = NULL; - if (rwlock_is_contended(&pool->migrate_lock)) + if (spin_is_contended(&pool->lock)) break; } @@ -2084,11 +2075,9 @@ static unsigned long __zs_compact(struct zs_pool *pool, pages_freed += class->pages_per_zspage; } else migrate_write_unlock(src_zspage); - spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + spin_unlock(&pool->lock); cond_resched(); - write_lock(&pool->migrate_lock); - spin_lock(&class->lock); + spin_lock(&pool->lock); } if (src_zspage) { @@ -2096,8 +2085,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, migrate_write_unlock(src_zspage); } - spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + spin_unlock(&pool->lock); return pages_freed; } @@ -2200,7 +2188,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + spin_lock_init(&pool->lock); pool->name = kstrdup(name, GFP_KERNEL); if (!pool->name) @@ -2271,7 +2259,6 @@ struct zs_pool *zs_create_pool(const char *name) class->index = i; class->pages_per_zspage = pages_per_zspage; class->objs_per_zspage = objs_per_zspage; - spin_lock_init(&class->lock); pool->size_class[i] = class; for (fullness = ZS_EMPTY; fullness < NR_ZS_FULLNESS; fullness++) From patchwork Fri Nov 18 18:24:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 22445 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp355575wrr; Fri, 18 Nov 2022 10:52:39 -0800 (PST) X-Google-Smtp-Source: AA0mqf47hXc+OXnGYwFIGSQoHfUenirNoCxj1ras+tgXtObQftFJ8PNJ4Ez4821PmDmnq3qbKzSw X-Received: by 2002:a17:906:28c4:b0:78d:b8b3:f027 with SMTP id p4-20020a17090628c400b0078db8b3f027mr7047286ejd.439.1668797558990; Fri, 18 Nov 2022 10:52:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668797558; cv=none; d=google.com; s=arc-20160816; b=Brk/0JvGbxVxGWX7E/pmEnTUucE55WFIhwuActmYTJqp9kg68ydeQWSjHnW5/LE+oR nFh+MsrOOxlIvNgo47GF3/zRaXBZ8m7NH5RMBhz0HqoP5xLT2mL4GlZpc/EobdD+sdkf skF8WrvvwFkbrHV/vBRyBN7DM6khuLBelJ3beHYkyG2uK88Axn+M1WLS5LBl1+NP+qNO N0IYsRtw0U0II7lAB3wMvFxtTJ0AyPJZH7GQTX+EX/JsVqJKZA2HCwiALGn9sBpn5Xru zd7cl+pP40daEpf/xh9JnaCkZhXsEAOu9uMBMEdG4kofOL0vz4okUbVY+oHTvyvHXqd5 tA8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+nOnhYHcTkk9/NzeLLNRVXEnZ77Tqa2lTVIFHU0KWSM=; b=aZ/z8ezgkP/Lflcm0x+SrebQRtCosFmFBgjIopqCGDUa+t5GrSU2+ZEldMXBmFdWoD 1B2bmpZ9Djxgyjf6A7RlVY5savISV4w7ymyWtGOEa55LN45WmJB8nJXD7Ggl4my77DuG Tyyc5NLnAAHCqnAgVoDjH26rqMAPaB/VrwxfQfK2V5V+4CMwBzXLkubs+0/cit+zaa37 rI9HYe0sLm8NDAnP+r87nJAOkG5oi81vDX8KLwKtL1jLHdClZH7GLbS34SmeI/h3YKcr sQ1pKBZVMGUmnQ0mDLwDElpAVGV7i/VEIRmrz8FYz+Bs+GPv2Ky1Qf5Y5ziavm1W2Y3H sEgg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=MggomevQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cy2-20020a0564021c8200b00462750bbe08si3407938edb.522.2022.11.18.10.52.13; Fri, 18 Nov 2022 10:52:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=MggomevQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242814AbiKRSZl (ORCPT + 99 others); Fri, 18 Nov 2022 13:25:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242706AbiKRSZL (ORCPT ); Fri, 18 Nov 2022 13:25:11 -0500 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33A5A93CC0 for ; Fri, 18 Nov 2022 10:24:15 -0800 (PST) Received: by mail-pj1-x102d.google.com with SMTP id k5so5196481pjo.5 for ; Fri, 18 Nov 2022 10:24:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+nOnhYHcTkk9/NzeLLNRVXEnZ77Tqa2lTVIFHU0KWSM=; b=MggomevQAHa406kdroPp0BE2cDvQIay2Gb8NdQCWaLxdg4h8AQot1ASw4tHnrl7FLd 56+aica0cqemAuYdQspMkeDpWecDQelDR9Gc9qbXdKo1TguX0YVwNP11Jvcbka9DyMf+ cORxRCKMhXQ7OyYFD2CRDeypFzgvstHAfqBBEM6IA+kdMR1ocfxwFRlEe0sxcoWY3i8D 59gaHU51TAmpnx0iN7dZYwT7Xy8bComPIQp3m/pze4CX4XMOM9l/S9jx3itXvUzIFRhB NOpaB2oc7zutl+sPpv4XDKz1Jng9/0SSFGfSAH3pYZiUBDrD1YeHDg+r4oh4Zt/0o3/c qM/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+nOnhYHcTkk9/NzeLLNRVXEnZ77Tqa2lTVIFHU0KWSM=; b=vW4HXeeaey/fvMTnXPn7PBMmr/I7ZmA1AQQnHLexGcKjazuCDmQHylmKChJZIyVgD+ 5ahSfVQSZWV+YsxvqZ0WrS/dlyPizHFUQjYwY2Tr3NUfwn94x8P391W3TmAFhUi+p8dA 2g5xBGveeG0h5zLpOg5kNXhKwJIz36CyVdx7O8N8GFjPiSyzx1QgUSl4JaOy/59sOUhT sejqohv3MN5rdTG8xGcM3mIfElmNwtJ1xSTU1JgOSgexgwJinPLouTgzlzPlsnyorhB2 51v65/mcUzLbSdVxzuNKV9TxJH/B/hygO977I2tPmqI7oZiRpy5wvlSZeJfIpuwq5HpX sFYw== X-Gm-Message-State: ANoB5pnrkNunthkZ6zwEH7dtw1SBsBnc0KUnYvz5UI37vaycSiM857Ja LKxt5l1aqWrumHaZ0qDj14TVvWtE7Qn5IQ== X-Received: by 2002:a17:90a:6343:b0:213:b853:5e45 with SMTP id v3-20020a17090a634300b00213b8535e45mr14994881pjs.97.1668795854724; Fri, 18 Nov 2022 10:24:14 -0800 (PST) Received: from localhost (fwdproxy-prn-010.fbsv.net. [2a03:2880:ff:a::face:b00c]) by smtp.gmail.com with ESMTPSA id rj6-20020a17090b3e8600b00217cdc4b0a5sm5671207pjb.16.2022.11.18.10.24.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 10:24:14 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v5 4/6] zsmalloc: Add a LRU to zs_pool to keep track of zspages in LRU order Date: Fri, 18 Nov 2022 10:24:05 -0800 Message-Id: <20221118182407.82548-5-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221118182407.82548-1-nphamcs@gmail.com> References: <20221118182407.82548-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749861069406643876?= X-GMAIL-MSGID: =?utf-8?q?1749861069406643876?= This helps determines the coldest zspages as candidates for writeback. Signed-off-by: Nhat Pham --- mm/zsmalloc.c | 45 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 326faa751f0a..9e7b54324181 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -239,6 +239,11 @@ struct zs_pool { /* Compact classes */ struct shrinker shrinker; +#ifdef CONFIG_ZPOOL + /* List tracking the zspages in LRU order by most recently added object */ + struct list_head lru; +#endif + #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; #endif @@ -260,6 +265,12 @@ struct zspage { unsigned int freeobj; struct page *first_page; struct list_head list; /* fullness list */ + +#ifdef CONFIG_ZPOOL + /* links the zspage to the lru list in the pool */ + struct list_head lru; +#endif + struct zs_pool *pool; #ifdef CONFIG_COMPACTION rwlock_t lock; @@ -352,6 +363,18 @@ static void cache_free_zspage(struct zs_pool *pool, struct zspage *zspage) kmem_cache_free(pool->zspage_cachep, zspage); } +#ifdef CONFIG_ZPOOL +/* Moves the zspage to the front of the zspool's LRU */ +static void move_to_front(struct zs_pool *pool, struct zspage *zspage) +{ + assert_spin_locked(&pool->lock); + + if (!list_empty(&zspage->lru)) + list_del(&zspage->lru); + list_add(&zspage->lru, &pool->lru); +} +#endif + /* pool->lock(which owns the handle) synchronizes races */ static void record_obj(unsigned long handle, unsigned long obj) { @@ -953,6 +976,9 @@ static void free_zspage(struct zs_pool *pool, struct size_class *class, } remove_zspage(class, zspage, ZS_EMPTY); +#ifdef CONFIG_ZPOOL + list_del(&zspage->lru); +#endif __free_zspage(pool, class, zspage); } @@ -998,6 +1024,10 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) off %= PAGE_SIZE; } +#ifdef CONFIG_ZPOOL + INIT_LIST_HEAD(&zspage->lru); +#endif + set_freeobj(zspage, 0); } @@ -1418,9 +1448,8 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) fix_fullness_group(class, zspage); record_obj(handle, obj); class_stat_inc(class, OBJ_USED, 1); - spin_unlock(&pool->lock); - return handle; + goto out; } spin_unlock(&pool->lock); @@ -1444,6 +1473,11 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); +out: +#ifdef CONFIG_ZPOOL + /* Move the zspage to front of pool's LRU */ + move_to_front(pool, zspage); +#endif spin_unlock(&pool->lock); return handle; @@ -1967,6 +2001,9 @@ static void async_free_zspage(struct work_struct *work) VM_BUG_ON(fullness != ZS_EMPTY); class = pool->size_class[class_idx]; spin_lock(&pool->lock); +#ifdef CONFIG_ZPOOL + list_del(&zspage->lru); +#endif __free_zspage(pool, class, zspage); spin_unlock(&pool->lock); } @@ -2278,6 +2315,10 @@ struct zs_pool *zs_create_pool(const char *name) */ zs_register_shrinker(pool); +#ifdef CONFIG_ZPOOL + INIT_LIST_HEAD(&pool->lru); +#endif + return pool; err: From patchwork Fri Nov 18 18:24:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 22439 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp350581wrr; Fri, 18 Nov 2022 10:39:04 -0800 (PST) X-Google-Smtp-Source: AA0mqf7fRknrI7+MK7L79vr2FZOSOrlJ8z1atcqs7FPmwUes2lK2Hlh42J2flYk70piQgApUyHdO X-Received: by 2002:a17:90a:5911:b0:218:7b32:d353 with SMTP id k17-20020a17090a591100b002187b32d353mr7118266pji.100.1668796743816; Fri, 18 Nov 2022 10:39:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668796743; cv=none; d=google.com; s=arc-20160816; b=MwqpHV5bJhUoJ6XZBc5+kG38qNhCptZAusdhzo9ENvNSekGrZQuJ6vNociYuMpLi/1 hU5wMfsMwSZLY38wv935pYUzQGQcUJmY0nNRX3Od6qldBZ+Z5P/2f2xjfwTFEdQFgSXJ LHq48Re0ncSSVqAB903XFhitNSgrLH8LmnxzP2hTqMoYAtGVhwjea0psjru9F+FpeV/C +zLhaKdsO/2eeB3x+WQI1XjLgQqr0KlEtyvjr3Njw7QPUI0w5v6da+KY94FC6hL20zcX n0OwsW/UoOjUBtlHsBznKo8x7xRxwHVKtMj7Wd1yK0Wz4rG+lFHI08g0loW/5NCndwIk mVrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=awQN/ZCJSxJ2Bey51fZL37YCVo4g2Qf+pzYIcGHdiWA=; b=i/yAPG5bjRAedup7Idpg/tUVI3jL0h2goFLfTw2gk8RcWAIFRB4/0YmXF4pG6Ed8Qq rK5t6zO+j0INVfD3R194YTUtmrpYpZ91eNTCzNiL0vooiieVzJPVjGpJ3CEHjdkUOK+E tLulaZbV/qAs1TfcX7z19R9aEk1P6GXgfjg4PG/HvNg/l1Bee1e3uq48BcZGmMiTpV1p Bk3sMmBOSVvMpUa5XGke5FC3bJyY3BFmDzb5odp+89JgNKfwFL4UgFYusIocm/5EJ0Qt 1e0uFK94yTHThHJeyMvHjpQJh3gKVS1Rn0smbkEtRm7d0BDRU4HKSiumbvUNUIP8ucw5 4M3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=XHyzobnK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w1-20020a636201000000b0043c0b4f2b68si4307348pgb.318.2022.11.18.10.38.49; Fri, 18 Nov 2022 10:39:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=XHyzobnK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242403AbiKRSZp (ORCPT + 99 others); Fri, 18 Nov 2022 13:25:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242726AbiKRSZM (ORCPT ); Fri, 18 Nov 2022 13:25:12 -0500 Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D8539BA0A for ; Fri, 18 Nov 2022 10:24:16 -0800 (PST) Received: by mail-pl1-x629.google.com with SMTP id 4so5311045pli.0 for ; Fri, 18 Nov 2022 10:24:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=awQN/ZCJSxJ2Bey51fZL37YCVo4g2Qf+pzYIcGHdiWA=; b=XHyzobnKe+oOaTjXSZzBJ1qS4nD4Q+36AJReyTdqnyvQbIRh0qPu4d+50B1ntmMX1z 9D55pqnESb27MCIbC8engqlryb+LhkQQxOGgVxDN95uxG6ebO/zB9CHKuBZJhKT6IHYD Cksw1RnIxRkMOjeEWGKU6N/jv0/d8f8ZQC+ZzdTSF29Ols5t9pxDZe4VFtM2b0IdL1WF YdqOBimS4bedY5ugdRKIB+KWN4OeGWgdORHmG5y/niC+nJmS59fujKKFpWTEtrmiGarj Whfn9Z69Q1Glv6N29+tbeWYSEy5RSTwZz5mg7pNQRE5xmfmayOptcVZzZ2uDdPNEX5pc 10uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=awQN/ZCJSxJ2Bey51fZL37YCVo4g2Qf+pzYIcGHdiWA=; b=phPJgjquYmRxinlA8XUU2+l+2jarYalLnYhWpdzosz3p7hem7vKv8x81m2sN3aHy9T WSo0jW8jtM9yDj8CPLD3BNg/rQausvJZCxoCeTpy9mcvLoZjxWrTbYZ8trdjKzRLXVI9 2U80f/p2BTh2/DkDYo8aSaMLmxDyeG6hVzSfpcjV/cH738vSjjoYU4WQ8HOFynxDxYU8 SI0wguSPtMySD9safm6lHKtMRd+SySfZ8RjIPVyPZNUBd0Othxa83eAmGO99Ghq8HEVn A3YGrLh+QRph3GE2zKn3LBXNHaplAO4EELsnubtLZmc3Mym3m01CK/uAl+L967oqqzaz zzbA== X-Gm-Message-State: ANoB5pldUphs9pfTok7Nd7Ky+UMVScai9E/hhPx3UYg+xnCia3/bXhV+ d2BPSBIVrPZM5MIIy2P8T2z7Iz8oPtmWqQ== X-Received: by 2002:a17:902:b613:b0:188:f570:7bd6 with SMTP id b19-20020a170902b61300b00188f5707bd6mr692050pls.97.1668795855932; Fri, 18 Nov 2022 10:24:15 -0800 (PST) Received: from localhost (fwdproxy-prn-011.fbsv.net. [2a03:2880:ff:b::face:b00c]) by smtp.gmail.com with ESMTPSA id a16-20020a170902ecd000b0017f49b41c12sm4051234plh.173.2022.11.18.10.24.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 10:24:15 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v5 5/6] zsmalloc: Add zpool_ops field to zs_pool to store evict handlers Date: Fri, 18 Nov 2022 10:24:06 -0800 Message-Id: <20221118182407.82548-6-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221118182407.82548-1-nphamcs@gmail.com> References: <20221118182407.82548-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749860214823946746?= X-GMAIL-MSGID: =?utf-8?q?1749860214823946746?= This adds a new field to zs_pool to store evict handlers for writeback, analogous to the zbud allocator. Signed-off-by: Nhat Pham Acked-by: Minchan Kim --- mm/zsmalloc.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9e7b54324181..3ff86f57d08c 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -242,6 +242,8 @@ struct zs_pool { #ifdef CONFIG_ZPOOL /* List tracking the zspages in LRU order by most recently added object */ struct list_head lru; + struct zpool *zpool; + const struct zpool_ops *zpool_ops; #endif #ifdef CONFIG_ZSMALLOC_STAT @@ -394,7 +396,14 @@ static void *zs_zpool_create(const char *name, gfp_t gfp, * different contexts and its caller must provide a valid * gfp mask. */ - return zs_create_pool(name); + struct zs_pool *pool = zs_create_pool(name); + + if (pool) { + pool->zpool = zpool; + pool->zpool_ops = zpool_ops; + } + + return pool; } static void zs_zpool_destroy(void *pool) From patchwork Fri Nov 18 18:24:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 22438 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp348923wrr; Fri, 18 Nov 2022 10:34:42 -0800 (PST) X-Google-Smtp-Source: AA0mqf4ZZvNpUqwMBtL/RytPKCR+YYJbzusT3MvLVHILkJRUSI1k5v1YfF7Rro1fXG4QEvWYpQ13 X-Received: by 2002:a17:90a:d145:b0:211:7e51:9d65 with SMTP id t5-20020a17090ad14500b002117e519d65mr14525512pjw.220.1668796482598; Fri, 18 Nov 2022 10:34:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668796482; cv=none; d=google.com; s=arc-20160816; b=STFdJcQ5Lmd8J6U28g0huZAOyUzb/NRgKlVFjfAQk7lYJ8TS3ZuIubom3UwIhIaLcn QqkFyq8lUMAn3TVkhz72ittHyRrgnOyGVa6e+b+sChvZEHBYx4pwmxZCHUi1O7j6jG2a lSmR12N430c7igVbZRAyTOgxvAHdm5NxkoN8fCeA873XKMeBUPuSVsCvTUbKkU3K0WsL DjAjNAkBxjMm+SVfYUvtXLa3gRtA+NQq4drkn76sR/S5Nccj30cypbBCekPm1u+lzOI8 9GxPR6UTU9E65+5s3Ny45D6L4h1sesfxyEHGaNhauAPTmlAUrkmIkstYwMlpxYHg7+kL CMHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5iZr6qlLAxIk/PcvnkVOO0x20MGB/KyzSEMXijENIaY=; b=eIHXZu1chbxmogdCGaZH/Y9+srT56E/AtkSnrSTF62NLlIhS6iX2Q0sTOW6dWXR7xl Uv0YmHWJP/7bk4lCQu/HynzsQZ79mAzEoKIjLKrWJfYMQOaH9AJt3zs334TAhowblzM8 bh0Wsi9sWnzdTswzaSHQfj7Yuj/wPmGWbW5GK3I21utWlrXPC73w1/W65iyw5n3jzdGb 34Hg9nUuL3fZ25GLkoPEbwBxYVjbwk2R9bvk/KAwd6x0ynSGQqmHqpJIQzvIScBtW+6c PSnyoJNTydH0A5+NCsIgznm7c98LeBeGMsGBD3JUyyUh1R3/LFhORUM1l4slK9yVG+1L 9NdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=JFcNmxk+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p68-20020a634247000000b00476df6a66f7si4493059pga.264.2022.11.18.10.34.28; Fri, 18 Nov 2022 10:34:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=JFcNmxk+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242742AbiKRSZw (ORCPT + 99 others); Fri, 18 Nov 2022 13:25:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242663AbiKRSZO (ORCPT ); Fri, 18 Nov 2022 13:25:14 -0500 Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C48579BA25 for ; Fri, 18 Nov 2022 10:24:17 -0800 (PST) Received: by mail-pl1-x62c.google.com with SMTP id jn7so3415878plb.13 for ; Fri, 18 Nov 2022 10:24:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5iZr6qlLAxIk/PcvnkVOO0x20MGB/KyzSEMXijENIaY=; b=JFcNmxk+HYx+2nlmTKGyuGpLtysB/nXfeKQerLt9vCTPkTSJmB892sdhkbvz1IYnbx +EYiY8n+yk2xET8vbkOOqlEJc3xUTfWSbdTQB9/5mWcu3hVQC0KPtFlqVhFZTt+Ji3Iy n2ADtmzAsbROyoH7aIy1MbHYfJmIL+mTy7HK37CfaWPHwS98GcgWzW4KNwh1yk26r36C +9uzfIX7GUZsXRpuVd2hzkBxlJxsByd7mow8ZwkPKR1up7BiYVYgqv+hj4ErztBfip1D aOmn9+z07NtQ8sWRlypCUnHXbnmfUr3coIofB54IaD//P9dxQFZpUq9cDnln3h3jT3Mh Y4Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5iZr6qlLAxIk/PcvnkVOO0x20MGB/KyzSEMXijENIaY=; b=as8hLOrmUL3s2iJw1quJ8e9MEp3OM+8rAhcqx7Hs9XgagbZ0NI28frOCLDJLBPF9Lg qIPSHF5jxmAYVwDDWyP0a5xurwCvi08STUhXlIyIDbOe1QmdM0d/L01YvLjzUDyhlr7u yRJSJ4tUmxiyxp2Z2O+aj+a7cvMOqMGnYcqRFJh6Zx8sJFq/PesUse2XcqvuknCjP49J VgShJDatFO5vj+2h2zD1ao0loIXGl+zqFucjM48uhyR9n4goM6F8BUYMV6YLawWzmIop daYIQVSivetey8jqj6SfKnMnrhgfKCdhtndtIuYaFEym4F7voJR5yFMFw94sed58FDel cmZQ== X-Gm-Message-State: ANoB5plsg5ekn754+abwdQXk/qda1Jc+cLcKCk9QXVNiVwwhjm5f4Xmo yRmLOIZpiUBMS8cjqB0KbDI= X-Received: by 2002:a17:902:8c8e:b0:179:f6ed:2ca8 with SMTP id t14-20020a1709028c8e00b00179f6ed2ca8mr775802plo.14.1668795857199; Fri, 18 Nov 2022 10:24:17 -0800 (PST) Received: from localhost (fwdproxy-prn-119.fbsv.net. [2a03:2880:ff:77::face:b00c]) by smtp.gmail.com with ESMTPSA id c15-20020a63ea0f000000b004393f60db36sm3146190pgi.32.2022.11.18.10.24.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 10:24:16 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v5 6/6] zsmalloc: Implement writeback mechanism for zsmalloc Date: Fri, 18 Nov 2022 10:24:07 -0800 Message-Id: <20221118182407.82548-7-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221118182407.82548-1-nphamcs@gmail.com> References: <20221118182407.82548-1-nphamcs@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749859940133028002?= X-GMAIL-MSGID: =?utf-8?q?1749859940133028002?= This commit adds the writeback mechanism for zsmalloc, analogous to the zbud allocator. Zsmalloc will attempt to determine the coldest zspage (i.e least recently used) in the pool, and attempt to write back all the stored compressed objects via the pool's evict handler. Signed-off-by: Nhat Pham --- mm/zsmalloc.c | 193 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 182 insertions(+), 11 deletions(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 3ff86f57d08c..d73b9f9e9adf 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -271,12 +271,13 @@ struct zspage { #ifdef CONFIG_ZPOOL /* links the zspage to the lru list in the pool */ struct list_head lru; + bool under_reclaim; + /* list of unfreed handles whose objects have been reclaimed */ + unsigned long *deferred_handles; #endif struct zs_pool *pool; -#ifdef CONFIG_COMPACTION rwlock_t lock; -#endif }; struct mapping_area { @@ -297,10 +298,11 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -#ifdef CONFIG_COMPACTION static void migrate_lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); + +#ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage); static void migrate_write_lock_nested(struct zspage *zspage); static void migrate_write_unlock(struct zspage *zspage); @@ -308,9 +310,6 @@ static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage); #else -static void migrate_lock_init(struct zspage *zspage) {} -static void migrate_read_lock(struct zspage *zspage) {} -static void migrate_read_unlock(struct zspage *zspage) {} static void migrate_write_lock(struct zspage *zspage) {} static void migrate_write_lock_nested(struct zspage *zspage) {} static void migrate_write_unlock(struct zspage *zspage) {} @@ -425,6 +424,27 @@ static void zs_zpool_free(void *pool, unsigned long handle) zs_free(pool, handle); } +static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries); + +static int zs_zpool_shrink(void *pool, unsigned int pages, + unsigned int *reclaimed) +{ + unsigned int total = 0; + int ret = -EINVAL; + + while (total < pages) { + ret = zs_reclaim_page(pool, 8); + if (ret < 0) + break; + total++; + } + + if (reclaimed) + *reclaimed = total; + + return ret; +} + static void *zs_zpool_map(void *pool, unsigned long handle, enum zpool_mapmode mm) { @@ -463,6 +483,7 @@ static struct zpool_driver zs_zpool_driver = { .malloc_support_movable = true, .malloc = zs_zpool_malloc, .free = zs_zpool_free, + .shrink = zs_zpool_shrink, .map = zs_zpool_map, .unmap = zs_zpool_unmap, .total_size = zs_zpool_total_size, @@ -936,6 +957,23 @@ static int trylock_zspage(struct zspage *zspage) return 0; } +#ifdef CONFIG_ZPOOL +/* + * Free all the deferred handles whose objects are freed in zs_free. + */ +static void free_handles(struct zs_pool *pool, struct zspage *zspage) +{ + unsigned long handle = (unsigned long)zspage->deferred_handles; + + while (handle) { + unsigned long nxt_handle = handle_to_obj(handle); + + cache_free_handle(pool, handle); + handle = nxt_handle; + } +} +#endif + static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { @@ -950,6 +988,11 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(fg != ZS_EMPTY); +#ifdef CONFIG_ZPOOL + /* Free all deferred handles from zs_free */ + free_handles(pool, zspage); +#endif + next = page = get_first_page(zspage); do { VM_BUG_ON_PAGE(!PageLocked(page), page); @@ -1035,6 +1078,8 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) #ifdef CONFIG_ZPOOL INIT_LIST_HEAD(&zspage->lru); + zspage->under_reclaim = false; + zspage->deferred_handles = NULL; #endif set_freeobj(zspage, 0); @@ -1542,12 +1587,26 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_free(class->size, obj); class_stat_dec(class, OBJ_USED, 1); + +#ifdef CONFIG_ZPOOL + if (zspage->under_reclaim) { + /* + * Reclaim needs the handles during writeback. It'll free + * them along with the zspage when it's done with them. + * + * Record current deferred handle at the memory location + * whose address is given by handle. + */ + record_obj(handle, (unsigned long)zspage->deferred_handles); + zspage->deferred_handles = (unsigned long *)handle; + spin_unlock(&pool->lock); + return; + } +#endif fullness = fix_fullness_group(class, zspage); - if (fullness != ZS_EMPTY) - goto out; + if (fullness == ZS_EMPTY) + free_zspage(pool, class, zspage); - free_zspage(pool, class, zspage); -out: spin_unlock(&pool->lock); cache_free_handle(pool, handle); } @@ -1747,7 +1806,7 @@ static enum fullness_group putback_zspage(struct size_class *class, return fullness; } -#ifdef CONFIG_COMPACTION +#if defined(CONFIG_ZPOOL) || defined(CONFIG_COMPACTION) /* * To prevent zspage destroy during migration, zspage freeing should * hold locks of all pages in the zspage. @@ -1789,6 +1848,24 @@ static void lock_zspage(struct zspage *zspage) } migrate_read_unlock(zspage); } +#endif /* defined(CONFIG_ZPOOL) || defined(CONFIG_COMPACTION) */ + +#ifdef CONFIG_ZPOOL +/* + * Unlocks all the pages of the zspage. + * + * pool->lock must be held before this function is called + * to prevent the underlying pages from migrating. + */ +static void unlock_zspage(struct zspage *zspage) +{ + struct page *page = get_first_page(zspage); + + do { + unlock_page(page); + } while ((page = get_next_page(page)) != NULL); +} +#endif /* CONFIG_ZPOOL */ static void migrate_lock_init(struct zspage *zspage) { @@ -1805,6 +1882,7 @@ static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) read_unlock(&zspage->lock); } +#ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage) { write_lock(&zspage->lock); @@ -2369,6 +2447,99 @@ void zs_destroy_pool(struct zs_pool *pool) } EXPORT_SYMBOL_GPL(zs_destroy_pool); +#ifdef CONFIG_ZPOOL +static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) +{ + int i, obj_idx, ret = 0; + unsigned long handle; + struct zspage *zspage; + struct page *page; + enum fullness_group fullness; + + /* Lock LRU and fullness list */ + spin_lock(&pool->lock); + if (list_empty(&pool->lru)) { + spin_unlock(&pool->lock); + return -EINVAL; + } + + for (i = 0; i < retries; i++) { + struct size_class *class; + + zspage = list_last_entry(&pool->lru, struct zspage, lru); + list_del(&zspage->lru); + + /* zs_free may free objects, but not the zspage and handles */ + zspage->under_reclaim = true; + + class = zspage_class(pool, zspage); + fullness = get_fullness_group(class, zspage); + + /* Lock out object allocations and object compaction */ + remove_zspage(class, zspage, fullness); + + spin_unlock(&pool->lock); + + /* Lock backing pages into place */ + lock_zspage(zspage); + + obj_idx = 0; + page = zspage->first_page; + while (1) { + handle = find_alloced_obj(class, page, &obj_idx); + if (!handle) { + page = get_next_page(page); + if (!page) + break; + obj_idx = 0; + continue; + } + + /* + * This will write the object and call zs_free. + * + * zs_free will free the object, but the + * under_reclaim flag prevents it from freeing + * the zspage altogether. This is necessary so + * that we can continue working with the + * zspage potentially after the last object + * has been freed. + */ + ret = pool->zpool_ops->evict(pool->zpool, handle); + if (ret) + goto next; + + obj_idx++; + } + +next: + /* For freeing the zspage, or putting it back in the pool and LRU list. */ + spin_lock(&pool->lock); + zspage->under_reclaim = false; + + if (!get_zspage_inuse(zspage)) { + /* + * Fullness went stale as zs_free() won't touch it + * while the page is removed from the pool. Fix it + * up for the check in __free_zspage(). + */ + zspage->fullness = ZS_EMPTY; + + __free_zspage(pool, class, zspage); + spin_unlock(&pool->lock); + return 0; + } + + putback_zspage(class, zspage); + list_add(&zspage->lru, &pool->lru); + unlock_zspage(zspage); + } + + spin_unlock(&pool->lock); + return -EAGAIN; +} +#endif /* CONFIG_ZPOOL */ + static int __init zs_init(void) { int ret;