From patchwork Thu Feb 16 21:22:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikulas Patocka X-Patchwork-Id: 58239 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp540477wrn; Thu, 16 Feb 2023 13:33:36 -0800 (PST) X-Google-Smtp-Source: AK7set9RNcE8RQje3m+D6qIV9LIwnq7NGpkYqbKF8qLq8S14vURBvHDifcmnAJo8wpv3GT8IRlGR X-Received: by 2002:a17:90b:4c07:b0:234:b75c:1847 with SMTP id na7-20020a17090b4c0700b00234b75c1847mr1907037pjb.30.1676583216530; Thu, 16 Feb 2023 13:33:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676583216; cv=none; d=google.com; s=arc-20160816; b=pVKPk0F+kmyzLWveE1140/WfdJQFUqyeePopxRS2Hwl0coujSLDhZBR6zltIl+2in9 BoVxvvgUkul11e/0wUW9pONg3pZyfd747UUkjKW+SD8M1X9mRxkm6px6ijmZKvzSP8OC 7dwHVdzztJl9H+zOYby6Um8lTk68LRx+aejCEbHMYjT58Ruu8vtFCK9gaj7NIR1SqmwM hTF9//1DuTTzEJ0yCPj/7icj1KGx3hz0bpPbGmQSSsvVK0lTxjMiiccVt9WlabEWaPdq Zq3WixSX0Q85T7NuNLTuaVAVtIEMEom/tm6xNukvptLXpsakMh5k1Yz1AYOMwZqxu6LS QCOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:user-agent:message-id:subject:cc:to :from:date:dkim-signature; bh=Wf4v/tNzjT2VRs1P16c/6f29cG3mDdqrWzEZUinry/Q=; b=VM6/rB7O42x8HyQxUnt+0XjFv10/s1OuUJe85CBMotEaNxTw0YCsmEJxaBKRDl6kw0 TU3Iw2xmZrhVlejFq7vMui7DtnVwGCmwIPwPGh5+6E6OD0p58VQxsKNblJ8iPFS6wYsE JmKJPQ/t3GDtfvrDSkbeVfV6vpJCQtN3keicYYefPonVWKzwIuqF94LoMbAzRfWbL1Pu EPdJ9HQi93kv4pUuSVQMtkj6AVGGui3Yq8GwJcoEIfTGz4QK9na68KEX5+1jvPIuc3yy IBm03Tf+m1iZsGU+1OKim85VnLCKdOiMMhxjCxa8eq95NDa2hRAuweRlbd+F7BbqDqnU sTUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=H3ffhRKb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o17-20020a17090b0c9100b0022c2eb502bcsi2538392pjz.36.2023.02.16.13.33.22; Thu, 16 Feb 2023 13:33:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=H3ffhRKb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229714AbjBPVXj (ORCPT + 99 others); Thu, 16 Feb 2023 16:23:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229448AbjBPVXi (ORCPT ); Thu, 16 Feb 2023 16:23:38 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 058487EE0 for ; Thu, 16 Feb 2023 13:22:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1676582570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=Wf4v/tNzjT2VRs1P16c/6f29cG3mDdqrWzEZUinry/Q=; b=H3ffhRKbEv9NxCVemuMfad3ORalTBN2YvkSV8QVTTxduvIK4axPKvBQS/wkZAbvNBEVMRP Trpf95fdwUnLDzWEOdaK4S9BFpqWn4k45Mt0FqsKgjDKBNRNczucWTxE9hSGCNlZ86oI74 NadOmbcYRPD9cwsw3yR67LvbvVybxeQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-643-zENzyEZKM0KIUmysLL-VZA-1; Thu, 16 Feb 2023 16:22:47 -0500 X-MC-Unique: zENzyEZKM0KIUmysLL-VZA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 952611C0758A; Thu, 16 Feb 2023 21:22:46 +0000 (UTC) Received: from file01.intranet.prod.int.rdu2.redhat.com (file01.intranet.prod.int.rdu2.redhat.com [10.11.5.7]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 86A0440CF8E4; Thu, 16 Feb 2023 21:22:46 +0000 (UTC) Received: from file01.intranet.prod.int.rdu2.redhat.com (localhost [127.0.0.1]) by file01.intranet.prod.int.rdu2.redhat.com (8.14.4/8.14.4) with ESMTP id 31GLMkxg007334; Thu, 16 Feb 2023 16:22:46 -0500 Received: from localhost (mpatocka@localhost) by file01.intranet.prod.int.rdu2.redhat.com (8.14.4/8.14.4/Submit) with ESMTP id 31GLMksq007330; Thu, 16 Feb 2023 16:22:46 -0500 X-Authentication-Warning: file01.intranet.prod.int.rdu2.redhat.com: mpatocka owned process doing -bs Date: Thu, 16 Feb 2023 16:22:46 -0500 (EST) From: Mikulas Patocka X-X-Sender: mpatocka@file01.intranet.prod.int.rdu2.redhat.com To: Matthew Wilcox cc: snitzer@kernel.org, Yang Shi , mgorman@techsingularity.net, agk@redhat.com, dm-devel@redhat.com, akpm@linux-foundation.org, linux-block@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2] dm-crypt: allocate compound pages if possible Message-ID: User-Agent: Alpine 2.21 (LRH 202 2017-01-01) MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758024922773937201?= X-GMAIL-MSGID: =?utf-8?q?1758024922773937201?= It was reported that allocating pages for the write buffer in dm-crypt causes measurable overhead [1]. This patch changes dm-crypt to allocate compound pages if they are available. If not, we fall back to the mempool. [1] https://listman.redhat.com/archives/dm-devel/2023-February/053284.html Signed-off-by: Mikulas Patocka --- drivers/md/dm-crypt.c | 48 +++++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 15 deletions(-) Index: linux-2.6/drivers/md/dm-crypt.c =================================================================== --- linux-2.6.orig/drivers/md/dm-crypt.c 2023-02-16 20:40:15.000000000 +0100 +++ linux-2.6/drivers/md/dm-crypt.c 2023-02-16 21:56:34.000000000 +0100 @@ -1657,6 +1657,9 @@ static void crypt_free_buffer_pages(stru * In order to not degrade performance with excessive locking, we try * non-blocking allocations without a mutex first but on failure we fallback * to blocking allocations with a mutex. + * + * In order to reduce allocation overhead, we try to allocate compound pages in + * the first pass. If they are not available, we fall back to the mempool. */ static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned size) { @@ -1664,8 +1667,8 @@ static struct bio *crypt_alloc_buffer(st struct bio *clone; unsigned int nr_iovecs = (size + PAGE_SIZE - 1) >> PAGE_SHIFT; gfp_t gfp_mask = GFP_NOWAIT | __GFP_HIGHMEM; - unsigned i, len, remaining_size; - struct page *page; + unsigned remaining_size; + unsigned order = MAX_ORDER - 1; retry: if (unlikely(gfp_mask & __GFP_DIRECT_RECLAIM)) @@ -1678,20 +1681,34 @@ retry: remaining_size = size; - for (i = 0; i < nr_iovecs; i++) { - page = mempool_alloc(&cc->page_pool, gfp_mask); - if (!page) { + while (remaining_size) { + struct page *pages; + unsigned size_to_add; + unsigned remaining_order = __fls((remaining_size + PAGE_SIZE - 1) >> PAGE_SHIFT); + order = min(order, remaining_order); + + while (order > 0) { + pages = alloc_pages(gfp_mask + | __GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN | __GFP_COMP, + order); + if (likely(pages != NULL)) + goto have_pages; + order--; + } + + pages = mempool_alloc(&cc->page_pool, gfp_mask); + if (!pages) { crypt_free_buffer_pages(cc, clone); bio_put(clone); gfp_mask |= __GFP_DIRECT_RECLAIM; + order = 0; goto retry; } - len = (remaining_size > PAGE_SIZE) ? PAGE_SIZE : remaining_size; - - bio_add_page(clone, page, len, 0); - - remaining_size -= len; +have_pages: + size_to_add = min((unsigned)PAGE_SIZE << order, remaining_size); + bio_add_page(clone, pages, size_to_add, 0); + remaining_size -= size_to_add; } /* Allocate space for integrity tags */ @@ -1709,12 +1726,13 @@ retry: static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone) { - struct bio_vec *bv; - struct bvec_iter_all iter_all; + struct folio_iter fi; - bio_for_each_segment_all(bv, clone, iter_all) { - BUG_ON(!bv->bv_page); - mempool_free(bv->bv_page, &cc->page_pool); + bio_for_each_folio_all(fi, clone) { + if (folio_test_large(fi.folio)) + folio_put(fi.folio); + else + mempool_free(&fi.folio->page, &cc->page_pool); } }