From patchwork Wed Sep 13 16:56:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139026 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp8311vqb; Wed, 13 Sep 2023 11:37:57 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFo/YFbz8jn6wC2MbEPryIMbcBm2BSYqxBFHHpAjeS4+47o5DW4cNvxpNrAus45vhD3JM1t X-Received: by 2002:a05:6358:41a7:b0:133:291:f9ac with SMTP id w39-20020a05635841a700b001330291f9acmr4977079rwc.25.1694630276739; Wed, 13 Sep 2023 11:37:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694630276; cv=none; d=google.com; s=arc-20160816; b=0RhtCsdlRFQfqYYtLalmuIQTrl29Q6g0w/IgtyrYliGlb+vT9Z2zWYeoD2kysq2ONk olSmfgbIz21KcOkInec3CODRH5tKsdmdDTz7J7YA1N0F3SO8LhOfu8fmsIqVqZIuTEKh jF/Bl3yHiNzHaNdOr6Xmp6o+LQeMcw2VqWElcnbHnplNTox1ZbpOGwQpE0/w2+b4MGck tWL8lNVEuuWqPXnNDiqxduMQhdIUsQ+kjdCR5vojcXYZx1ezG+KmyYWtlC72zyHlPyMM gcXKL71J50qxvZdqm/DKl8YyfPmydgVvbhOy2AfecJlncSCZ51rVdafzi69TkMDr5SSW 5X9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=iCJtvcwek/PzSsuqar23vXotUlaq5CuwRpVtwMRiY9g=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=rEUku8YwRdjWsyAZBd9DnBzNUKqQE67v9+dQ9yD5P3OKpMV4OjbDeyuMii89f5vh/Q TQkkMNs1wSiTJI1tbYuUkyC7qWStcspmFBKRRnhpt9ebPRMKyO/7cb2o/yuq2fuOMevo vVJPPzDadRl9Orr728r6vDeGsosMiwUNupSwTlvKPlPYqYsjqfYY+aUqrXb0UM65GYsv U7n/XieN2mRBzibGChay1RnUIw3UPdXGGdMb2G2NZODA/W1MJLLetDc/46iUUIA6BQeb z5k9A9K8W8kPQ+DHPwt9hOcNI6T1U9pHr9IKS9uHNnPq6y1m1Qpw/+6GOg1KICy/fgV7 IHEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jSBrjgKR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id x26-20020a63171a000000b00577960a815csi4126086pgl.188.2023.09.13.11.37.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 11:37:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jSBrjgKR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id E9D4D823D9D7; Wed, 13 Sep 2023 09:58:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230300AbjIMQ57 (ORCPT + 35 others); Wed, 13 Sep 2023 12:57:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229924AbjIMQ5s (ORCPT ); Wed, 13 Sep 2023 12:57:48 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 91A7B19B6 for ; Wed, 13 Sep 2023 09:57:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624220; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iCJtvcwek/PzSsuqar23vXotUlaq5CuwRpVtwMRiY9g=; b=jSBrjgKRn2qOFrcCTR+yWnEdDyQsyM1UHj7iOoHbrIdWzsWkWrWk80/HFdaltWwBlbWPti yl63PeBZ82iX3JO6ZHoRMhCA7O67knTmBk3XsOR+W34JjaP8i+EN4Lo8FPcTIoazYYEOzT iPBRlXENqhEOilSt1CBOzGJpkyQyHRg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-674-3UG6PQn5O3qmluWTH-Ob_w-1; Wed, 13 Sep 2023 12:56:56 -0400 X-MC-Unique: 3UG6PQn5O3qmluWTH-Ob_w-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EA3DC185A79B; Wed, 13 Sep 2023 16:56:55 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D05340C6EA8; Wed, 13 Sep 2023 16:56:54 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 01/13] iov_iter: Add a benchmarking kunit test Date: Wed, 13 Sep 2023 17:56:36 +0100 Message-ID: <20230913165648.2570623-2-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:58:38 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776948637131370679 X-GMAIL-MSGID: 1776948637131370679 --- lib/kunit_iov_iter.c | 181 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) diff --git a/lib/kunit_iov_iter.c b/lib/kunit_iov_iter.c index 859b67c4d697..478fea956f58 100644 --- a/lib/kunit_iov_iter.c +++ b/lib/kunit_iov_iter.c @@ -756,6 +756,184 @@ static void __init iov_kunit_extract_pages_xarray(struct kunit *test) KUNIT_SUCCEED(); } +static void iov_kunit_free_page(void *data) +{ + __free_page(data); +} + +static void __init iov_kunit_benchmark_print_stats(struct kunit *test, + unsigned int *samples) +{ + unsigned long total = 0; + int i; + + for (i = 0; i < 16; i++) { + total += samples[i]; + kunit_info(test, "run %x: %u uS\n", i, samples[i]); + } + + kunit_info(test, "avg %lu uS\n", total / 16); +} + +/* + * Time copying 256MiB through an ITER_BVEC. + */ +static void __init iov_kunit_benchmark_bvec(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page, **pages; + unsigned int samples[16]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (i = 0; i < npages; i++) + bvec_set_page(&bvec[i], page, PAGE_SIZE, 0); + + /* Create a single large buffer to copy to/from. */ + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) + pages[i] = page; + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < 16; i++) { + iov_iter_bvec(&iter, ITER_DEST, bvec, npages, size); + a = ktime_get_real(); + copied = copy_to_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_BVEC in 256 page chunks. + */ +static void __init iov_kunit_benchmark_bvec_split(struct kunit *test) +{ + struct iov_iter iter; + struct bio_vec *bvec; + struct page *page, **pages; + unsigned int samples[16]; + ktime_t a, b; + ssize_t copied; + size_t size, npages = 64; + void *scratch; + int i, j; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + /* Create a single large buffer to copy to/from. */ + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) + pages[i] = page; + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over BVEC:\n"); + for (i = 0; i < 16; i++) { + size = 256 * 1024 * 1024; + a = ktime_get_real(); + do { + size_t part = min(size, npages * PAGE_SIZE); + + bvec = kunit_kmalloc_array(test, npages, sizeof(bvec[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, bvec); + for (j = 0; j < npages; j++) + bvec_set_page(&bvec[j], page, PAGE_SIZE, 0); + + iov_iter_bvec(&iter, ITER_DEST, bvec, npages, part); + copied = copy_to_iter(scratch, part, &iter); + KUNIT_EXPECT_EQ(test, copied, part); + size -= part; + } while (size > 0); + b = ktime_get_real(); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + +/* + * Time copying 256MiB through an ITER_XARRAY. + */ +static void __init iov_kunit_benchmark_xarray(struct kunit *test) +{ + struct iov_iter iter; + struct xarray *xarray; + struct page *page, **pages; + unsigned int samples[16]; + ktime_t a, b; + ssize_t copied; + size_t size = 256 * 1024 * 1024, npages = size / PAGE_SIZE; + void *scratch; + int i; + + /* Allocate a page and tile it repeatedly in the buffer. */ + page = alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, page); + kunit_add_action_or_reset(test, iov_kunit_free_page, page); + + xarray = iov_kunit_create_xarray(test); + + for (i = 0; i < npages; i++) { + void *x = xa_store(xarray, i, page, GFP_KERNEL); + + KUNIT_ASSERT_FALSE(test, xa_is_err(x)); + } + + /* Create a single large buffer to copy to/from. */ + pages = kunit_kmalloc_array(test, npages, sizeof(pages[0]), GFP_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, pages); + for (i = 0; i < npages; i++) + pages[i] = page; + + scratch = vmap(pages, npages, VM_MAP | VM_MAP_PUT_PAGES, PAGE_KERNEL); + KUNIT_ASSERT_NOT_NULL(test, scratch); + kunit_add_action_or_reset(test, iov_kunit_unmap, scratch); + + /* Perform and time a bunch of copies. */ + kunit_info(test, "Benchmarking copy_to_iter() over XARRAY:\n"); + for (i = 0; i < 16; i++) { + iov_iter_xarray(&iter, ITER_DEST, xarray, 0, size); + a = ktime_get_real(); + copied = copy_to_iter(scratch, size, &iter); + b = ktime_get_real(); + KUNIT_EXPECT_EQ(test, copied, size); + samples[i] = ktime_to_us(ktime_sub(b, a)); + } + + iov_kunit_benchmark_print_stats(test, samples); + KUNIT_SUCCEED(); +} + static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_copy_to_kvec), KUNIT_CASE(iov_kunit_copy_from_kvec), @@ -766,6 +944,9 @@ static struct kunit_case __refdata iov_kunit_cases[] = { KUNIT_CASE(iov_kunit_extract_pages_kvec), KUNIT_CASE(iov_kunit_extract_pages_bvec), KUNIT_CASE(iov_kunit_extract_pages_xarray), + KUNIT_CASE(iov_kunit_benchmark_bvec), + KUNIT_CASE(iov_kunit_benchmark_bvec_split), + KUNIT_CASE(iov_kunit_benchmark_xarray), {} }; From patchwork Wed Sep 13 16:56:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139100 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp94986vqb; Wed, 13 Sep 2023 14:24:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE8gR1O0OQ7ont0H0PUcn4ajRq6iRduebJCOYHH5eV+lOrdgTiGJfUmjYVVIANRqzzSqVRx X-Received: by 2002:a17:90b:1902:b0:269:4fe8:687 with SMTP id mp2-20020a17090b190200b002694fe80687mr5772116pjb.19.1694640288729; Wed, 13 Sep 2023 14:24:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694640288; cv=none; d=google.com; s=arc-20160816; b=gADqtfwI4geUpqMM1bhdrCBnoWi8yUW9WtWOxl0d7aqR+kgxvy9czfxuM5hbF2TEIk rUkmidUanyujdCOWlj6w/LU9Dfax3vUFJe+OQ3nYu3juVzW+CiDiX4qIi3jzTqAg9D+a SHUakJ4p+WSIT3n/uZGdwLxBitSnuzMA2VCrZbCL8ARne8PyJNe3QOlGOPVKW5sHDRhc uuCcFtLd8fUjpFtGkJqLc4VGZe8iU+1j7oPiqbM5ZZNvOU13+VcVahLHhJZSpyzwgIya vBjECVrq9wHlJpzzbnd9d2MGKqp+8LyoRdQjOQ07JgCPDduGPyWYi77/tqG9rCah+uwv D3TA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=437kiFAHLHJfEBaWwZrq8WSevs6s9dyVAFCFY8Y8ee4=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=REyDgFJGJlbxkfmUmzwbJh6VEMAO5pL4Oh7imcklIrS9N+Mh+eMNRkOSENku4wJgcO 6AZNfhJkVGVdseRvV1v11BpwcPoa/6wIxvN7FtjyJK6JjYY2X+z/w2Df0Me/VbT537n5 u+GZIVBNFIG2G/i6JkR7L+0a4sd0qjUr0/kfma6vQCP6wgrcRje8SyXZSOBjkzPjUC29 4hV6zNqNeORiwqNwO3hD0iqa9pO/YpEuSf+ymaD97lcayx+Sfe9BCn1a0FZl4D2Zoc33 z55LwiDZGfwE/5e19xk1b3nk4D2mQ5PklEypMu4V4Sx+YtVC/S8pX+JKQmT1oo7kTvPZ 1f2w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fxBUWlEF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id mn20-20020a17090b189400b002631e44f48csi130603pjb.179.2023.09.13.14.24.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 14:24:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=fxBUWlEF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 52F1B8483140; Wed, 13 Sep 2023 09:58:06 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229923AbjIMQ5s (ORCPT + 35 others); Wed, 13 Sep 2023 12:57:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229846AbjIMQ5p (ORCPT ); Wed, 13 Sep 2023 12:57:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 18086B7 for ; Wed, 13 Sep 2023 09:57:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624221; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=437kiFAHLHJfEBaWwZrq8WSevs6s9dyVAFCFY8Y8ee4=; b=fxBUWlEFWTBs1hZteao+3nCNpUdjZ3nHn0T7gQEPPRLoYkn8M/liah9N6qpYwtraeel9Hu b8O+vhQyqtvmOy6dQmyLWuv4jW4xQLoQLnlg6POLqmpQ/9N0DHz89PwZ1pvzyKhcRXNJj3 E8eLbnc3HpUfcnbiLLg0IVG0lLmIcVk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-313-ofpFxPGzPwO-sSg4BhMp7A-1; Wed, 13 Sep 2023 12:56:59 -0400 X-MC-Unique: ofpFxPGzPwO-sSg4BhMp7A-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5F9CA8032FE; Wed, 13 Sep 2023 16:56:58 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id A7EE42026D4B; Wed, 13 Sep 2023 16:56:56 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 02/13] iov_iter: Renumber ITER_* constants Date: Wed, 13 Sep 2023 17:56:37 +0100 Message-ID: <20230913165648.2570623-3-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:58:13 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776959135249824908 X-GMAIL-MSGID: 1776959135249824908 Renumber the ITER_* iterator-type constants to put things in the same order as in the iteration functions and to group user-backed iterators at the bottom. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/uio.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 42bce38a8e87..d1801c46e89e 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -21,12 +21,12 @@ struct kvec { enum iter_type { /* iter types */ + ITER_UBUF, ITER_IOVEC, - ITER_KVEC, ITER_BVEC, + ITER_KVEC, ITER_XARRAY, ITER_DISCARD, - ITER_UBUF, }; #define ITER_SOURCE 1 // == WRITE From patchwork Wed Sep 13 16:56:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139329 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp129286vqi; Wed, 13 Sep 2023 22:39:28 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHv7ZVOpfo8crjkES9vivTw/i106oTMeSP0QP1ZrsNUBm6aLmL3J4qYBhjtHX1tVyfbKSSr X-Received: by 2002:a05:6a20:5481:b0:13d:8876:4c97 with SMTP id i1-20020a056a20548100b0013d88764c97mr5442847pzk.16.1694669968295; Wed, 13 Sep 2023 22:39:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694669968; cv=none; d=google.com; s=arc-20160816; b=aA8JgCdaUCvZMWUf6i6HouAU7gwn1deN9HalvqcdrA2rYDTiY/84mH4BCU6If2QxNB 7Jg50ySaA8cnwfHFlezSMY5fYFXTAvJ+E1aDAw4YrGVRvDc4aUs+Aq9gFVhQPVq7hwVg IlkhE/qf+6/aEbqgxk/Ha6bt6m94FfyuPiY/vr+Bk0Ese0m3mw+kt5JRcYwjFle/5kkk 5BsNiQ228ylv1iY8iigJzMgFLMhCx7c5RVZ9Cji/v3NjNrPwQPz+icWDmZcrbfXQm7+o aOYuRQwPy2UbT1dZXNZJjyuMUfhBEKqpjpxGvz7BJo54+yGxfNK0SyKevOIRA4FHrpi9 I4Cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=z7dUGMUCdPzyoXpN55evy5eiv8axvEjBjfUl9Q9dpbg=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=DxqKjCRLGCsJy/TW+wb8tMsc1vYbDJygpAW1/ogWp5B6RkxOlIHM/nsfQF7447KcQ4 rbo2Zd5hP9RD0xIJN8rmh2ErzIahm1aPcmjJ963nt2K6I9JmLlqCgYxSLb0Y/6c7aZoO X7IZ0NGr3CNmrCwXA+V5dr1qDznseurgIVhSq1BTqxGiOfEZde+2YCplx6wy0jNnCz5X 0IT3TfGBPS58Q4Mky+t63viMITWnE3J0GSsq7HqlOHKfj0UFGIDWNJQJgTQAxoC+L5/v /k6q4b3lpa647Gx++2T0XT4ardYXRvakRSfb+LcdQ/SmIKRT8SISFOBlENjsZ2zGKsmr QD+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OZGolSBv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id oc13-20020a17090b1c0d00b0026b49dac834si1078142pjb.34.2023.09.13.22.39.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 22:39:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=OZGolSBv; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 1126B822E552; Wed, 13 Sep 2023 09:58:26 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230146AbjIMQ6B (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbjIMQ5x (ORCPT ); Wed, 13 Sep 2023 12:57:53 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 995E31BD0 for ; Wed, 13 Sep 2023 09:57:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z7dUGMUCdPzyoXpN55evy5eiv8axvEjBjfUl9Q9dpbg=; b=OZGolSBvzk3B/YVM/aDJSgTCXUA8wJzeN5NCyALftfAaK1dkgjfO2Jae3hfwOkL1k79HsT XcOZ0GiNFP7HcM2ZHRtGDJASFlymEfAeFgVwOwiivOctz6m7umJVEhU7ETRKhv7UMW+D9Z nSVBsx6O5x5Sv0bNpI92g10vV+k4pnY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-354-B53T9xcDN2imp2fxxNS_HQ-1; Wed, 13 Sep 2023 12:57:01 -0400 X-MC-Unique: B53T9xcDN2imp2fxxNS_HQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A0E9C185A79C; Wed, 13 Sep 2023 16:57:00 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 07E2263F9C; Wed, 13 Sep 2023 16:56:58 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 03/13] iov_iter: Derive user-backedness from the iterator type Date: Wed, 13 Sep 2023 17:56:38 +0100 Message-ID: <20230913165648.2570623-4-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:58:26 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776990256627555498 X-GMAIL-MSGID: 1776990256627555498 Use the iterator type to determine whether an iterator is user-backed or not rather than using a special flag for it. Now that ITER_UBUF and ITER_IOVEC are 0 and 1, they can be checked with a single comparison. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/uio.h | 4 +--- lib/iov_iter.c | 1 - 2 files changed, 1 insertion(+), 4 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index d1801c46e89e..e2a248dad80b 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -43,7 +43,6 @@ struct iov_iter { bool copy_mc; bool nofault; bool data_source; - bool user_backed; union { size_t iov_offset; int last_offset; @@ -143,7 +142,7 @@ static inline unsigned char iov_iter_rw(const struct iov_iter *i) static inline bool user_backed_iter(const struct iov_iter *i) { - return i->user_backed; + return iter_is_ubuf(i) || iter_is_iovec(i); } /* @@ -383,7 +382,6 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, *i = (struct iov_iter) { .iter_type = ITER_UBUF, .copy_mc = false, - .user_backed = true, .data_source = direction, .ubuf = buf, .count = count, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 27234a820eeb..227c9f536b94 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -290,7 +290,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, .iter_type = ITER_IOVEC, .copy_mc = false, .nofault = false, - .user_backed = true, .data_source = direction, .__iov = iov, .nr_segs = nr_segs, From patchwork Wed Sep 13 16:56:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp295462vqi; Thu, 14 Sep 2023 05:04:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHEzwUNCi4gXhESGV1o5mXGak6qnLAsv3GFue6Lh9YMBJZ8YyunN02cjXfnzhQD5bkhYQzB X-Received: by 2002:a05:6a20:3c92:b0:14e:3daf:bef9 with SMTP id b18-20020a056a203c9200b0014e3dafbef9mr5800768pzj.16.1694693083052; Thu, 14 Sep 2023 05:04:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694693083; cv=none; d=google.com; s=arc-20160816; b=T2zs+fpSdWKBpzw6hXd/cPvbaBj92A8ul08Ep7SZGQAce7EO0nRf5XcFmUCdxvazIn d2SARR5NBVh1tN3qvf0jlnHwiUjye/LjnBf/I7izPlSZvcZ00nDbYXRCDDBoIM73uMEF ZWjNrqhukteICSdp1B9D7TFV2WXCK0ZxNpN+cP6mRT1nlQigKsfObn931G8sDnWRyMUT d8KoYi35NAefETEI7Ls4GefR4WG8nZzB39KiuNO45wtCA0znuu8A8ObeTqaMQT8qy82U k0j8rHMqMXGWRzqv9o1tl1+rmnuBUTAWbKA+8SeUWgYAPDEDXtSepb1LGDslVetRp6pb W2qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=D4sDntfB05lYx2oXLtvEA75x1n6eFZDOWAb9msr/5sA=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=saZ7mU1Se49VIDbHXA0MYlc+4GuNQwjStG34POzFwFvpKJOduVwzcRInoyyCdZfcgX 3GSQj/uRVgIfmAh+1BrU7KyA0ssjVfWrp0d+3exjVwm70q7hSnAnAMUk7gmtUqp8EN/3 ysKKW8QpoDPBIdC93w4Y/NM/dF00KQRsv2+bwkakI1HozTw3/GWXJ0UT7t786vqqay5z 1bzSNUqciV6FpMt/6flEbEZ9BZ2xd0OXc06dMFDgqxIMT0rREUlxmzEmHHpZUUIFlhxL Nu1Kk15a/pTMO+Gzq+kMZMAo/nNFrBQ2pNJEqepM18VbRT9udkdu1ctgXRcLNdsF2f2n yCQg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eT82Z5w6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id q6-20020a170902a3c600b001bf2dc67da3si1408734plb.209.2023.09.14.05.04.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Sep 2023 05:04:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=eT82Z5w6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id D4F958339658; Wed, 13 Sep 2023 09:58:22 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230421AbjIMQ6F (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230204AbjIMQ56 (ORCPT ); Wed, 13 Sep 2023 12:57:58 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 6D01E1BE3 for ; Wed, 13 Sep 2023 09:57:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624227; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D4sDntfB05lYx2oXLtvEA75x1n6eFZDOWAb9msr/5sA=; b=eT82Z5w6s0hvbRezjwcsEEACUMmUyuNFXtxvMlfA3O+89eIcKl7+iBmSd7O9FC8UngBFDU 8846+3+QZvgnMSw+PTN/tvX5IYnzF2VOh23RaD8SjTVfpB+H0SGDS9DDDWK7Bxp/U7BuzO rcAlNuP9al6FGvzBrPsZIhyF5YTw4XE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-611-ULwguQmgMDW4TifQeFpDzQ-1; Wed, 13 Sep 2023 12:57:04 -0400 X-MC-Unique: ULwguQmgMDW4TifQeFpDzQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1FD8A101B456; Wed, 13 Sep 2023 16:57:03 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 47CA92026D4B; Wed, 13 Sep 2023 16:57:01 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 04/13] iov_iter: Convert iterate*() to inline funcs Date: Wed, 13 Sep 2023 17:56:39 +0100 Message-ID: <20230913165648.2570623-5-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:58:23 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777014494097725524 X-GMAIL-MSGID: 1777014494097725524 Convert the iov_iter iteration macros to inline functions to make the code easier to follow. The functions are marked __always_inline as we don't want to end up with indirect calls in the code. This, however, leaves dealing with ->copy_mc in an awkard situation since the step function (memcpy_from_iter_mc()) needs to test the flag in the iterator, but isn't passed the iterator. This will be dealt with in a follow-up patch. The variable names in the per-type iterator functions have been harmonised as much as possible and made clearer as to the variable purpose. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org Link: https://lore.kernel.org/r/3710261.1691764329@warthog.procyon.org.uk/ # v1 Link: https://lore.kernel.org/r/855.1692047347@warthog.procyon.org.uk/ # v2 Link: https://lore.kernel.org/r/20230816120741.534415-1-dhowells@redhat.com/ # v3 --- Notes: Changes ======= ver #4) - Fix iterate_bvec() and iterate_kvec() to update iter->bvec and iter->kvec after subtracting it to calculate iter->nr_segs. - Change iterate_xarray() to use start+progress rather than increasing start to reduce code size. ver #3) - Use min_t(size_t,) not min() to avoid a warning on Hexagon. - Inline all the step functions. ver #2) - Rebased on top of Willy's changes in linux-next. - Change the checksum argument to the iteration functions to be a general void* and use it to pass iter->copy_mc flag to memcpy_from_iter_mc() to avoid using a function pointer. - Arrange the end of the iterate_*() functions to look the same to give the optimiser the best chance. - Make iterate_and_advance() a wrapper around iterate_and_advance2(). - Adjust iterate_and_advance2() to use if-else-if-else-if-else rather than switch(), to put ITER_BVEC before KVEC and to mark UBUF and IOVEC as likely(). - Move "iter->count += progress" into iterate_and_advance2() from the iterate functions. - Mark a number of the iterator helpers with __always_inline. - Fix _copy_from_iter_flushcache() to use memcpy_from_iter_flushcache() not memcpy_from_iter(). lib/iov_iter.c | 611 ++++++++++++++++++++++++++++++------------------- 1 file changed, 370 insertions(+), 241 deletions(-) diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 227c9f536b94..b3ce6fa5f7a5 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -14,188 +14,263 @@ #include #include -/* covers ubuf and kbuf alike */ -#define iterate_buf(i, n, base, len, off, __p, STEP) { \ - size_t __maybe_unused off = 0; \ - len = n; \ - base = __p + i->iov_offset; \ - len -= (STEP); \ - i->iov_offset += len; \ - n = len; \ -} - -/* covers iovec and kvec alike */ -#define iterate_iovec(i, n, base, len, off, __p, STEP) { \ - size_t off = 0; \ - size_t skip = i->iov_offset; \ - do { \ - len = min(n, __p->iov_len - skip); \ - if (likely(len)) { \ - base = __p->iov_base + skip; \ - len -= (STEP); \ - off += len; \ - skip += len; \ - n -= len; \ - if (skip < __p->iov_len) \ - break; \ - } \ - __p++; \ - skip = 0; \ - } while (n); \ - i->iov_offset = skip; \ - n = off; \ -} - -#define iterate_bvec(i, n, base, len, off, p, STEP) { \ - size_t off = 0; \ - unsigned skip = i->iov_offset; \ - while (n) { \ - unsigned offset = p->bv_offset + skip; \ - unsigned left; \ - void *kaddr = kmap_local_page(p->bv_page + \ - offset / PAGE_SIZE); \ - base = kaddr + offset % PAGE_SIZE; \ - len = min(min(n, (size_t)(p->bv_len - skip)), \ - (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); \ - left = (STEP); \ - kunmap_local(kaddr); \ - len -= left; \ - off += len; \ - skip += len; \ - if (skip == p->bv_len) { \ - skip = 0; \ - p++; \ - } \ - n -= len; \ - if (left) \ - break; \ - } \ - i->iov_offset = skip; \ - n = off; \ -} - -#define iterate_xarray(i, n, base, len, __off, STEP) { \ - __label__ __out; \ - size_t __off = 0; \ - struct folio *folio; \ - loff_t start = i->xarray_start + i->iov_offset; \ - pgoff_t index = start / PAGE_SIZE; \ - XA_STATE(xas, i->xarray, index); \ - \ - len = PAGE_SIZE - offset_in_page(start); \ - rcu_read_lock(); \ - xas_for_each(&xas, folio, ULONG_MAX) { \ - unsigned left; \ - size_t offset; \ - if (xas_retry(&xas, folio)) \ - continue; \ - if (WARN_ON(xa_is_value(folio))) \ - break; \ - if (WARN_ON(folio_test_hugetlb(folio))) \ - break; \ - offset = offset_in_folio(folio, start + __off); \ - while (offset < folio_size(folio)) { \ - base = kmap_local_folio(folio, offset); \ - len = min(n, len); \ - left = (STEP); \ - kunmap_local(base); \ - len -= left; \ - __off += len; \ - n -= len; \ - if (left || n == 0) \ - goto __out; \ - offset += len; \ - len = PAGE_SIZE; \ - } \ - } \ -__out: \ - rcu_read_unlock(); \ - i->iov_offset += __off; \ - n = __off; \ -} - -#define __iterate_and_advance(i, n, base, len, off, I, K) { \ - if (unlikely(i->count < n)) \ - n = i->count; \ - if (likely(n)) { \ - if (likely(iter_is_ubuf(i))) { \ - void __user *base; \ - size_t len; \ - iterate_buf(i, n, base, len, off, \ - i->ubuf, (I)) \ - } else if (likely(iter_is_iovec(i))) { \ - const struct iovec *iov = iter_iov(i); \ - void __user *base; \ - size_t len; \ - iterate_iovec(i, n, base, len, off, \ - iov, (I)) \ - i->nr_segs -= iov - iter_iov(i); \ - i->__iov = iov; \ - } else if (iov_iter_is_bvec(i)) { \ - const struct bio_vec *bvec = i->bvec; \ - void *base; \ - size_t len; \ - iterate_bvec(i, n, base, len, off, \ - bvec, (K)) \ - i->nr_segs -= bvec - i->bvec; \ - i->bvec = bvec; \ - } else if (iov_iter_is_kvec(i)) { \ - const struct kvec *kvec = i->kvec; \ - void *base; \ - size_t len; \ - iterate_iovec(i, n, base, len, off, \ - kvec, (K)) \ - i->nr_segs -= kvec - i->kvec; \ - i->kvec = kvec; \ - } else if (iov_iter_is_xarray(i)) { \ - void *base; \ - size_t len; \ - iterate_xarray(i, n, base, len, off, \ - (K)) \ - } \ - i->count -= n; \ - } \ -} -#define iterate_and_advance(i, n, base, len, off, I, K) \ - __iterate_and_advance(i, n, base, len, off, I, ((void)(K),0)) - -static int copyout(void __user *to, const void *from, size_t n) +typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, + void *priv, void *priv2); +typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, size_t len, + void *priv, void *priv2); + +static __always_inline +size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) { - if (should_fail_usercopy()) - return n; - if (access_ok(to, n)) { - instrument_copy_to_user(to, from, n); - n = raw_copy_to_user(to, from, n); + void __user *base = iter->ubuf; + size_t progress = 0, remain; + + remain = step(base + iter->iov_offset, 0, len, priv, priv2); + progress = len - remain; + iter->iov_offset += progress; + return progress; +} + +static __always_inline +size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) +{ + const struct iovec *p = iter->__iov; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->__iov = p; + iter->nr_segs -= p - iter->__iov; + iter->iov_offset = skip; + return progress; +} + +static __always_inline +size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct kvec *p = iter->kvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->nr_segs -= p - iter->kvec; + iter->kvec = p; + iter->iov_offset = skip; + return progress; +} + +static __always_inline +size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct bio_vec *p = iter->bvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t offset = p->bv_offset + skip, part; + void *kaddr = kmap_local_page(p->bv_page + offset / PAGE_SIZE); + + part = min3(len, + (size_t)(p->bv_len - skip), + (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); + remain = step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); + kunmap_local(kaddr); + consumed = part - remain; + len -= consumed; + progress += consumed; + skip += consumed; + if (skip >= p->bv_len) { + skip = 0; + p++; + } + if (remain) + break; + } while (len); + + iter->nr_segs -= p - iter->bvec; + iter->bvec = p; + iter->iov_offset = skip; + return progress; +} + +static __always_inline +size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + struct folio *folio; + size_t progress = 0; + loff_t start = iter->xarray_start + iter->iov_offset; + pgoff_t index = start / PAGE_SIZE; + XA_STATE(xas, iter->xarray, index); + + rcu_read_lock(); + xas_for_each(&xas, folio, ULONG_MAX) { + size_t remain, consumed, offset, part, flen; + + if (xas_retry(&xas, folio)) + continue; + if (WARN_ON(xa_is_value(folio))) + break; + if (WARN_ON(folio_test_hugetlb(folio))) + break; + + offset = offset_in_folio(folio, start + progress); + flen = min(folio_size(folio) - offset, len); + + while (flen) { + void *base = kmap_local_folio(folio, offset); + + part = min_t(size_t, flen, + PAGE_SIZE - offset_in_page(offset)); + remain = step(base, progress, part, priv, priv2); + kunmap_local(base); + + consumed = part - remain; + progress += consumed; + len -= consumed; + + if (remain || len == 0) + goto out; + flen -= consumed; + offset += consumed; + } } - return n; + +out: + rcu_read_unlock(); + iter->iov_offset += progress; + return progress; } -static int copyout_nofault(void __user *to, const void *from, size_t n) +static __always_inline +size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_ustep_f ustep, iov_step_f step) { - long res; + size_t progress; + if (unlikely(iter->count < len)) + len = iter->count; + if (unlikely(!len)) + return 0; + + if (likely(iter_is_ubuf(iter))) + progress = iterate_ubuf(iter, len, priv, priv2, ustep); + else if (likely(iter_is_iovec(iter))) + progress = iterate_iovec(iter, len, priv, priv2, ustep); + else if (iov_iter_is_bvec(iter)) + progress = iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress = iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress = iterate_xarray(iter, len, priv, priv2, step); + else + progress = len; + iter->count -= progress; + return progress; +} + +static __always_inline +size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, + iov_ustep_f ustep, iov_step_f step) +{ + return iterate_and_advance2(iter, len, priv, NULL, ustep, step); +} + +static __always_inline +size_t copy_to_user_iter(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ if (should_fail_usercopy()) - return n; + return len; + if (access_ok(iter_to, len)) { + from += progress; + instrument_copy_to_user(iter_to, from, len); + len = raw_copy_to_user(iter_to, from, len); + } + return len; +} - res = copy_to_user_nofault(to, from, n); +static __always_inline +size_t copy_to_user_iter_nofault(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + ssize_t res; - return res < 0 ? n : res; + if (should_fail_usercopy()) + return len; + + from += progress; + res = copy_to_user_nofault(iter_to, from, len); + return res < 0 ? len : res; } -static int copyin(void *to, const void __user *from, size_t n) +static __always_inline +size_t copy_from_user_iter(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - size_t res = n; + size_t res = len; if (should_fail_usercopy()) - return n; - if (access_ok(from, n)) { - instrument_copy_from_user_before(to, from, n); - res = raw_copy_from_user(to, from, n); - instrument_copy_from_user_after(to, from, n, res); + return len; + if (access_ok(iter_from, len)) { + to += progress; + instrument_copy_from_user_before(to, iter_from, len); + res = raw_copy_from_user(to, iter_from, len); + instrument_copy_from_user_after(to, iter_from, len, res); } return res; } +static __always_inline +size_t memcpy_to_iter(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + memcpy(iter_to, from + progress, len); + return 0; +} + +static __always_inline +size_t memcpy_from_iter(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + memcpy(to + progress, iter_from, len); + return 0; +} + /* * fault_in_iov_iter_readable - fault in iov iterator for reading * @i: iterator @@ -312,23 +387,29 @@ size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return 0; if (user_backed_iter(i)) might_fault(); - iterate_and_advance(i, bytes, base, len, off, - copyout(base, addr + off, len), - memcpy(base, addr + off, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, (void *)addr, + copy_to_user_iter, memcpy_to_iter); } EXPORT_SYMBOL(_copy_to_iter); #ifdef CONFIG_ARCH_HAS_COPY_MC -static int copyout_mc(void __user *to, const void *from, size_t n) -{ - if (access_ok(to, n)) { - instrument_copy_to_user(to, from, n); - n = copy_mc_to_user((__force void *) to, from, n); +static __always_inline +size_t copy_to_user_iter_mc(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + if (access_ok(iter_to, len)) { + from += progress; + instrument_copy_to_user(iter_to, from, len); + len = copy_mc_to_user(iter_to, from, len); } - return n; + return len; +} + +static __always_inline +size_t memcpy_to_iter_mc(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + return copy_mc_to_kernel(iter_to, from + progress, len); } /** @@ -361,22 +442,20 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) return 0; if (user_backed_iter(i)) might_fault(); - __iterate_and_advance(i, bytes, base, len, off, - copyout_mc(base, addr + off, len), - copy_mc_to_kernel(base, addr + off, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, (void *)addr, + copy_to_user_iter_mc, memcpy_to_iter_mc); } EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ -static void *memcpy_from_iter(struct iov_iter *i, void *to, const void *from, - size_t size) +static size_t memcpy_from_iter_mc(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - if (iov_iter_is_copy_mc(i)) - return (void *)copy_mc_to_kernel(to, from, size); - return memcpy(to, from, size); + struct iov_iter *iter = priv2; + + if (iov_iter_is_copy_mc(iter)) + return copy_mc_to_kernel(to + progress, iter_from, len); + return memcpy_from_iter(iter_from, progress, len, to, priv2); } size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) @@ -386,30 +465,46 @@ size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) if (user_backed_iter(i)) might_fault(); - iterate_and_advance(i, bytes, base, len, off, - copyin(addr + off, base, len), - memcpy_from_iter(i, addr + off, base, len) - ) - - return bytes; + return iterate_and_advance2(i, bytes, addr, i, + copy_from_user_iter, + memcpy_from_iter_mc); } EXPORT_SYMBOL(_copy_from_iter); +static __always_inline +size_t copy_from_user_iter_nocache(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + return __copy_from_user_inatomic_nocache(to + progress, iter_from, len); +} + size_t _copy_from_iter_nocache(void *addr, size_t bytes, struct iov_iter *i) { if (WARN_ON_ONCE(!i->data_source)) return 0; - iterate_and_advance(i, bytes, base, len, off, - __copy_from_user_inatomic_nocache(addr + off, base, len), - memcpy(addr + off, base, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter_nocache, + memcpy_from_iter); } EXPORT_SYMBOL(_copy_from_iter_nocache); #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE +static __always_inline +size_t copy_from_user_iter_flushcache(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + return __copy_from_user_flushcache(to + progress, iter_from, len); +} + +static __always_inline +size_t memcpy_from_iter_flushcache(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + memcpy_flushcache(to + progress, iter_from, len); + return 0; +} + /** * _copy_from_iter_flushcache - write destination through cpu cache * @addr: destination kernel address @@ -431,12 +526,9 @@ size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i) if (WARN_ON_ONCE(!i->data_source)) return 0; - iterate_and_advance(i, bytes, base, len, off, - __copy_from_user_flushcache(addr + off, base, len), - memcpy_flushcache(addr + off, base, len) - ) - - return bytes; + return iterate_and_advance(i, bytes, addr, + copy_from_user_iter_flushcache, + memcpy_from_iter_flushcache); } EXPORT_SYMBOL_GPL(_copy_from_iter_flushcache); #endif @@ -508,10 +600,9 @@ size_t copy_page_to_iter_nofault(struct page *page, unsigned offset, size_t byte void *kaddr = kmap_local_page(page); size_t n = min(bytes, (size_t)PAGE_SIZE - offset); - iterate_and_advance(i, n, base, len, off, - copyout_nofault(base, kaddr + offset + off, len), - memcpy(base, kaddr + offset + off, len) - ) + n = iterate_and_advance(i, bytes, kaddr, + copy_to_user_iter_nofault, + memcpy_to_iter); kunmap_local(kaddr); res += n; bytes -= n; @@ -554,14 +645,25 @@ size_t copy_page_from_iter(struct page *page, size_t offset, size_t bytes, } EXPORT_SYMBOL(copy_page_from_iter); -size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +static __always_inline +size_t zero_to_user_iter(void __user *iter_to, size_t progress, + size_t len, void *priv, void *priv2) { - iterate_and_advance(i, bytes, base, len, count, - clear_user(base, len), - memset(base, 0, len) - ) + return clear_user(iter_to, len); +} - return bytes; +static __always_inline +size_t zero_to_iter(void *iter_to, size_t progress, + size_t len, void *priv, void *priv2) +{ + memset(iter_to, 0, len); + return 0; +} + +size_t iov_iter_zero(size_t bytes, struct iov_iter *i) +{ + return iterate_and_advance(i, bytes, NULL, + zero_to_user_iter, zero_to_iter); } EXPORT_SYMBOL(iov_iter_zero); @@ -586,10 +688,9 @@ size_t copy_page_from_iter_atomic(struct page *page, size_t offset, } p = kmap_atomic(page) + offset; - iterate_and_advance(i, n, base, len, off, - copyin(p + off, base, len), - memcpy_from_iter(i, p + off, base, len) - ) + n = iterate_and_advance2(i, n, p, i, + copy_from_user_iter, + memcpy_from_iter_mc); kunmap_atomic(p); copied += n; offset += n; @@ -1180,32 +1281,64 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); +static __always_inline +size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_from_user(iter_from, to + progress, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_from_iter_csum(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + return 0; +} + size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) { - __wsum sum, next; - sum = *csum; if (WARN_ON_ONCE(!i->data_source)) return 0; - - iterate_and_advance(i, bytes, base, len, off, ({ - next = csum_and_copy_from_user(base, addr + off, len); - sum = csum_block_add(sum, next, off); - next ? 0 : len; - }), ({ - sum = csum_and_memcpy(addr + off, base, len, sum, off); - }) - ) - *csum = sum; - return bytes; + return iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); } EXPORT_SYMBOL(csum_and_copy_from_iter); +static __always_inline +size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_to_user(from + progress, iter_to, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_to_iter_csum(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + return 0; +} + size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, struct iov_iter *i) { struct csum_state *csstate = _csstate; - __wsum sum, next; + __wsum sum; if (WARN_ON_ONCE(i->data_source)) return 0; @@ -1219,14 +1352,10 @@ size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, } sum = csum_shift(csstate->csum, csstate->off); - iterate_and_advance(i, bytes, base, len, off, ({ - next = csum_and_copy_to_user(addr + off, base, len); - sum = csum_block_add(sum, next, off); - next ? 0 : len; - }), ({ - sum = csum_and_memcpy(base, addr + off, len, sum, off); - }) - ) + + bytes = iterate_and_advance2(i, bytes, (void *)addr, &sum, + copy_to_user_iter_csum, + memcpy_to_iter_csum); csstate->csum = csum_shift(sum, csstate->off); csstate->off += bytes; return bytes; From patchwork Wed Sep 13 16:56:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139248 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp65445vqi; Wed, 13 Sep 2023 19:30:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFhg/wFdSEIy4KsqiZQ1lVtP3HAs6lGmuOLKHb/G7C7ca4FWsllQDr/p+ozsvaeJpVuhJWk X-Received: by 2002:a81:8490:0:b0:586:cf7:2207 with SMTP id u138-20020a818490000000b005860cf72207mr4147359ywf.14.1694658629415; Wed, 13 Sep 2023 19:30:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694658629; cv=none; d=google.com; s=arc-20160816; b=PoaK8+ijT0o5IFMMrvhLQEAMJPmjLZDl2xpU9KJ7KVDh5IuY+tdBg5wpttVHhAP+04 4FBWdfmIPBQHUwW+D0BNTXItupO9XC9jUWmIfJ2IY1uWqqct48jdiOJf4F6uYpifOb+F ZpNQp3Xql9yECG+WvxJRyynhXLnqh1WguFYJ4hXel2vsnrCwr6zfcqrTjvWQ91Ug4/r+ F3AsdWcGRZR7Fy2+ba1t+Qt6Bx6of/ojsNAQCwqOuHbojfzbmL8zLg4Iug3psakya8yI IVgvIqwrm4/8k348RcMvvYaCXZWRGWv6j419P5/ql4MrgxlbbZU8hqPShRVrhllJYmQZ rjAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7f30E5rfqXOL8OIlALoKZyRz07q/M/xUz+BJRcAuap4=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=ipyv6DAZ+bK5QW8oKPV9wpvssvo1rO4IHSzvhGMwnnE8Wq3TmRqhQ9/lfEAG2DkHdW bjW4CMskRyRDSQKsCF3VWyQ/t3Y6vFoQkpOH4plRgF4BCG1f409E7P5C+UHCpAoKJODk QjnHd1tGbFqJukdpMAgR2lS4X4LrGe8kCWMFRwzyfy/P2+92ZJeLIXoVfa2pQM3HX+d0 0feGkWoirUHM/jPslhLUs+FPCZHv79o8WPTCUoyw+x+V+SpJTNAi/qlZHszk0yN75MM1 WglHzAZ6ecMI710JJV+vztNNKwHsAOWnaDikSpnwqarFtxG8robvbtrIzk9X8H+Py2jf i6Sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=X8Io225g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id e23-20020a62aa17000000b0068fe7c4147esi520948pff.391.2023.09.13.19.30.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 19:30:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=X8Io225g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 37C5F80B31F7; Wed, 13 Sep 2023 09:58:48 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231349AbjIMQ6U (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230364AbjIMQ6C (ORCPT ); Wed, 13 Sep 2023 12:58:02 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id C1C931BE4 for ; Wed, 13 Sep 2023 09:57:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7f30E5rfqXOL8OIlALoKZyRz07q/M/xUz+BJRcAuap4=; b=X8Io225goG2hh+l9kcO05IlnRznggCDmmxuez8sTVbo3dXyS5KFrVVClmLvb4JzDsraoFP h5iJOUPy8iAvpigEi8tFJCUlKYBxiTI/r8TagDnHvS/TSgfpBtJdo+FVJb0zjN5iK9w6RH 8ZEg6XyviFmhuNdCOyCRv73uO/fDjdg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-510-cMka_6aIPNiC8mOgNkW5Mw-1; Wed, 13 Sep 2023 12:57:06 -0400 X-MC-Unique: cMka_6aIPNiC8mOgNkW5Mw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9603B185A797; Wed, 13 Sep 2023 16:57:05 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id D5E5F40C6EBF; Wed, 13 Sep 2023 16:57:03 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 05/13] iov: Move iterator functions to a header file Date: Wed, 13 Sep 2023 17:56:40 +0100 Message-ID: <20230913165648.2570623-6-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:58:48 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776978366890095631 X-GMAIL-MSGID: 1776978366890095631 Move the iterator functions to a header file so that other operations that need to scan over an iterator can be added. For instance, the rbd driver could use this to scan a buffer to see if it is all zeros and libceph could use this to generate a crc. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/iov_iter.h | 261 +++++++++++++++++++++++++++++++++++++++ lib/iov_iter.c | 197 +---------------------------- 2 files changed, 262 insertions(+), 196 deletions(-) create mode 100644 include/linux/iov_iter.h diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h new file mode 100644 index 000000000000..836854847cdf --- /dev/null +++ b/include/linux/iov_iter.h @@ -0,0 +1,261 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* I/O iterator iteration building functions. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ + +#ifndef _LINUX_IOV_ITER_H +#define _LINUX_IOV_ITER_H + +#include +#include + +typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, + void *priv, void *priv2); +typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, size_t len, + void *priv, void *priv2); + +/* + * Handle ITER_UBUF. + */ +static __always_inline +size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) +{ + void __user *base = iter->ubuf; + size_t progress = 0, remain; + + remain = step(base + iter->iov_offset, 0, len, priv, priv2); + progress = len - remain; + iter->iov_offset += progress; + return progress; +} + +/* + * Handle ITER_IOVEC. + */ +static __always_inline +size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_ustep_f step) +{ + const struct iovec *p = iter->__iov; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->__iov = p; + iter->nr_segs -= p - iter->__iov; + iter->iov_offset = skip; + return progress; +} + +/* + * Handle ITER_KVEC. + */ +static __always_inline +size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct kvec *p = iter->kvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t part = min(len, p->iov_len - skip); + + if (likely(part)) { + remain = step(p->iov_base + skip, progress, part, priv, priv2); + consumed = part - remain; + progress += consumed; + skip += consumed; + len -= consumed; + if (skip < p->iov_len) + break; + } + p++; + skip = 0; + } while (len); + + iter->nr_segs -= p - iter->kvec; + iter->kvec = p; + iter->iov_offset = skip; + return progress; +} + +/* + * Handle ITER_BVEC. + */ +static __always_inline +size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + const struct bio_vec *p = iter->bvec; + size_t progress = 0, skip = iter->iov_offset; + + do { + size_t remain, consumed; + size_t offset = p->bv_offset + skip, part; + void *kaddr = kmap_local_page(p->bv_page + offset / PAGE_SIZE); + + part = min3(len, + (size_t)(p->bv_len - skip), + (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); + remain = step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); + kunmap_local(kaddr); + consumed = part - remain; + len -= consumed; + progress += consumed; + skip += consumed; + if (skip >= p->bv_len) { + skip = 0; + p++; + } + if (remain) + break; + } while (len); + + iter->nr_segs -= p - iter->bvec; + iter->bvec = p; + iter->iov_offset = skip; + return progress; +} + +/* + * Handle ITER_XARRAY. + */ +static __always_inline +size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void *priv2, + iov_step_f step) +{ + struct folio *folio; + size_t progress = 0; + loff_t start = iter->xarray_start + iter->iov_offset; + pgoff_t index = start / PAGE_SIZE; + XA_STATE(xas, iter->xarray, index); + + rcu_read_lock(); + xas_for_each(&xas, folio, ULONG_MAX) { + size_t remain, consumed, offset, part, flen; + + if (xas_retry(&xas, folio)) + continue; + if (WARN_ON(xa_is_value(folio))) + break; + if (WARN_ON(folio_test_hugetlb(folio))) + break; + + offset = offset_in_folio(folio, start + progress); + flen = min(folio_size(folio) - offset, len); + + while (flen) { + void *base = kmap_local_folio(folio, offset); + + part = min_t(size_t, flen, + PAGE_SIZE - offset_in_page(offset)); + remain = step(base, progress, part, priv, priv2); + kunmap_local(base); + + consumed = part - remain; + progress += consumed; + len -= consumed; + + if (remain || len == 0) + goto out; + flen -= consumed; + offset += consumed; + } + } + +out: + rcu_read_unlock(); + iter->iov_offset += progress; + return progress; +} + +/** + * iterate_and_advance2 - Iterate over an iterator + * @iter: The iterator to iterate over. + * @len: The amount to iterate over. + * @priv: Data for the step functions. + * @priv2: More data for the step functions. + * @ustep: Function for UBUF/IOVEC iterators; given __user addresses. + * @step: Function for other iterators; given kernel addresses. + * + * Iterate over the next part of an iterator, up to the specified length. The + * buffer is presented in segments, which for kernel iteration are broken up by + * physical pages and mapped, with the mapped address being presented. + * + * Two step functions, @step and @ustep, must be provided, one for handling + * mapped kernel addresses and the other is given user addresses which have the + * potential to fault since no pinning is performed. + * + * The step functions are passed the address and length of the segment, @priv, + * @priv2 and the amount of data so far iterated over (which can, for example, + * be added to @priv to point to the right part of a second buffer). The step + * functions should return the amount of the segment they didn't process (ie. 0 + * indicates complete processsing). + * + * This function returns the amount of data processed (ie. 0 means nothing was + * processed and the value of @len means processes to completion). + */ +static __always_inline +size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_ustep_f ustep, iov_step_f step) +{ + size_t progress; + + if (unlikely(iter->count < len)) + len = iter->count; + if (unlikely(!len)) + return 0; + + if (likely(iter_is_ubuf(iter))) + progress = iterate_ubuf(iter, len, priv, priv2, ustep); + else if (likely(iter_is_iovec(iter))) + progress = iterate_iovec(iter, len, priv, priv2, ustep); + else if (iov_iter_is_bvec(iter)) + progress = iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress = iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress = iterate_xarray(iter, len, priv, priv2, step); + else + progress = len; + iter->count -= progress; + return progress; +} + +/** + * iterate_and_advance - Iterate over an iterator + * @iter: The iterator to iterate over. + * @len: The amount to iterate over. + * @priv: Data for the step functions. + * @ustep: Function for UBUF/IOVEC iterators; given __user addresses. + * @step: Function for other iterators; given kernel addresses. + * + * As iterate_and_advance2(), but priv2 is always NULL. + */ +static __always_inline +size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, + iov_ustep_f ustep, iov_step_f step) +{ + return iterate_and_advance2(iter, len, priv, NULL, ustep, step); +} + +#endif /* _LINUX_IOV_ITER_H */ diff --git a/lib/iov_iter.c b/lib/iov_iter.c index b3ce6fa5f7a5..65374ee91ecd 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -13,202 +13,7 @@ #include #include #include - -typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len, - void *priv, void *priv2); -typedef size_t (*iov_ustep_f)(void __user *iter_base, size_t progress, size_t len, - void *priv, void *priv2); - -static __always_inline -size_t iterate_ubuf(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_ustep_f step) -{ - void __user *base = iter->ubuf; - size_t progress = 0, remain; - - remain = step(base + iter->iov_offset, 0, len, priv, priv2); - progress = len - remain; - iter->iov_offset += progress; - return progress; -} - -static __always_inline -size_t iterate_iovec(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_ustep_f step) -{ - const struct iovec *p = iter->__iov; - size_t progress = 0, skip = iter->iov_offset; - - do { - size_t remain, consumed; - size_t part = min(len, p->iov_len - skip); - - if (likely(part)) { - remain = step(p->iov_base + skip, progress, part, priv, priv2); - consumed = part - remain; - progress += consumed; - skip += consumed; - len -= consumed; - if (skip < p->iov_len) - break; - } - p++; - skip = 0; - } while (len); - - iter->__iov = p; - iter->nr_segs -= p - iter->__iov; - iter->iov_offset = skip; - return progress; -} - -static __always_inline -size_t iterate_kvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_step_f step) -{ - const struct kvec *p = iter->kvec; - size_t progress = 0, skip = iter->iov_offset; - - do { - size_t remain, consumed; - size_t part = min(len, p->iov_len - skip); - - if (likely(part)) { - remain = step(p->iov_base + skip, progress, part, priv, priv2); - consumed = part - remain; - progress += consumed; - skip += consumed; - len -= consumed; - if (skip < p->iov_len) - break; - } - p++; - skip = 0; - } while (len); - - iter->nr_segs -= p - iter->kvec; - iter->kvec = p; - iter->iov_offset = skip; - return progress; -} - -static __always_inline -size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_step_f step) -{ - const struct bio_vec *p = iter->bvec; - size_t progress = 0, skip = iter->iov_offset; - - do { - size_t remain, consumed; - size_t offset = p->bv_offset + skip, part; - void *kaddr = kmap_local_page(p->bv_page + offset / PAGE_SIZE); - - part = min3(len, - (size_t)(p->bv_len - skip), - (size_t)(PAGE_SIZE - offset % PAGE_SIZE)); - remain = step(kaddr + offset % PAGE_SIZE, progress, part, priv, priv2); - kunmap_local(kaddr); - consumed = part - remain; - len -= consumed; - progress += consumed; - skip += consumed; - if (skip >= p->bv_len) { - skip = 0; - p++; - } - if (remain) - break; - } while (len); - - iter->nr_segs -= p - iter->bvec; - iter->bvec = p; - iter->iov_offset = skip; - return progress; -} - -static __always_inline -size_t iterate_xarray(struct iov_iter *iter, size_t len, void *priv, void *priv2, - iov_step_f step) -{ - struct folio *folio; - size_t progress = 0; - loff_t start = iter->xarray_start + iter->iov_offset; - pgoff_t index = start / PAGE_SIZE; - XA_STATE(xas, iter->xarray, index); - - rcu_read_lock(); - xas_for_each(&xas, folio, ULONG_MAX) { - size_t remain, consumed, offset, part, flen; - - if (xas_retry(&xas, folio)) - continue; - if (WARN_ON(xa_is_value(folio))) - break; - if (WARN_ON(folio_test_hugetlb(folio))) - break; - - offset = offset_in_folio(folio, start + progress); - flen = min(folio_size(folio) - offset, len); - - while (flen) { - void *base = kmap_local_folio(folio, offset); - - part = min_t(size_t, flen, - PAGE_SIZE - offset_in_page(offset)); - remain = step(base, progress, part, priv, priv2); - kunmap_local(base); - - consumed = part - remain; - progress += consumed; - len -= consumed; - - if (remain || len == 0) - goto out; - flen -= consumed; - offset += consumed; - } - } - -out: - rcu_read_unlock(); - iter->iov_offset += progress; - return progress; -} - -static __always_inline -size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv, - void *priv2, iov_ustep_f ustep, iov_step_f step) -{ - size_t progress; - - if (unlikely(iter->count < len)) - len = iter->count; - if (unlikely(!len)) - return 0; - - if (likely(iter_is_ubuf(iter))) - progress = iterate_ubuf(iter, len, priv, priv2, ustep); - else if (likely(iter_is_iovec(iter))) - progress = iterate_iovec(iter, len, priv, priv2, ustep); - else if (iov_iter_is_bvec(iter)) - progress = iterate_bvec(iter, len, priv, priv2, step); - else if (iov_iter_is_kvec(iter)) - progress = iterate_kvec(iter, len, priv, priv2, step); - else if (iov_iter_is_xarray(iter)) - progress = iterate_xarray(iter, len, priv, priv2, step); - else - progress = len; - iter->count -= progress; - return progress; -} - -static __always_inline -size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, - iov_ustep_f ustep, iov_step_f step) -{ - return iterate_and_advance2(iter, len, priv, NULL, ustep, step); -} +#include static __always_inline size_t copy_to_user_iter(void __user *iter_to, size_t progress, From patchwork Wed Sep 13 16:56:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139412 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp178566vqi; Thu, 14 Sep 2023 00:47:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFhks+bN1jwLv78V0+15SeQtHWVEpZ9bTypNjqwVY1LpQlCfYHSmj/POhs5w81hN7jL/3pa X-Received: by 2002:a05:6358:919e:b0:133:ac7:c84b with SMTP id j30-20020a056358919e00b001330ac7c84bmr5012851rwa.12.1694677622868; Thu, 14 Sep 2023 00:47:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694677622; cv=none; d=google.com; s=arc-20160816; b=hWkGj/PyenxF/CL/rhH+oCnAG3EvSGBV9XOkbwPzDCvvXykGP2kG5fE/9bwWur6kJS QlER0iYbt4gNY7Uk6YJLzq56V624vr5CzGxF9r9W8Czqz2zerh5xxa/d6Rt1+ue4oNGw O4mGoTk0ujGijswICQPkjSoWg/OEZcY+nI8j+YFS421C7wU12HAckdSmsJJD3jIPPBYT yBOVYdKDmrkMdoJ38TKJDOMU7sFDiKT1TcGElsAPpR6iCoxURvR+HJPYK2h7ulYRLIyi Mf45cN6aHTcTCNiZpqvEgK6++RX1tjWh4FefoB5/7oI/4v0GIQLooJxIpD86Vd4j1FeD oCKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rhHz2rkHFFU/pzH+hQA/LimHSXRi7J1g2T3941mk2Gc=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=Fa/K2KfQ0PUuaJJ+6lu9dLaT0aEznvNNWuVCnvQ6u7TfcYCT/2G2pRHqkXclPMzs0C gmHkUVI/r/Jd+9OIoLMw6iyNWQByAZGgtWYoKw2b4e0vbnoxja4Od2avwmERefPuxYOg OfijHOqFEtE6vfRoRFYxxfwANaIalBaATWWKyUIC3ef6njl7Gt34/TLwgQlp1wM/Qz0d 5r7FeFnJLzJ5IzECkea9OnvKeDH+9zuPSYzp7M3zZTTzbtaeNC0O0YtsXJRfbhmcNgIB XO4214AiV+l8Jhzl92wNFqN/wirVbarRN6PgQiupL3CzVQuCp/9wGVZHU2HpE+llW8vU kFHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=AjUM9IHV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id w185-20020a6362c2000000b0055c872b768fsi940050pgb.531.2023.09.14.00.47.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Sep 2023 00:47:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=AjUM9IHV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 589138229D01; Wed, 13 Sep 2023 10:00:05 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231438AbjIMQ61 (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230451AbjIMQ6H (ORCPT ); Wed, 13 Sep 2023 12:58:07 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 92DFC1BF6 for ; Wed, 13 Sep 2023 09:57:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624235; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rhHz2rkHFFU/pzH+hQA/LimHSXRi7J1g2T3941mk2Gc=; b=AjUM9IHV8EyrerwDylNpW8nXDKqeNL7PHx7rAPvhZ6YXLvsa/PJCesiK68L5PcNrQelbF9 jgyvkkDpbPMZ8Jq7lZdsSJqlvDvmK+0UCQqxMFyD5DpAwthuyuB8Nal+SGF1lwpsVqKIyM KBnJ8VcCNYz4BqODqsrPlcA3H61M+64= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-670-2tcJ0K_0Mga54Zz4FQYBNQ-1; Wed, 13 Sep 2023 12:57:08 -0400 X-MC-Unique: 2tcJ0K_0Mga54Zz4FQYBNQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D5F09101A529; Wed, 13 Sep 2023 16:57:07 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3B4AF1054FC1; Wed, 13 Sep 2023 16:57:06 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 06/13] iov_iter: Add a kernel-type iterator-only iteration function Date: Wed, 13 Sep 2023 17:56:41 +0100 Message-ID: <20230913165648.2570623-7-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 10:00:05 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776998283475460610 X-GMAIL-MSGID: 1776998283475460610 Add an iteration function that can only iterate over kernel internal-type iterators (ie. BVEC, KVEC, XARRAY) and not user-backed iterators (ie. UBUF and IOVEC). This allows for smaller iterators to be built when it is known the caller won't have a user-backed iterator. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/iov_iter.h | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h index 836854847cdf..46e628640934 100644 --- a/include/linux/iov_iter.h +++ b/include/linux/iov_iter.h @@ -258,4 +258,39 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv, return iterate_and_advance2(iter, len, priv, NULL, ustep, step); } +/** + * iterate_and_advance_kernel - Iterate over a kernel iterator + * @iter: The iterator to iterate over. + * @len: The amount to iterate over. + * @priv: Data for the step functions. + * @step: Processing function; given kernel addresses. + * + * Like iterate_and_advance2(), but rejected UBUF and IOVEC iterators and does + * not take a user-step function. + */ +static __always_inline +size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv, + void *priv2, iov_step_f step) +{ + size_t progress; + + if (unlikely(iter->count < len)) + len = iter->count; + if (unlikely(!len)) + return 0; + + if (iov_iter_is_bvec(iter)) + progress = iterate_bvec(iter, len, priv, priv2, step); + else if (iov_iter_is_kvec(iter)) + progress = iterate_kvec(iter, len, priv, priv2, step); + else if (iov_iter_is_xarray(iter)) + progress = iterate_xarray(iter, len, priv, priv2, step); + else if (WARN_ON_ONCE(unlikely(iter_is_ubuf(iter) || iter_is_iovec(iter)))) + progress = 0; + else + progress = len; + iter->count -= progress; + return progress; +} + #endif /* _LINUX_IOV_ITER_H */ From patchwork Wed Sep 13 16:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139245 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp62365vqi; Wed, 13 Sep 2023 19:21:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFHbiDcNG/dwUZ/J7NLHqLEvid/UC0TkFAb+BHM1go3gbt9IdWEKthm2NkxBX1nzEq2SHyw X-Received: by 2002:a17:90a:5908:b0:26f:4685:5b6c with SMTP id k8-20020a17090a590800b0026f46855b6cmr779916pji.19.1694658062437; Wed, 13 Sep 2023 19:21:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694658062; cv=none; d=google.com; s=arc-20160816; b=vqt3JuDjyC3Zmu78tD6dXtkVyysQdSo06WFnkKm5i60YzomJHkIQtreUUr8YSGFtFq AWviVhTIEfAGGBhODQJFZgUl6G5agR2wQ25nMmMJi/2KONwksJMDwaLpSYPKVJkPnP1g r2oho4rdJbkH5FSpHEI1Fbv2SjdVWau5WIwzOFsBsI3B8y+L471V6YBLlCYVDNUhyNW0 Zh/yTuISsFLW+v6B6f7pQNbFU6KbayvVvQCWmGq4MPVZE/Y0E8n2nec/nC5mPbpQ1em7 ZpC1B6/yu8prfwp0DGVZb2qbdA8VKMbF70U4uyOy2YGh8vuvcba06OICtnBIBgl03R1l sv2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xJaTbGB+togQTw9SIdIdixTkcz5/QJ2b3c5fZ9o3sSg=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=StZZ+sp5CiRFZXPe2MScpOC+FIpYeS376BoNz9IQTkA90W2zKoUYaxcXGQwac1NUTM cIxnQ2xB+H8rDkLuGeCoFyDLg6WlSqZizTDpq46suFHPUt8aPMyPUteUCJz0MEYFFOcp sgwWZCWfWEylOifAGGarF/v9822r07dqML0TYwRWYJBghew42eSmgQW/k4FLENvQrqtv DqHrDizetntO0jFXZAnB9kfma/hqGPvZE3JrkPTkYKu63tGhAm/LI8AwILNtKZ/Lf4SP giA0PcX8LgmhrFRcW3YDEv0nZzIml2cA+wCN61nZF8CXKtVCjUKWxfC7bz3YuIWXYu92 u3RA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bSzUOGuL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id q93-20020a17090a756600b0026404467b7dsi2766096pjk.110.2023.09.13.19.21.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 19:21:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bSzUOGuL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id E733E80A8553; Wed, 13 Sep 2023 09:59:09 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230497AbjIMQ6h (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229533AbjIMQ6L (ORCPT ); Wed, 13 Sep 2023 12:58:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 0B7321BF8 for ; Wed, 13 Sep 2023 09:57:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624237; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xJaTbGB+togQTw9SIdIdixTkcz5/QJ2b3c5fZ9o3sSg=; b=bSzUOGuLCNd5uJVIDY/AMr17mZvBn2JquLxMEGJsV4fiG0CUs62twXaivGbTVgA8VS3xyE msckqlrkwYanyggNUgVkWF3GFr8/BMQiM+RQEdVXUTSHPAHcX0sp1OIACAEmRlixIa/oPr KNJ6k7Ei9ZiC78Mu5K+fZTyqWVKSi6w= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-694-6buX1QvqNt-weeixz82rJw-1; Wed, 13 Sep 2023 12:57:12 -0400 X-MC-Unique: 6buX1QvqNt-weeixz82rJw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 222FF18312C9; Wed, 13 Sep 2023 16:57:10 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7B42840C6EBF; Wed, 13 Sep 2023 16:57:08 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 07/13] iov_iter: Make copy_from_iter() always handle MCE Date: Wed, 13 Sep 2023 17:56:42 +0100 Message-ID: <20230913165648.2570623-8-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:59:10 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776977772659702663 X-GMAIL-MSGID: 1776977772659702663 Make copy_from_iter() always catch an MCE and return a short copy and make the coredump code rely on that. This requires arch support in the form of a memcpy_mc() function that returns the length copied. [?] Is it better to kill the thread in the event of an MCE occurring? Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- arch/x86/include/asm/mce.h | 23 +++++++++++++++++++++++ fs/coredump.c | 1 - lib/iov_iter.c | 12 +++++------- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h index 180b1cbfcc4e..77ce2044536c 100644 --- a/arch/x86/include/asm/mce.h +++ b/arch/x86/include/asm/mce.h @@ -353,4 +353,27 @@ static inline void mce_hygon_feature_init(struct cpuinfo_x86 *c) { return mce_am unsigned long copy_mc_fragile_handle_tail(char *to, char *from, unsigned len); +static __always_inline __must_check +size_t memcpy_mc(void *to, const void *from, size_t len) +{ +#ifdef CONFIG_ARCH_HAS_COPY_MC + /* + * If CPU has FSRM feature, use 'rep movs'. + * Otherwise, use rep_movs_alternative. + */ + asm volatile( + "1:\n\t" + ALTERNATIVE("rep movsb", + "call rep_movs_alternative", ALT_NOT(X86_FEATURE_FSRM)) + "2:\n" + _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_DEFAULT_MCE_SAFE) + :"+c" (len), "+D" (to), "+S" (from), ASM_CALL_CONSTRAINT + : : "memory", "rax", "r8", "r9", "r10", "r11"); +#else + memcpy(to, from, len); + return 0; +#endif + return len; +} + #endif /* _ASM_X86_MCE_H */ diff --git a/fs/coredump.c b/fs/coredump.c index 9d235fa14ab9..ad54102a5e14 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -884,7 +884,6 @@ static int dump_emit_page(struct coredump_params *cprm, struct page *page) pos = file->f_pos; bvec_set_page(&bvec, page, PAGE_SIZE, 0); iov_iter_bvec(&iter, ITER_SOURCE, &bvec, 1, PAGE_SIZE); - iov_iter_set_copy_mc(&iter); n = __kernel_write_iter(cprm->file, &iter, &pos); if (n != PAGE_SIZE) return 0; diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 65374ee91ecd..b574601783bc 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -14,6 +14,7 @@ #include #include #include +#include static __always_inline size_t copy_to_user_iter(void __user *iter_to, size_t progress, @@ -253,14 +254,11 @@ size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i) EXPORT_SYMBOL_GPL(_copy_mc_to_iter); #endif /* CONFIG_ARCH_HAS_COPY_MC */ -static size_t memcpy_from_iter_mc(void *iter_from, size_t progress, - size_t len, void *to, void *priv2) +static __always_inline +size_t memcpy_from_iter_mc(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) { - struct iov_iter *iter = priv2; - - if (iov_iter_is_copy_mc(iter)) - return copy_mc_to_kernel(to + progress, iter_from, len); - return memcpy_from_iter(iter_from, progress, len, to, priv2); + return memcpy_mc(to + progress, iter_from, len); } size_t _copy_from_iter(void *addr, size_t bytes, struct iov_iter *i) From patchwork Wed Sep 13 16:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139249 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp65897vqi; Wed, 13 Sep 2023 19:31:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGTzRGtelkjEwjwxtm3NQYcZuDFGebje6fkObvZjxDKFODlsUKyLjNi/vms1LDbr+b+m5KX X-Received: by 2002:a81:9246:0:b0:586:a684:e7b6 with SMTP id j67-20020a819246000000b00586a684e7b6mr4586857ywg.9.1694658708927; Wed, 13 Sep 2023 19:31:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694658708; cv=none; d=google.com; s=arc-20160816; b=aacVvd6JgQBI9UNdhYJVXc2yxqffU/n4gi8itvi5yeSSaFrpg03jTJxhDTGj5qo1ks VHUwsLygAbAYI/CUAPreSonHo2n58X+iW0Ivwqawx4DeMK02AQyISGKRzGU4yeih0NML AVk9Vtk0K1THloS/E4WHMBfFS5Ng+4JT7zW51IF0JovhY6avEq7o1cMXevX7RPojUv/A megEOE71MkZNRdfdKCjowD1fQ+rC73ETTFrzO8m67NxijBvcNoqlGTsu1OuSoFoFXk79 H0W9xcEZE3ZvMvw781m+EH+1yqXVylUy28BcCKg3reimnOOygl+AUtcK13gHznjEBqJv 12KA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=p+t8ChtS3C0mCAtd18JzNSIJcCnKtYlpMvVgh9PlkwE=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=nHbZ+iky2AHmy+NBSXE6wDr5mKiqyyrCJpfWDlyo/C4uaCRZz2iYTYXRlZ6KQWIKPb F3jQZkwN0sIqBWXJQ3CBPcFc9rfa6N8M4aI9yQC35mxriL+eHhZF4vvujEuVTFAqzZ1K eIqeza/lGtDWeEB4jLi5me3zkEdZ0h+8JugQCKw2lV+lsOz53PfotjWMEJDKSHePY0y2 Hjlv/AdEfG+aVXz2mqyNM9is2jpNlZXxiYFLD9aDxL6lAWCEZC/TUlpexyQBCmfvb3YA ESNOyCKv9ul6lf/d+LDpSZt3Fdk9PkUI/lESyXkDtPdEjAfqeXfwUlX326OQnnRbFwmk q9/g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UQPF4PWs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id fi37-20020a056a0039a500b0068fb6fc3ff1si595665pfb.209.2023.09.13.19.31.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 19:31:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=UQPF4PWs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 9C65280B90A5; Wed, 13 Sep 2023 09:59:05 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229923AbjIMQ6S (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230359AbjIMQ6C (ORCPT ); Wed, 13 Sep 2023 12:58:02 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 93E851BE8 for ; Wed, 13 Sep 2023 09:57:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p+t8ChtS3C0mCAtd18JzNSIJcCnKtYlpMvVgh9PlkwE=; b=UQPF4PWsnCasFEdyyysVTcjyKbiJexdcwJzzX/ZEtw8yAi2J7HyAlLEsyS0w3Bj21DDxBO usWo7iqC4U+uS6GwPByAIIRnq6Of9xCdJOhCrKo1rEsgVIVrBFnCtk2YANkkgGk1VxNQ3f gfbHE/BvFvLR6nsntTyYuAIJD3R+hwY= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-542-5xC_0HubPGGLzilTHkSTMQ-1; Wed, 13 Sep 2023 12:57:13 -0400 X-MC-Unique: 5xC_0HubPGGLzilTHkSTMQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 783FB85531D; Wed, 13 Sep 2023 16:57:12 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id D43F240C6EA8; Wed, 13 Sep 2023 16:57:10 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 08/13] iov_iter: Remove the copy_mc flag and associated functions Date: Wed, 13 Sep 2023 17:56:43 +0100 Message-ID: <20230913165648.2570623-9-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:59:05 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776978450258001813 X-GMAIL-MSGID: 1776978450258001813 copy_to/from_iter() now always return a short copy on MCE, so we don't need to have special handling. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org --- include/linux/uio.h | 16 ---------------- lib/iov_iter.c | 5 ----- 2 files changed, 21 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index e2a248dad80b..cc250892872e 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -40,7 +40,6 @@ struct iov_iter_state { struct iov_iter { u8 iter_type; - bool copy_mc; bool nofault; bool data_source; union { @@ -251,22 +250,8 @@ size_t _copy_from_iter_flushcache(void *addr, size_t bytes, struct iov_iter *i); #ifdef CONFIG_ARCH_HAS_COPY_MC size_t _copy_mc_to_iter(const void *addr, size_t bytes, struct iov_iter *i); -static inline void iov_iter_set_copy_mc(struct iov_iter *i) -{ - i->copy_mc = true; -} - -static inline bool iov_iter_is_copy_mc(const struct iov_iter *i) -{ - return i->copy_mc; -} #else #define _copy_mc_to_iter _copy_to_iter -static inline void iov_iter_set_copy_mc(struct iov_iter *i) { } -static inline bool iov_iter_is_copy_mc(const struct iov_iter *i) -{ - return false; -} #endif size_t iov_iter_zero(size_t bytes, struct iov_iter *); @@ -381,7 +366,6 @@ static inline void iov_iter_ubuf(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter) { .iter_type = ITER_UBUF, - .copy_mc = false, .data_source = direction, .ubuf = buf, .count = count, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index b574601783bc..66d2a16774fe 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -169,7 +169,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter) { .iter_type = ITER_IOVEC, - .copy_mc = false, .nofault = false, .data_source = direction, .__iov = iov, @@ -629,7 +628,6 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter){ .iter_type = ITER_KVEC, - .copy_mc = false, .data_source = direction, .kvec = kvec, .nr_segs = nr_segs, @@ -646,7 +644,6 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction, WARN_ON(direction & ~(READ | WRITE)); *i = (struct iov_iter){ .iter_type = ITER_BVEC, - .copy_mc = false, .data_source = direction, .bvec = bvec, .nr_segs = nr_segs, @@ -675,7 +672,6 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int direction, BUG_ON(direction & ~1); *i = (struct iov_iter) { .iter_type = ITER_XARRAY, - .copy_mc = false, .data_source = direction, .xarray = xarray, .xarray_start = start, @@ -699,7 +695,6 @@ void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count) BUG_ON(direction != READ); *i = (struct iov_iter){ .iter_type = ITER_DISCARD, - .copy_mc = false, .data_source = false, .count = count, .iov_offset = 0 From patchwork Wed Sep 13 16:56:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139056 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp45322vqb; Wed, 13 Sep 2023 12:41:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEbh+gqG9xVab75T41H2egeosOj6G4wsUogkys99rnswTwCb198qJznlskej80MplZir2dq X-Received: by 2002:a17:90b:1b47:b0:271:c314:a591 with SMTP id nv7-20020a17090b1b4700b00271c314a591mr3305840pjb.47.1694634099481; Wed, 13 Sep 2023 12:41:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694634099; cv=none; d=google.com; s=arc-20160816; b=zHC4dTIDmzrZqg1aZgsJGvOVxNrfaNVsw20WwQVyf5pe0zqIC/vTF8J6kKFyOc1jSa 0R7XHMkoJ5fMwoeuaY8P85jTPiGtS1YMEA/2Rg7IwWFHt+0zjH9kIYvzvSxHqKcYExJE BvEu3cbwpCJx84z5WlLqiLunxsHYxNGYQ4TcgemBhtBZoInf/OsTq0yj/Az0s1FBrZY0 yPDcu+eYaGFrsj/cNiZFa75c623icf1WYlTxnfXNnR+czBm5swpK5VV1lIxvbexjuekY ylRoH5iMxblx6XIooyZG0/Kujn3GzJzEISk5U6sIzT/VS3I2bUZfqOm7ZlZ9yBU/8C62 xmbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9uMtbW5W1n5t6FTdNdAKrkGJdOcWoyjxyU/ZljquCxs=; fh=JMFAeSz7Gk3tUajr4TCvrb0uYdtniI0MVPhc5ULEcNg=; b=KggFFIk3RgSzo7xVwcW4UUaXCDvVoYH/p0oTrICFKYZHjb4lwgjCurRZVb3SFFgmDO G6hrufidRFvvP+yMM66XATlyb+J6ixJIeugesALRhrmqkzqpcgEGyiqzFAtFPfOITxFP 1BkYG4t/KBP8Cl5qQ5bXih2b5VhxrzfteIs9uBX6/zcZDs2xmviQF2jzepu750xQNIQQ TcoRhGJ7oHYbvZQ4fY/B4pfnl9GUKiVEV/0Nk7sVUr15ayvkJ2035xUoIY26eqK+sK/x TokMT2Lp2GwVtBoS994S6pTtHFNZ/yQbgNdvzoDAcDVuPjifwBGOL1oyiZIz9vSXCj00 U5dA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MEzJL8x2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id u6-20020a17090a1f0600b0027405eddd3fsi2130738pja.19.2023.09.13.12.41.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 12:41:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MEzJL8x2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id A092B81F06BB; Wed, 13 Sep 2023 09:58:59 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230204AbjIMQ6d (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231132AbjIMQ6N (ORCPT ); Wed, 13 Sep 2023 12:58:13 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2B3DD1BFC for ; Wed, 13 Sep 2023 09:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9uMtbW5W1n5t6FTdNdAKrkGJdOcWoyjxyU/ZljquCxs=; b=MEzJL8x2v4yrpugmXF5YHB4kgJEcu5MdsYXicGFpnARWDcct1GzpO10Gmr/vC7lA2JI2w7 wN1U8GNbEcUNEFTqbkei1vw8hsJKhm7QaD5T9F6CrP1kxOQfVuN1wAxSk9kVef+PvKXkBv MQ2+3gRPikXikY69i8mEieLOWtO4PDE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-453-8IErGMcgOqWpOUPhN4ftHA-1; Wed, 13 Sep 2023 12:57:15 -0400 X-MC-Unique: 8IErGMcgOqWpOUPhN4ftHA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B771D8219BE; Wed, 13 Sep 2023 16:57:14 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 139332026D4B; Wed, 13 Sep 2023 16:57:12 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 09/13] iov_iter, net: Move csum_and_copy_to/from_iter() to net/ Date: Wed, 13 Sep 2023 17:56:44 +0100 Message-ID: <20230913165648.2570623-10-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:58:59 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776952645204870022 X-GMAIL-MSGID: 1776952645204870022 Move csum_and_copy_to/from_iter() to net code now that the iteration framework can be #included. Signed-off-by: David Howells --- include/linux/skbuff.h | 25 ++++++++++++ include/linux/uio.h | 18 --------- lib/iov_iter.c | 89 ------------------------------------------ net/core/datagram.c | 50 +++++++++++++++++++++++- net/core/skbuff.c | 33 ++++++++++++++++ 5 files changed, 107 insertions(+), 108 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 4174c4b82d13..d0656cc11c16 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3679,6 +3679,31 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int l return __skb_put_padto(skb, len, true); } +static inline __wsum csum_and_memcpy(void *to, const void *from, size_t len, + __wsum sum, size_t off) +{ + __wsum next = csum_partial_copy_nocheck(from, to, len); + return csum_block_add(sum, next, off); +} + +struct csum_state { + __wsum csum; + size_t off; +}; + +size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); + +static __always_inline __must_check +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, + __wsum *csum, struct iov_iter *i) +{ + size_t copied = csum_and_copy_from_iter(addr, bytes, csum, i); + if (likely(copied == bytes)) + return true; + iov_iter_revert(i, copied); + return false; +} + static inline int skb_add_data(struct sk_buff *skb, struct iov_iter *from, int copy) { diff --git a/include/linux/uio.h b/include/linux/uio.h index cc250892872e..c335b95626af 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -326,24 +326,6 @@ iov_iter_npages_cap(struct iov_iter *i, int maxpages, size_t max_bytes) return npages; } -struct csum_state { - __wsum csum; - size_t off; -}; - -size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *csstate, struct iov_iter *i); -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); - -static __always_inline __must_check -bool csum_and_copy_from_iter_full(void *addr, size_t bytes, - __wsum *csum, struct iov_iter *i) -{ - size_t copied = csum_and_copy_from_iter(addr, bytes, csum, i); - if (likely(copied == bytes)) - return true; - iov_iter_revert(i, copied); - return false; -} size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 66d2a16774fe..5b2d053f057f 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -10,7 +10,6 @@ #include #include #include -#include #include #include #include @@ -179,13 +178,6 @@ void iov_iter_init(struct iov_iter *i, unsigned int direction, } EXPORT_SYMBOL(iov_iter_init); -static __wsum csum_and_memcpy(void *to, const void *from, size_t len, - __wsum sum, size_t off) -{ - __wsum next = csum_partial_copy_nocheck(from, to, len); - return csum_block_add(sum, next, off); -} - size_t _copy_to_iter(const void *addr, size_t bytes, struct iov_iter *i) { if (WARN_ON_ONCE(i->data_source)) @@ -1079,87 +1071,6 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); -static __always_inline -size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, - size_t len, void *to, void *priv2) -{ - __wsum next, *csum = priv2; - - next = csum_and_copy_from_user(iter_from, to + progress, len); - *csum = csum_block_add(*csum, next, progress); - return next ? 0 : len; -} - -static __always_inline -size_t memcpy_from_iter_csum(void *iter_from, size_t progress, - size_t len, void *to, void *priv2) -{ - __wsum *csum = priv2; - - *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); - return 0; -} - -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, - struct iov_iter *i) -{ - if (WARN_ON_ONCE(!i->data_source)) - return 0; - return iterate_and_advance2(i, bytes, addr, csum, - copy_from_user_iter_csum, - memcpy_from_iter_csum); -} -EXPORT_SYMBOL(csum_and_copy_from_iter); - -static __always_inline -size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, - size_t len, void *from, void *priv2) -{ - __wsum next, *csum = priv2; - - next = csum_and_copy_to_user(from + progress, iter_to, len); - *csum = csum_block_add(*csum, next, progress); - return next ? 0 : len; -} - -static __always_inline -size_t memcpy_to_iter_csum(void *iter_to, size_t progress, - size_t len, void *from, void *priv2) -{ - __wsum *csum = priv2; - - *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); - return 0; -} - -size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, - struct iov_iter *i) -{ - struct csum_state *csstate = _csstate; - __wsum sum; - - if (WARN_ON_ONCE(i->data_source)) - return 0; - if (unlikely(iov_iter_is_discard(i))) { - // can't use csum_memcpy() for that one - data is not copied - csstate->csum = csum_block_add(csstate->csum, - csum_partial(addr, bytes, 0), - csstate->off); - csstate->off += bytes; - return bytes; - } - - sum = csum_shift(csstate->csum, csstate->off); - - bytes = iterate_and_advance2(i, bytes, (void *)addr, &sum, - copy_to_user_iter_csum, - memcpy_to_iter_csum); - csstate->csum = csum_shift(sum, csstate->off); - csstate->off += bytes; - return bytes; -} -EXPORT_SYMBOL(csum_and_copy_to_iter); - size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, struct iov_iter *i) { diff --git a/net/core/datagram.c b/net/core/datagram.c index 176eb5834746..37c89d0933b7 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -50,7 +50,7 @@ #include #include #include -#include +#include #include #include @@ -716,6 +716,54 @@ int zerocopy_sg_from_iter(struct sk_buff *skb, struct iov_iter *from) } EXPORT_SYMBOL(zerocopy_sg_from_iter); +static __always_inline +size_t copy_to_user_iter_csum(void __user *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_to_user(from + progress, iter_to, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +static __always_inline +size_t memcpy_to_iter_csum(void *iter_to, size_t progress, + size_t len, void *from, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + return 0; +} + +static size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, + struct iov_iter *i) +{ + struct csum_state *csstate = _csstate; + __wsum sum; + + if (WARN_ON_ONCE(i->data_source)) + return 0; + if (unlikely(iov_iter_is_discard(i))) { + // can't use csum_memcpy() for that one - data is not copied + csstate->csum = csum_block_add(csstate->csum, + csum_partial(addr, bytes, 0), + csstate->off); + csstate->off += bytes; + return bytes; + } + + sum = csum_shift(csstate->csum, csstate->off); + + bytes = iterate_and_advance2(i, bytes, (void *)addr, &sum, + copy_to_user_iter_csum, + memcpy_to_iter_csum); + csstate->csum = csum_shift(sum, csstate->off); + csstate->off += bytes; + return bytes; +} + /** * skb_copy_and_csum_datagram - Copy datagram to an iovec iterator * and update a checksum. diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 4eaf7ed0d1f4..5dbdfce2d05f 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -62,6 +62,7 @@ #include #include #include +#include #include #include @@ -6931,3 +6932,35 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, return spliced ?: ret; } EXPORT_SYMBOL(skb_splice_from_iter); + +static __always_inline +size_t memcpy_from_iter_csum(void *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum *csum = priv2; + + *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + return 0; +} + +static __always_inline +size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, + size_t len, void *to, void *priv2) +{ + __wsum next, *csum = priv2; + + next = csum_and_copy_from_user(iter_from, to + progress, len); + *csum = csum_block_add(*csum, next, progress); + return next ? 0 : len; +} + +size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, + struct iov_iter *i) +{ + if (WARN_ON_ONCE(!i->data_source)) + return 0; + return iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); +} +EXPORT_SYMBOL(csum_and_copy_from_iter); From patchwork Wed Sep 13 16:56:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp139769vqi; Wed, 13 Sep 2023 23:06:53 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH3OsxXjEzz+Ftcha5LLALNr74PJCMEAwFF7uivE3Kagee7GM/jn7NxjB5+yJvw+OpGmQUw X-Received: by 2002:a17:902:e884:b0:1bf:8916:8d21 with SMTP id w4-20020a170902e88400b001bf89168d21mr1409671plg.13.1694671612744; Wed, 13 Sep 2023 23:06:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694671612; cv=none; d=google.com; s=arc-20160816; b=DD41yBNVmZcRu1YHaloNKl6xz+eOzBbJwrVG9U+YHNoPGj57zygeQW1Vjhm5JEk1iJ ZMb6Kc7HFQw5ob/GR0cnZM6ZFoqJU59vvI5AYExTcpREBGgvQtXhKuzdkf0hXzxOTikz 5ynd0ahWvmwmGognKYzraK88C0+MBU2k9+0hgD0z9Da2iXmDqt7FlEOHsr6NJHWCi6vf AKycOFAoVA+1WI/uGIa8qdghg1kRsocm3HF2w8axR5JA1WYuJ0w0Cdvvckp82KQ1iFT0 GODww+MRSFj64TYhFuDG25yaCvVG8pQ3a8m9wJXkOyp07/GfvpvHAezTgv1l+cjGKNCy nL2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i+29vAFZrXF9naWng0ipN/2z31gEil+IMifE0VymLF4=; fh=D00b+ExO44XKLceYW8cVdIEOVtr2e1iL7r354EPHkWw=; b=qTieTFWcykfN2B/GQlLSZpcipuYp8hCvpxbc0IxHfqp1pHqhR2Wx4Cy5POnWqDNtLj qMiljtg5K5WjAGRlSM/CjayvJCPr12KH2ZeFkH/km2shiY+8BNy/wMHxN1j2C+YYKLky wy388KEJ8/Qg4WVgAn2SRZ2ijXi8N1Yjv1GR4O96wa3Wa+5JvUFc2Q2lLfOX9SMWW4Es bXc5w+mSUYXxC19aARoHII/uupQFBxx1iQssj9pL3yaeMmKd4rwZpziIkfqrA5qLWfs6 +pQ/1aXURJWDL6Kv/Pkw5xgrUf/vH63lHzUDx05CvbUJwEJgN15hu4StMwmrA85jpkw7 AvwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LT3y4Q7L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id lg8-20020a170902fb8800b001c2027fa3f5si956800plb.114.2023.09.13.23.06.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 23:06:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=LT3y4Q7L; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 7E054828DFE6; Wed, 13 Sep 2023 09:59:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231345AbjIMQ6u (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231328AbjIMQ6U (ORCPT ); Wed, 13 Sep 2023 12:58:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 643361BF1 for ; Wed, 13 Sep 2023 09:57:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i+29vAFZrXF9naWng0ipN/2z31gEil+IMifE0VymLF4=; b=LT3y4Q7Lmppy8WKwJwLzSdPbdfBQfMfvl3WzYLbPmqVfBHcwFFVIxK0xoeE9xSlirXtU1W XEWzNXt08ZUbmFa+Ajn/uc2gjlqDbaGe42lJUMsOeq0xvthRMXzwe2fv1/PmwkEAsNBK1t uaXdzhoxT5VvMOqXspl7JPrDUK8bXGM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-536-VozdxTMENjmI9CZVmuDJyA-1; Wed, 13 Sep 2023 12:57:18 -0400 X-MC-Unique: VozdxTMENjmI9CZVmuDJyA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 728E5857A9E; Wed, 13 Sep 2023 16:57:17 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5DF682026D4B; Wed, 13 Sep 2023 16:57:15 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 10/13] iov_iter, net: Fold in csum_and_memcpy() Date: Wed, 13 Sep 2023 17:56:45 +0100 Message-ID: <20230913165648.2570623-11-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:59:28 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776991980771405251 X-GMAIL-MSGID: 1776991980771405251 Fold csum_and_memcpy() in to its callers. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 7 ------- net/core/datagram.c | 3 ++- net/core/skbuff.c | 3 ++- 3 files changed, 4 insertions(+), 9 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index d0656cc11c16..c81ef5d76953 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3679,13 +3679,6 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int l return __skb_put_padto(skb, len, true); } -static inline __wsum csum_and_memcpy(void *to, const void *from, size_t len, - __wsum sum, size_t off) -{ - __wsum next = csum_partial_copy_nocheck(from, to, len); - return csum_block_add(sum, next, off); -} - struct csum_state { __wsum csum; size_t off; diff --git a/net/core/datagram.c b/net/core/datagram.c index 37c89d0933b7..452620dd41e4 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -732,8 +732,9 @@ size_t memcpy_to_iter_csum(void *iter_to, size_t progress, size_t len, void *from, void *priv2) { __wsum *csum = priv2; + __wsum next = csum_partial_copy_nocheck(from, iter_to, len); - *csum = csum_and_memcpy(iter_to, from + progress, len, *csum, progress); + *csum = csum_block_add(*csum, next, progress); return 0; } diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 5dbdfce2d05f..3efed86321db 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6938,8 +6938,9 @@ size_t memcpy_from_iter_csum(void *iter_from, size_t progress, size_t len, void *to, void *priv2) { __wsum *csum = priv2; + __wsum next = csum_partial_copy_nocheck(iter_from, to + progress, len); - *csum = csum_and_memcpy(to + progress, iter_from, len, *csum, progress); + *csum = csum_block_add(*csum, next, progress); return 0; } From patchwork Wed Sep 13 16:56:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139091 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp85479vqb; Wed, 13 Sep 2023 14:04:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH31+ijUDRe1UlZCL0TZcFo77Qt3cuSirK6EW78tGpsOPlJL8NeOW5HJ0jNAwMMWntNVi3V X-Received: by 2002:a05:6870:b487:b0:1b0:b13:c16 with SMTP id y7-20020a056870b48700b001b00b130c16mr4633109oap.2.1694639086020; Wed, 13 Sep 2023 14:04:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694639085; cv=none; d=google.com; s=arc-20160816; b=OrtYRAYGqSHg2Kri5qjzLNGBth4aCW1EjLQbEP3necD6j5zq/sidtIb0CDPl0QSDKz ybs08pbpixc9psoIlc3gjWNGSAzQqYs0wmfyGc5yKolvZJR7XhH1b42l5dxriThW5ES4 Z9Jra+5Tf6FswgqRGtp8Mv+svJ6CQ5P3w/ddwtTAOAhXzUth3ICrJ2OL6dhzr6FCiivA 0Ew/t3uWptJelVFEExwRtVPMKw5pnVIw3KpGOKeK1EARA21AuPRG5LkFL/6sv4cbe/C0 aAQzbJUIoACuaB+1dQL5HISaCno7U4fYzKDzIfzXb5WYfQNOX2BooiuqRdA4UoUkoqnX 8a9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OEnm3txfdKWMLGG8rrdFEcQBV2WWgkp8Kq4P7DhBB1I=; fh=D00b+ExO44XKLceYW8cVdIEOVtr2e1iL7r354EPHkWw=; b=qGZwrFSmKz6Gut+VaTlwrj317HxtdFQMQHPm1TMCFAXs9Him75CfeRDJ2iUUHsgDUZ nRhsAkfxcaW/lHEXBYV1hsrrhjMPe/dg+mpYh/jAt0qKjzvkx0GN7x191aSj7eWaHiax KVcZjrgNeM1UDoRIjFYtCRLu/CAG8lnAw9bQL/RcSg/dgsYsWQBXsbL6YFrXFy8ymIL8 TlxqlJWsmuNJz+G0i9Zr081BvIgAD4wTsNVrJNaUH75J14GvCeCyYX/xqpk4djuLhebY HJwJL3v2MfJAT1U2VJhK5ZL/p1th/F5s0CXlMy0JhPnw6sqrr9oO10VcG993HRnEtbAZ HvGQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KAOSma6p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id f20-20020a63f114000000b0056fed6cf408si10664660pgi.602.2023.09.13.14.04.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 14:04:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=KAOSma6p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 1270980BEDC6; Wed, 13 Sep 2023 09:59:20 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230458AbjIMQ6y (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230381AbjIMQ6T (ORCPT ); Wed, 13 Sep 2023 12:58:19 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 80E151FC0 for ; Wed, 13 Sep 2023 09:57:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OEnm3txfdKWMLGG8rrdFEcQBV2WWgkp8Kq4P7DhBB1I=; b=KAOSma6pM5lmy2GZIiOA5owQ26thITzEZfnHcNb/4i97I4hymihXQtNu/E/wF0HRTzjd9p VVp8E8x7ZdLxERI9jRo3K82fLEHxiEyUG+7e19JuX9HH+a7nWbRSxHvfM9HSjhL6YN1zl9 eIYHzyFEKNO7Nv+6UwDmvWeC3LnHrec= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-60-yGDwbzOOMhetGn8REiVGKQ-1; Wed, 13 Sep 2023 12:57:21 -0400 X-MC-Unique: yGDwbzOOMhetGn8REiVGKQ-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 32F07185A79B; Wed, 13 Sep 2023 16:57:20 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D8792026D4B; Wed, 13 Sep 2023 16:57:18 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 11/13] iov_iter, net: Merge csum_and_copy_from_iter{,_full}() together Date: Wed, 13 Sep 2023 17:56:46 +0100 Message-ID: <20230913165648.2570623-12-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:59:20 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776957874297403438 X-GMAIL-MSGID: 1776957874297403438 Move csum_and_copy_from_iter_full() out of line and then merge csum_and_copy_from_iter() into its only caller. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/skbuff.h | 19 ++----------------- net/core/datagram.c | 5 +++++ net/core/skbuff.c | 20 +++++++++++++------- 3 files changed, 20 insertions(+), 24 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index c81ef5d76953..be402f55f6d6 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -3679,23 +3679,8 @@ static inline int __must_check skb_put_padto(struct sk_buff *skb, unsigned int l return __skb_put_padto(skb, len, true); } -struct csum_state { - __wsum csum; - size_t off; -}; - -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i); - -static __always_inline __must_check -bool csum_and_copy_from_iter_full(void *addr, size_t bytes, - __wsum *csum, struct iov_iter *i) -{ - size_t copied = csum_and_copy_from_iter(addr, bytes, csum, i); - if (likely(copied == bytes)) - return true; - iov_iter_revert(i, copied); - return false; -} +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, __wsum *csum, struct iov_iter *i) + __must_check; static inline int skb_add_data(struct sk_buff *skb, struct iov_iter *from, int copy) diff --git a/net/core/datagram.c b/net/core/datagram.c index 452620dd41e4..722311eeee18 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -738,6 +738,11 @@ size_t memcpy_to_iter_csum(void *iter_to, size_t progress, return 0; } +struct csum_state { + __wsum csum; + size_t off; +}; + static size_t csum_and_copy_to_iter(const void *addr, size_t bytes, void *_csstate, struct iov_iter *i) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 3efed86321db..2bfa6a7ba244 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6955,13 +6955,19 @@ size_t copy_from_user_iter_csum(void __user *iter_from, size_t progress, return next ? 0 : len; } -size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, - struct iov_iter *i) +bool csum_and_copy_from_iter_full(void *addr, size_t bytes, + __wsum *csum, struct iov_iter *i) { + size_t copied; + if (WARN_ON_ONCE(!i->data_source)) - return 0; - return iterate_and_advance2(i, bytes, addr, csum, - copy_from_user_iter_csum, - memcpy_from_iter_csum); + return false; + copied = iterate_and_advance2(i, bytes, addr, csum, + copy_from_user_iter_csum, + memcpy_from_iter_csum); + if (likely(copied == bytes)) + return true; + iov_iter_revert(i, copied); + return false; } -EXPORT_SYMBOL(csum_and_copy_from_iter); +EXPORT_SYMBOL(csum_and_copy_from_iter_full); From patchwork Wed Sep 13 16:56:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:a8d:b0:3f2:4152:657d with SMTP id gr13csp64673vqb; Wed, 13 Sep 2023 13:19:48 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEkE5gGD/0xhtcAFpmAzettFyB7VwAyX8/9VHeNlXqKPB7seuUUIFW5j2+pBI7eVo4/qK4d X-Received: by 2002:a17:902:ec85:b0:1c3:308b:ecb9 with SMTP id x5-20020a170902ec8500b001c3308becb9mr4428509plg.11.1694636388463; Wed, 13 Sep 2023 13:19:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694636388; cv=none; d=google.com; s=arc-20160816; b=nqpMHu6SuAYIo7nHkSoalKX0A1FG68diAHDuNrKu3w218uLS14EObLdBer5UbK2udD fthlPrtkkzu1hHu1SsWm+d+lSn4FFPLxTxmeE9/Wl79kimXk/06pKZJ3aXdtMQfg5vLK ZVDIHpTNqmnQTOy+g1VTkpSqEYOYTCFdHUOhQET0GXyj+0RYrLKJeF4FtyzFEj5WxXzG ahuTeG2oFx9Jm67KYfqO7N0tfzN1UCryRW40M/4wSlL447XwL5oGo+nIGT7VDBTCRWqI 5o0Q5PbBxMlTPyF3kXJzKgKMbOHYu6w0LQ3aJmT7kBe4azad27BEe8kzKNSNO+sUPWEF X8uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=c2st9mzpODW5EfnFYy73QjLkmO48eOaJEcOSoL9qczM=; fh=D00b+ExO44XKLceYW8cVdIEOVtr2e1iL7r354EPHkWw=; b=OLRt8UXE7T8ksn3XU3ChXvoTAxzEiOxgUBmC7+PxqUwoT4bo46Yak9jXD6B1tCfYot WlKHxhkSRn6gJ6T7xs0AYB2YnOTAVrAuqqlrUK01PGQ+u6aaXukWkhuJlO+NxRpYcuVE MVh32u1GEXl6t5AFB1bcXF2ajJVK+12whiVM3unXGOoVcMUxi5aWEKymEdeJlv1m6TMj TC/BlaBdvyDh+xYi4LlnW/ZsTX9VU/4kiT44JbuE69cg9PG2KM3RIe04pCXNYcU24pSK oGgu6dxrGJS6NKcFkawWzNe9Ixb+sMii5CAttx7NAfC7Vu5HmDg8DVZ07DfOd+O170eG rzoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jTPv8luH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id jx5-20020a170903138500b001ab1ba2572csi46198plb.240.2023.09.13.13.19.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 13:19:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jTPv8luH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id C21618246E1C; Wed, 13 Sep 2023 09:59:31 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231370AbjIMQ67 (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231326AbjIMQ6U (ORCPT ); Wed, 13 Sep 2023 12:58:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id EA9231BFE for ; Wed, 13 Sep 2023 09:57:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624249; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c2st9mzpODW5EfnFYy73QjLkmO48eOaJEcOSoL9qczM=; b=jTPv8luHErXXN8O+zfc9gdL2+9jP/3NiBXnbclwOHr01pbGPTiLkpGFLA464WMon6hoVee dJCWzxhVqAeicj3yJU/nWVfmvge0aaW13XQvAD3jQzzCDfCvrKU+rTThYAXpKg+jkLF3d1 7zOnqo5QcZBDzGuHHDbMksppNz1r3QU= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-628-Z0pMlrrIOVKNFRYw1td4xA-1; Wed, 13 Sep 2023 12:57:23 -0400 X-MC-Unique: Z0pMlrrIOVKNFRYw1td4xA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D46F2101FAA0; Wed, 13 Sep 2023 16:57:22 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id CAFFF40C6EA8; Wed, 13 Sep 2023 16:57:20 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 12/13] iov_iter, net: Move hash_and_copy_to_iter() to net/ Date: Wed, 13 Sep 2023 17:56:47 +0100 Message-ID: <20230913165648.2570623-13-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:59:31 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776955046060625509 X-GMAIL-MSGID: 1776955046060625509 Move hash_and_copy_to_iter() to be with its only caller in networking code. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- include/linux/uio.h | 3 --- lib/iov_iter.c | 20 -------------------- net/core/datagram.c | 19 +++++++++++++++++++ 3 files changed, 19 insertions(+), 23 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index c335b95626af..6bb00645fe46 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -326,9 +326,6 @@ iov_iter_npages_cap(struct iov_iter *i, int maxpages, size_t max_bytes) return npages; } -size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, - struct iov_iter *i); - struct iovec *iovec_from_user(const struct iovec __user *uvector, unsigned long nr_segs, unsigned long fast_segs, struct iovec *fast_iov, bool compat); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 5b2d053f057f..77520834ebe4 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1,5 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-only -#include #include #include #include @@ -1071,25 +1070,6 @@ ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, } EXPORT_SYMBOL(iov_iter_get_pages_alloc2); -size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, - struct iov_iter *i) -{ -#ifdef CONFIG_CRYPTO_HASH - struct ahash_request *hash = hashp; - struct scatterlist sg; - size_t copied; - - copied = copy_to_iter(addr, bytes, i); - sg_init_one(&sg, addr, copied); - ahash_request_set_crypt(hash, &sg, NULL, copied); - crypto_ahash_update(hash); - return copied; -#else - return 0; -#endif -} -EXPORT_SYMBOL(hash_and_copy_to_iter); - static int iov_npages(const struct iov_iter *i, int maxpages) { size_t skip = i->iov_offset, size = i->count; diff --git a/net/core/datagram.c b/net/core/datagram.c index 722311eeee18..103d46fa0eeb 100644 --- a/net/core/datagram.c +++ b/net/core/datagram.c @@ -61,6 +61,7 @@ #include #include #include +#include /* * Is a socket 'connection oriented' ? @@ -489,6 +490,24 @@ static int __skb_datagram_iter(const struct sk_buff *skb, int offset, return 0; } +static size_t hash_and_copy_to_iter(const void *addr, size_t bytes, void *hashp, + struct iov_iter *i) +{ +#ifdef CONFIG_CRYPTO_HASH + struct ahash_request *hash = hashp; + struct scatterlist sg; + size_t copied; + + copied = copy_to_iter(addr, bytes, i); + sg_init_one(&sg, addr, copied); + ahash_request_set_crypt(hash, &sg, NULL, copied); + crypto_ahash_update(hash); + return copied; +#else + return 0; +#endif +} + /** * skb_copy_and_hash_datagram_iter - Copy datagram to an iovec iterator * and update a hash. From patchwork Wed Sep 13 16:56:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Howells X-Patchwork-Id: 139253 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp68233vqi; Wed, 13 Sep 2023 19:39:52 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHqC9t2GSrcKJNMo0orNCigIWyviiQ363eWTRMLI3p/82ZaxIhZYQCSnyEg4q2XWeoGy4p0 X-Received: by 2002:a05:6a20:4403:b0:14c:512c:60d9 with SMTP id ce3-20020a056a20440300b0014c512c60d9mr4957202pzb.27.1694659192354; Wed, 13 Sep 2023 19:39:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694659192; cv=none; d=google.com; s=arc-20160816; b=mTgfYzz203HCzSwecOmmistZXpFsRMthas2YRjWi6NPN8I2QR8H9tksLaLosEyXcDN X8mqxmrC3CWjsBTL7DJgUxfeALJrJPNCXSfUrH7zVR0boaZ4+B3CqYyh0RMS5OjV7WR1 kxTv2vRCMOmNgnWf23+egBAhghgBlyfFL3OJdZkrLWkUCBzvqg0rlZqGKwFB5GInoRxN 1HGtbu82bzK2w9VDTIsV9bopcKN0/s6rB0Bj8z/AOQfbYdB88HMtJBHhHUvGLkK0dvbs +VoS8Ae3+Jt/r+P6Fh2dgtCYteXf/pS24ud/812xWlRtDQlhDyUZa/ydyj/eoSleA4LD QJKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jTW4hj6R6SH2WgBE9Xhmo5Z05cFVRaLxhGn+V3gLo/A=; fh=D00b+ExO44XKLceYW8cVdIEOVtr2e1iL7r354EPHkWw=; b=szHXVvg0mzIIa2XkKPePa4ICwpsFS42y1akb7nC4ztiR/rfxO3hbf1Rw8acE/baeKS AKNWwuSJWjkDRdm/FXZh68owwuwnzAEC2AvqJlE//W3zlJgBmJ2TpzoG6/wB4p79WuQF fBOHn+XhiR7SZIjFSu4+Khgk47b+1T/R6839VonhxjKizmW7QVhz/Py6ijB7y1JHGEj5 yqQS+R4FuAPkjftD08dGWvCB+Eu/ycwFhBPomKedF35mB7KNZnWZajC4X89HXOWg5Z+u HpX2LxwHAKInRjwtbdzm3T0h8CjqXapuN9mlzXla/MmHWMheXU1edPB6L8eVfXgVyVuV awGQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=XXRXzFeA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id w9-20020a17090a8a0900b0026d415dcbc8si2842657pjn.165.2023.09.13.19.39.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Sep 2023 19:39:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=XXRXzFeA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id A21F681F54E9; Wed, 13 Sep 2023 09:59:19 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230381AbjIMQ64 (ORCPT + 35 others); Wed, 13 Sep 2023 12:58:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231340AbjIMQ6U (ORCPT ); Wed, 13 Sep 2023 12:58:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 28AFF1FD6 for ; Wed, 13 Sep 2023 09:57:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694624252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jTW4hj6R6SH2WgBE9Xhmo5Z05cFVRaLxhGn+V3gLo/A=; b=XXRXzFeAIzUeppFvJyejJEB4fFd5x4Dv8x9jCk3u4zx1TdPaEfOsJshXT9eNcO+5GipQVL LFocez3lQj3tznAmzowfIX0eiQB5tZssP+DYDTo2TohP+y1yGbSmHp2bEQlgZ1tu8Fj8hg ZDq4MgWsX3XJx1B5lJhRnZHfcfH2vnQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-583-kw6qakgRMmKfBnOo7mji8A-1; Wed, 13 Sep 2023 12:57:27 -0400 X-MC-Unique: kw6qakgRMmKfBnOo7mji8A-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0D8F6855321; Wed, 13 Sep 2023 16:57:26 +0000 (UTC) Received: from warthog.procyon.org.com (unknown [10.42.28.216]) by smtp.corp.redhat.com (Postfix) with ESMTP id C823740C6EA8; Wed, 13 Sep 2023 16:57:23 +0000 (UTC) From: David Howells To: Al Viro , Linus Torvalds Cc: David Howells , Jens Axboe , Christoph Hellwig , Christian Brauner , David Laight , Matthew Wilcox , Jeff Layton , linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Subject: [PATCH v4 13/13] iov_iter: Create a fake device to allow iov_iter testing/benchmarking Date: Wed, 13 Sep 2023 17:56:48 +0100 Message-ID: <20230913165648.2570623-14-dhowells@redhat.com> In-Reply-To: <20230913165648.2570623-1-dhowells@redhat.com> References: <20230913165648.2570623-1-dhowells@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Wed, 13 Sep 2023 09:59:19 -0700 (PDT) X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1776978957702399444 X-GMAIL-MSGID: 1776978957702399444 Create a fake device to allow testing and benchmarking of UBUF and IOVEC iterators. /dev/iov-test is created and can be driven with pwritev() in which case it copies everything to a sink page and it can be written with preadv() in which case it copies repeatedly from a patterned page. The time taken is logged with tracepoints. This can be driven by something like: echo 1 >/sys/kernel/debug/tracing/events/iov_test/enable cmd="r -b 1M -V 256 0 256M"; xfs_io -c "open /dev/iov-test" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" cmd="w -b 1M -V 256 0 256M"; xfs_io -c "open /dev/iov-test" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" \ -c "$cmd" -c "$cmd" -c "$cmd" -c "$cmd" showing something like: ...: iov_test_read: size=10000000 done=10000000 ty=1 nr=256 dur=27653 ...: iov_test_write: size=10000000 done=10000000 ty=1 nr=256 dur=31792 in the trace log. Signed-off-by: David Howells cc: Alexander Viro cc: Jens Axboe cc: Christoph Hellwig cc: Christian Brauner cc: Matthew Wilcox cc: Linus Torvalds cc: David Laight cc: "David S. Miller" cc: Eric Dumazet cc: Jakub Kicinski cc: Paolo Abeni cc: linux-block@vger.kernel.org cc: linux-fsdevel@vger.kernel.org cc: linux-mm@kvack.org cc: netdev@vger.kernel.org --- lib/Kconfig.debug | 8 +++ lib/Makefile | 1 + lib/test_iov_iter.c | 134 ++++++++++++++++++++++++++++++++++++++ lib/test_iov_iter_trace.h | 80 +++++++++++++++++++++++ 4 files changed, 223 insertions(+) create mode 100644 lib/test_iov_iter.c create mode 100644 lib/test_iov_iter_trace.h diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index fa307f93fa2e..cf8392c51344 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2524,6 +2524,14 @@ config TEST_SYSCTL If unsure, say N. +config TEST_IOV_ITER + tristate "iov_iter test driver" + help + This creates a misc device that can be used as a way to test various + I/O iterator functions through the use of readv/writev and ioctl. + + If unsure, say N. + config BITFIELD_KUNIT tristate "KUnit test bitfield functions at runtime" if !KUNIT_ALL_TESTS depends on KUNIT diff --git a/lib/Makefile b/lib/Makefile index 740109b6e2c8..f6419544a749 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -65,6 +65,7 @@ CFLAGS_test_bitops.o += -Werror obj-$(CONFIG_CPUMASK_KUNIT_TEST) += cpumask_kunit.o obj-$(CONFIG_TEST_SYSCTL) += test_sysctl.o obj-$(CONFIG_TEST_IOV_ITER) += kunit_iov_iter.o +obj-$(CONFIG_TEST_IOV_ITER) += test_iov_iter.o obj-$(CONFIG_HASH_KUNIT_TEST) += test_hash.o obj-$(CONFIG_TEST_IDA) += test_ida.o obj-$(CONFIG_TEST_UBSAN) += test_ubsan.o diff --git a/lib/test_iov_iter.c b/lib/test_iov_iter.c new file mode 100644 index 000000000000..afa70647dbde --- /dev/null +++ b/lib/test_iov_iter.c @@ -0,0 +1,134 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* I/O iterator testing device. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ + +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include +#include +#include +#define CREATE_TRACE_POINTS +#include "test_iov_iter_trace.h" + +MODULE_DESCRIPTION("iov_iter testing"); +MODULE_AUTHOR("David Howells "); +MODULE_LICENSE("GPL"); + +static ssize_t iov_test_read_iter(struct kiocb *iocb, struct iov_iter *iter) +{ + struct folio *folio = iocb->ki_filp->private_data; + unsigned int nr_segs = iter->nr_segs; + size_t size = iov_iter_count(iter), fsize = folio_size(folio); + size_t copied = 0, offset = 0, i; + ktime_t a, b; + u8 *p; + + /* Pattern the buffer */ + p = kmap_local_folio(folio, 0); + for (i = 0; i < folio_size(folio); i++) + p[i] = i & 0xff; + kunmap_local(p); + + a = ktime_get_real(); + while (iov_iter_count(iter)) { + size_t done, part = min(iov_iter_count(iter), fsize - offset); + + done = copy_folio_to_iter(folio, offset, part, iter); + if (done == 0) + break; + copied += done; + offset = (offset + done) & (fsize - 1); + } + + b = ktime_get_real(); + trace_iov_test_read(size, copied, iov_iter_type(iter), nr_segs, + ktime_to_us(ktime_sub(b, a))); + return copied; +} + +static ssize_t iov_test_write_iter(struct kiocb *iocb, struct iov_iter *iter) +{ + struct folio *folio = iocb->ki_filp->private_data; + unsigned int nr_segs = iter->nr_segs; + size_t size = iov_iter_count(iter), fsize = folio_size(folio); + size_t copied = 0, offset = 0; + ktime_t a = ktime_get_real(), b; + + while (iov_iter_count(iter)) { + size_t done, part = min(iov_iter_count(iter), fsize - offset); + + done = copy_page_from_iter(folio_page(folio, 0), offset, part, iter); + if (done == 0) + break; + copied += done; + offset = (offset + done) & (fsize - 1); + } + + b = ktime_get_real(); + trace_iov_test_write(size, copied, iov_iter_type(iter), nr_segs, + ktime_to_us(ktime_sub(b, a))); + return copied; +} + +static int iov_test_open(struct inode *inode, struct file *file) +{ + struct folio *folio; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + folio = folio_alloc(GFP_KERNEL, 0); + if (!folio) + return -ENOMEM; + file->private_data = folio; + return 0; +} + +static int iov_test_release(struct inode *inode, struct file *file) +{ + struct folio *folio = file->private_data; + + folio_put(folio); + return 0; +} + +static const struct file_operations iov_test_fops = { + .owner = THIS_MODULE, + .open = iov_test_open, + .release = iov_test_release, + .read_iter = iov_test_read_iter, + .write_iter = iov_test_write_iter, + .splice_read = copy_splice_read, + .llseek = noop_llseek, +}; + +static struct miscdevice iov_test_dev = { + .minor = MISC_DYNAMIC_MINOR, + .name = "iov-test", + .fops = &iov_test_fops, +}; + +static int __init iov_test_init(void) +{ + int ret; + + ret = misc_register(&iov_test_dev); + if (ret < 0) + return ret; + pr_info("Loaded\n"); + return 0; +} +module_init(iov_test_init); + +static void __exit iov_test_exit(void) +{ + pr_info("Unloading\n"); + + misc_deregister(&iov_test_dev); +} +module_exit(iov_test_exit); diff --git a/lib/test_iov_iter_trace.h b/lib/test_iov_iter_trace.h new file mode 100644 index 000000000000..b99cade5d004 --- /dev/null +++ b/lib/test_iov_iter_trace.h @@ -0,0 +1,80 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* I/O iterator testing device. + * + * Copyright (C) 2023 Red Hat, Inc. All Rights Reserved. + * Written by David Howells (dhowells@redhat.com) + */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM iov_test + +#if !defined(_IOV_TEST_TRACE_H) || defined(TRACE_HEADER_MULTI_READ) +#define _IOV_TEST_TRACE_H + +#include + +TRACE_EVENT(iov_test_read, + TP_PROTO(size_t size, size_t done, enum iter_type type, + unsigned int nr_segs, u64 duration), + + TP_ARGS(size, done, type, nr_segs, duration), + + TP_STRUCT__entry( + __field(size_t, size) + __field(size_t, done) + __field(enum iter_type, type) + __field(unsigned int, nr_segs) + __field(u64, duration) + ), + + TP_fast_assign( + __entry->size = size; + __entry->done = done; + __entry->type = type; + __entry->nr_segs = nr_segs; + __entry->duration = duration; + ), + + TP_printk("size=%zx done=%zx ty=%u nr=%u dur=%llu", + __entry->size, + __entry->done, + __entry->type, + __entry->nr_segs, + __entry->duration) + ); + +TRACE_EVENT(iov_test_write, + TP_PROTO(size_t size, size_t done, enum iter_type type, + unsigned int nr_segs, u64 duration), + + TP_ARGS(size, done, type, nr_segs, duration), + + TP_STRUCT__entry( + __field(size_t, size) + __field(size_t, done) + __field(enum iter_type, type) + __field(unsigned int, nr_segs) + __field(u64, duration) + ), + + TP_fast_assign( + __entry->size = size; + __entry->done = done; + __entry->type = type; + __entry->nr_segs = nr_segs; + __entry->duration = duration; + ), + + TP_printk("size=%zx done=%zx ty=%u nr=%u dur=%llu", + __entry->size, + __entry->done, + __entry->type, + __entry->nr_segs, + __entry->duration) + ); + +#endif /* _IOV_TEST_TRACE_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH . +#define TRACE_INCLUDE_FILE test_iov_iter_trace +#include