From patchwork Wed Feb 22 14:36:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 60556 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp613941wrd; Wed, 22 Feb 2023 06:42:19 -0800 (PST) X-Google-Smtp-Source: AK7set9uoqfOGoFrbhCbU0s5pb2+NilUw/dcUNk9sqxuq1l4xkUGHzDZJcb1FScQ4zz84PAx7sXd X-Received: by 2002:aa7:cfd1:0:b0:4ab:2555:408a with SMTP id r17-20020aa7cfd1000000b004ab2555408amr9145037edy.22.1677076939373; Wed, 22 Feb 2023 06:42:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677076939; cv=none; d=google.com; s=arc-20160816; b=JTml7KRRGv7D2B6aqL5MXCCKeQVl5bEEcanJEXP3C3BAiu1JvyAT2+9ORRlbqKJQGD 0KrZyM9VHSKvW4yNCKf6++bre1AU43ucB1PI9wth6/dgebIIZBoXjQofEIP79dayP+KY YYCR0HRCHTh0ebuVKbYuTxpdp8kX6LrEZajxScCQt1nryCynrE566len9g8AETci+qq1 8Ch9ZZp9ELBy33tJq8Ln8axc0IcM0kwXsEfrPY6WLCD6yr0bYQebyiItmHExXlYE1/Rt UkxF3lMG9Q/7K/YjYyh5/EPYLo9Bj1orvB/eZNK/p94ORGlxmsk0n9/ILNVCghTLfpZX sr1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PvupsfzCcOKXDRykNFlbD3M522JsoKtEmFaAw+ESIZ4=; b=xHEthKgMPr2yhOiSGdjHkZcKndIxM8l8XWkpgv9zrqRO3JiLx7phoWAPaYfzmdbxjl Z5aZsbqnuTc7lsknrVqK92w7rIHy6BYIGQxNmrA3MsA/GWYXpLS/358LbfS8uejvrYE6 zC75g9HHMe8m7H2koWfP9RPd8L8KsVDhu4qHNRO6doiQnr3spoX5QNu8LeEBk/BQ7/fH sfXC1gz57zl75cTUBeA/R615XjK6YMMYkiLdKVyQuY6a8G1LzP010K9Q6YrBm1vImrut dNtholB0xvJAQRsmKqvzhDKW0pO/nTRKIRYwLKY8zJ3V6cIqfzP0QtrPpV0DnH9xsIed IFxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=kjEvxcRC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d8-20020aa7c1c8000000b004acc28d580bsi22911450edp.371.2023.02.22.06.41.56; Wed, 22 Feb 2023 06:42:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=kjEvxcRC; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231961AbjBVOjY (ORCPT + 99 others); Wed, 22 Feb 2023 09:39:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231351AbjBVOjU (ORCPT ); Wed, 22 Feb 2023 09:39:20 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E05837544; Wed, 22 Feb 2023 06:39:19 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id 6so7551409wrb.11; Wed, 22 Feb 2023 06:39:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PvupsfzCcOKXDRykNFlbD3M522JsoKtEmFaAw+ESIZ4=; b=kjEvxcRC92xwc7L6WYECbkxta2Ix9i8Ccqeo2j6WclYyqyqGOnOmAGeq5fFozV7/GA zjFFZf8Mn+2FHNLk7wvq8agSoXbjw0XLLp/fHEqmgLCYXbY/KworMugVrnXT+y1x8zGn +k5FVGgjdT83dqVY9yFf1e2soQ5uMgWwFNFUS9lfejLuIQh7ok2/BwZjI1oesGZlpyKs U4hDz3rOTWp9NzsAb6xaRYXIlpQBajp0MaCYo1yXMXHQw592KfYkYD7q2j9tU6IVGKV3 C8XBZvVXSs9xSwhrACo0XNVBbASbSld+Xoaldlp7TAOYSqV8daMnKO78611VDsw0q2ga /big== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PvupsfzCcOKXDRykNFlbD3M522JsoKtEmFaAw+ESIZ4=; b=6bFoaIiVBI8Qrwy5jxJtqB76UwiUSP5VqgYxtt+cqPwU1Q5QO8XYykkh89t/XKraCx pbti8QygJWjMKwYZb+h3XrmLbihzP7PsffNd8EmKTNml8JNHv35GfTd+bVDr4ynYE/1a qacvab0dBLuFMt4KymN3HLt7BbgageS2SkWQ8OU8wCTw2ebssyVS55ms/z07qXUXx/oB BeOybzlhVqRrfP3tLTL/7gkOWFyntWbvm6Yg6axq9BFRCpIz4J9IyCN6C3Hs4pOBJI7/ 16X4iDVvQGZMAgpVaji1z9HwjKjN5qNEWeHFbEWBWZ/QB/0X4Aco4qh0VeEJGTty5bXn dPQQ== X-Gm-Message-State: AO0yUKXX8NKjXdClANq3TApbydFdamEg6pRcYnJ4w4rhVKpy3mv9HuOn 2iMgXnmfUl7fdYKf5T6p6dICwHGW9H8= X-Received: by 2002:adf:dcc5:0:b0:2c7:ce2:6479 with SMTP id x5-20020adfdcc5000000b002c70ce26479mr1780754wrm.40.1677076757339; Wed, 22 Feb 2023 06:39:17 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:17 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 1/4] io_uring/rsrc: disallow multi-source reg buffers Date: Wed, 22 Feb 2023 14:36:48 +0000 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758542629105368721?= X-GMAIL-MSGID: =?utf-8?q?1758542629105368721?= If two or more mappings go back to back to each other they can be passed into io_uring to be registered as a single registered buffer. That would even work if mappings came from different sources, e.g. it's possible to mix in this way anon pages and pages from shmem or hugetlb. That is not a problem but it'd rather be less prone if we forbid such mixing. Cc: Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index a59fc02de598..8d7eb1548a04 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1162,14 +1162,17 @@ struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages) pret = pin_user_pages(ubuf, nr_pages, FOLL_WRITE | FOLL_LONGTERM, pages, vmas); if (pret == nr_pages) { + struct file *file = vmas[0]->vm_file; + /* don't support file backed memory */ for (i = 0; i < nr_pages; i++) { - struct vm_area_struct *vma = vmas[i]; - - if (vma_is_shmem(vma)) + if (vmas[i]->vm_file != file) { + ret = -EINVAL; + break; + } + if (!file) continue; - if (vma->vm_file && - !is_file_hugepages(vma->vm_file)) { + if (!vma_is_shmem(vmas[i]) && !is_file_hugepages(file)) { ret = -EOPNOTSUPP; break; } From patchwork Wed Feb 22 14:36:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 60557 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp613952wrd; Wed, 22 Feb 2023 06:42:22 -0800 (PST) X-Google-Smtp-Source: AK7set+qXWVeMKs1+ZeN8JmwPsNxWji2Hm/i4HPKnyrHxJ9cxld1g3cyT5HC/UjWeFQK2Uiq4gym X-Received: by 2002:a05:6a00:1bcc:b0:5a8:a464:4fe3 with SMTP id o12-20020a056a001bcc00b005a8a4644fe3mr5356038pfw.0.1677076941923; Wed, 22 Feb 2023 06:42:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677076941; cv=none; d=google.com; s=arc-20160816; b=YnYiNz3B8Nr3Fx6wyvUc3JjACgri8gxXd5aWqUCD0FE4tNNpGp0hOwX2AtoMEPfhYL GT73fQq/gOAQ+F25zn5WbSVl//oBhwqMyR4HvIAP6BvVfzlNI9oDlY0HMaa6oIgeciYT kWWMZ3hXJkm1Hs5JIuSaZrEsd59SDadUw4IVjrjesjFzdR0u/wrvBQhe9pyqGJF1zI4O cOFKTa1oU45wGvwWh/ESEfWB8JDdynQhME8lpN2BJ0/6pD6DfVugZ5tU1BGLlrS2x6yM 71g972Jy9fFEoc3s5lbEWj6DcMKAWmuyTvSaEdCgQ+jD8eNEcozs+vcpOQOAwxyinK1f V06Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aLlCzWw1AssfjgBXmFlk8NMc2CPaZuoh5Bi4Ozm/gs0=; b=zDRRf13mE2dzvmkJEnqH478zfvU/PCwgIGupzn6923B/tbW+UVP4BBVzySJyf7frJy RglMAqphN+Nmg/sN/hIr13Pv9hoH0gav6r+gGtWW9ttN0NwTixULCqtk0dov1m5fUy/s YzML25DdB0n2yRaXFiNaZtDEeK2lmGTLWuf0/oExpptXCnHU/bCrFdQNUJxrLDBGoVKw Pl0841Bg6+2J6GTruux3jdKm82/6BbeuMxhsDA/2mmUdUJkxoFXtDkuowhbNyYAZ72Lk ayyjc3F5iUDlXGasc/vssiFlqjRoHbOVr/1bO/wXFDYBFBgehJVqMrPTxtyhGU/sswm2 pR3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="l/ttigjx"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ca27-20020a056a00419b00b0059072f087b7si8374166pfb.135.2023.02.22.06.42.09; Wed, 22 Feb 2023 06:42:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="l/ttigjx"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231984AbjBVOj1 (ORCPT + 99 others); Wed, 22 Feb 2023 09:39:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231658AbjBVOjU (ORCPT ); Wed, 22 Feb 2023 09:39:20 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A060F37B65; Wed, 22 Feb 2023 06:39:19 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id j19-20020a05600c1c1300b003e9b564fae9so664333wms.2; Wed, 22 Feb 2023 06:39:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aLlCzWw1AssfjgBXmFlk8NMc2CPaZuoh5Bi4Ozm/gs0=; b=l/ttigjxsKa3YvoIQjINVBOoNPeK2/2BaOf/7PSu4DdWgzHhBKsJ+2ouhN4H2YXXqX LF475KPpC6+x1VOtBOjoUhvEsX5sOpizb4nSzJKBP1DK4Rq/ZQSaQ8Gio2IkjZD3svXP kzdzDQlDePQKDlDzrsyY0EwMiZKmconuJ464dYSKlomNQmuGfwCtLgM2y1jhP4CAvejq cGuk5kKw10CVAXzevFhF9wMm6ykyxMuXacliu6YH4Emh9K7PNmbntQNkMwGchde7Ymyf vmJMvN0FUNqzAqBe2rbeS7mt04v+uXAXBcYtBDVk2kcTmj2V0walKtKGtiZopb98lX1f GBHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aLlCzWw1AssfjgBXmFlk8NMc2CPaZuoh5Bi4Ozm/gs0=; b=65piNaVGrBTes1XJDzoRz7hPvIeW9a6t7cXbcFs4q4cH0u+ibfoLyDVy9srDkRn4fI gJRDfCICKTyUOWbGlymLKK94Va7oGkt9ohGWS324Uie7bhblU+HR+DwMwB1G5vtsok7N zhCk6HiUyzidHsLBhQAYwkta6En65dIqOG1dL8vJsqC7yD2B6iWmK+jagiJL8cSdyoSl TbBIh6xmuEEyerbfmbh/AZFiVIVcCRtPDtG/g1JAf2dCJX8G2EYTXZqHNgRZYLSC+SsZ UBF9OSu7GqFKPsxmj2Uluq0Wfl6+uddlysZh2QFjhQDpl+v8AUSpE7Jj5pg286cED+v+ ntZg== X-Gm-Message-State: AO0yUKVDTOxtQcAEdcFyjZvmn9y59Mb8bJ52aMYXmJdKc/GN7xQgINbB n832jbIa7HOgqpafpqnufIUok88cLd0= X-Received: by 2002:a05:600c:3d8c:b0:3e0:c5e:ad78 with SMTP id bi12-20020a05600c3d8c00b003e00c5ead78mr6031015wmb.7.1677076758017; Wed, 22 Feb 2023 06:39:18 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:17 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 2/4] io_uring/rsrc: fix a comment in io_import_fixed() Date: Wed, 22 Feb 2023 14:36:49 +0000 Message-Id: <1445461129902f506c46645d8659f78bd5b35008.1677041932.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758542631588499518?= X-GMAIL-MSGID: =?utf-8?q?1758542631588499518?= io_import_fixed() supports offsets, but "may not" means the opposite. Replace it with "might not" so the comments rather speaks about possible cases. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 8d7eb1548a04..53845e496881 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1338,7 +1338,7 @@ int io_import_fixed(int ddir, struct iov_iter *iter, return -EFAULT; /* - * May not be a start of buffer, set size appropriately + * Might not be a start of buffer, set size appropriately * and advance us to the beginning. */ offset = buf_addr - imu->ubuf; From patchwork Wed Feb 22 14:36:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 60559 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp614213wrd; Wed, 22 Feb 2023 06:42:59 -0800 (PST) X-Google-Smtp-Source: AK7set+O7ybSgV9b53RsBZaONGyi4G/fXMXdcUGNOQYpM/i5w/QndzGpZve7GG5x9MFkzk2rCxUh X-Received: by 2002:a17:906:3683:b0:7c0:e30a:d3e5 with SMTP id a3-20020a170906368300b007c0e30ad3e5mr15620847ejc.18.1677076979695; Wed, 22 Feb 2023 06:42:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677076979; cv=none; d=google.com; s=arc-20160816; b=WEr5a/2Zm8loEyQu3MezOVuz9QR+Pd6vT2HFoCHcYr/uC8iDX/9YKGRXFLPQXPvy5s 1B7IaZtw+naZ37+/ojwyUIEW9Q5IW+X8nQ8Koje/vrlMaA5ptt8DLnlANR16lB+IDBHN UgTw6bCb1fpGxVJtJNYwS1g4ohYF2FkbGerVixaJqGa57W9qZPvCiXsZATCb9E7m3TkR d24QyDlEpXvZVWLqANkmfdW+cedWypK5oy/QzMi4mWryaNBXm7dKMjI/RYYtoipLmhaa 7Zz7i7K4BA3vb+MpSusGc/SmfTFg62B7Yc+6+5Oc9spHJ+4szHxm6Ahqy1f5Hu/NOrVq ne5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=M22oHJnuucBcT6iKB9lxW+hMSBsBNFHqZcHQOuzIIKc=; b=zgcWFJVl+rwv8URXNVHOQMXf8OzLHTp8A7LlYLZQ27A9cPC7k51YaUXgkZ6atuw6R9 oi3VBRb6YPyqzj2omzI3yMIAQl3nDVI7TBVFLP0j7QbLaQQ4ZSRc6eqUXY7SuZBJpyEh 0D0FaUfnDsHCdlu889CN9US246KxD7v5kCaFMx5mEoI0MNc5/kHYeER48vYrPs1iIH0D gYnRsgNVMx3vqbVbBO2sxpEjN3f0ehsXuqWC93kVIR/sF53Rg3qyZfVWDO68/e8K66Cs 8XQhJxMGTG+jMoSKi2kz2EQEtFfvzAWrS5Vp3iybXIY+Sw4HEh0gWzwgUAor0kc7JxnW sVFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ZykuQEKl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id cj20-20020a17090715d400b008c03cbdabb3si14361664ejc.477.2023.02.22.06.42.36; Wed, 22 Feb 2023 06:42:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=ZykuQEKl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232062AbjBVOja (ORCPT + 99 others); Wed, 22 Feb 2023 09:39:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47548 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231862AbjBVOjV (ORCPT ); Wed, 22 Feb 2023 09:39:21 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93CE838030; Wed, 22 Feb 2023 06:39:20 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id t15so8067841wrz.7; Wed, 22 Feb 2023 06:39:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M22oHJnuucBcT6iKB9lxW+hMSBsBNFHqZcHQOuzIIKc=; b=ZykuQEKlBFq6ujC0ObdSGHbCSm/snEq4aV9KxuboP9EFXzEeY1U79ozyKgjFHHthmf 0hFMSfi6beUjiDTigg8SpQXuJDjkABt+JdL4wJEEh9eQIHVkcc7S47IJNmsM7P+Pg4n4 QDnxFwrpNzwPIKIEdFmw2BPWED5431FYF9tnIKllCgamReeOon8C+8hqImeyB5Of9dGK dM2Fu67Wb6NRVJGP3dLaWscF+0tEdqS0eebfY2uHxLgEdUahj9V19t+7P759MWEYrAqs ApjBt7YA37iJGalEGm2OI25QtjLMrC/rDcSSANiRzXnR4ku0lCFODiqWfW+u+udurx8+ GjAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M22oHJnuucBcT6iKB9lxW+hMSBsBNFHqZcHQOuzIIKc=; b=x56e1uctdnxWxLNmlu4ChzRbT/OugZLTyiXrVvPHGK8gF8Y1SDUPGi0/qY7ZYIEyoQ IPaLwIoCHDGqlsvbF4I/Xe+Gpg5jfeJ+Qz3LPXLAgX0b13KThzWhkD2w5jf3UUqJCaPl zj6GsXa7jcYUiQ+E8K4bs0Og1wVMJYj6EkYIKJJp2H4JSZVfuDUBCMMC/WgB0Im+M28r Q1fGn6sv5UVM4oTARBV6NavGKrwFCQvL/x3Xt+sCG+RwKfbIkW/gpJWNw+2XVISXGYxM i4mqwYOOf/YXVeTyxqzB0lrizKJLckB6yfc/vgpT8epetSAPNUkQ0ia5Xd4+QBei8ppP RBaw== X-Gm-Message-State: AO0yUKUiJTOlVz/4T4C5c6keldw5IkhWuXavLzC9yy3IWJN8Y5JSqC8c OD+yLskNd4GFmoU+QDtJ3F4Ip9bQkSI= X-Received: by 2002:a5d:45cb:0:b0:2c5:4f04:c50d with SMTP id b11-20020a5d45cb000000b002c54f04c50dmr7554750wrs.48.1677076758850; Wed, 22 Feb 2023 06:39:18 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:18 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 3/4] io_uring/rsrc: optimise single entry advance Date: Wed, 22 Feb 2023 14:36:50 +0000 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758542671046848531?= X-GMAIL-MSGID: =?utf-8?q?1758542671046848531?= Iterating within the first bvec entry should be essentially free, but we use iov_iter_advance() for that, which shows up in benchmark profiles taking up to 0.5% of CPU. Replace it with a hand coded version. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 53845e496881..ebbd2cea7582 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1364,7 +1364,10 @@ int io_import_fixed(int ddir, struct iov_iter *iter, const struct bio_vec *bvec = imu->bvec; if (offset <= bvec->bv_len) { - iov_iter_advance(iter, offset); + iter->bvec = bvec; + iter->nr_segs = bvec->bv_len; + iter->count -= offset; + iter->iov_offset = offset; } else { unsigned long seg_skip; From patchwork Wed Feb 22 14:36:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 60558 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp614017wrd; Wed, 22 Feb 2023 06:42:31 -0800 (PST) X-Google-Smtp-Source: AK7set/JUmc9FkUDUwLaza7UV5Qv5QCpAm0jEWZj17U5cv6oZDCENOGObHD8A584Ul3/PJ8/wdjC X-Received: by 2002:a17:90b:164f:b0:234:944d:dd6b with SMTP id il15-20020a17090b164f00b00234944ddd6bmr9437434pjb.12.1677076951631; Wed, 22 Feb 2023 06:42:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677076951; cv=none; d=google.com; s=arc-20160816; b=OsdMAG2oiliRo7UR/I8+IqPsXL5TDj6lG3U3PuB0HfxiYn6b/ashFttcH3MWVx7vy/ qIlIY/wgxRlsIVcB8r8fr7WC+/THW7Rv50q3GbJurSBoD4DbJEPd6W5RyrHGOmPPZ211 I64eeO9LGxuR0w3P9fetLIDTpA3hpi3lNZzRlqETCDS2yQYuS0q3Pd8QXGxOERbspTMO XNfDJHWscRRokrQTnguQ4cKPjgqQ/uxA873Nice85SeqYCHxaYG3qCrH/y+2cXb9IojM N0rS8zk/b6YDuuMJxH4tmz37HmYXDZIxRrjxI+RtJs+6cVOdeZ02SXKSIZJ46ZMBTjYn Q+hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Yqisb0p/1TgnFfChFg4hQVIMzPMIgaeBy+4V1ZQduJ4=; b=ucQYPWdOcKzpg+blWcFVEoIUso7PaILUCutRYlWJHQThu0TlX112ZXIrmQUSU9sXtC XALmcr6qR8WWw6amY62YLk/BLZ1dETNEgzeklYiDjqWihgB6gnaUIFdtniPxusQIN3FL h4FO35hSH3TWDSnpJbUHdBqOcrEd9vENJHHWH3oKbu8NRmXmVWTJHm+Jz+UXPp4YxJ/d B0aatFpnanzykHco7MzxmXamHiKEmVxNtBSfneJz1ygautr3POGO8VDks0K3qM+X2hOV g96QjeCD1ExanHPKOovpyjFqyGuGS+yBPZxa3dgr6bFT1ZIaevTkqncHtOfR6Sm28ET0 2a7A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Fg0LA5G3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds2-20020a17090b08c200b002369a258df1si6786733pjb.168.2023.02.22.06.42.18; Wed, 22 Feb 2023 06:42:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=Fg0LA5G3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230246AbjBVOjd (ORCPT + 99 others); Wed, 22 Feb 2023 09:39:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231600AbjBVOjW (ORCPT ); Wed, 22 Feb 2023 09:39:22 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C8183803E; Wed, 22 Feb 2023 06:39:21 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id r7so7797051wrz.6; Wed, 22 Feb 2023 06:39:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Yqisb0p/1TgnFfChFg4hQVIMzPMIgaeBy+4V1ZQduJ4=; b=Fg0LA5G3eErF0j0+tGhZ9F2ZwjIou9MWYHnsZZbtNv/lNRl+rkgFrA2Tc9infoD9e5 +15HKDFakYlv9lZiQCsj1qmClc8lWeS/HkiT5ZaWJnpgBN/HqJyC2j3WAi5d5EA4oVrc sAMI1q/GZVfTdR5YXHfHqkz7V1q5gfwU4mxUKz/mUa3GQvQlm9Pwjs2FMXDtvE2Pp4p+ L0IXcCLFNCFa2sAEx1Mr5ig9Q+JXxDYbj03y2FfRDfKUKLN/vjvnRWFeBfbQOT9ePSKt o80GCrstl32fZhtCKotgPXlldos+JaKLyEHqAlTiUfmLXIh3QNhNzcS6bi8Tyxop7CtO 9now== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Yqisb0p/1TgnFfChFg4hQVIMzPMIgaeBy+4V1ZQduJ4=; b=htMdd9gq2chFCkHcTkv9trMK/jX8D+wOaazY1Pn2VSz6CLUiqZ+5XZ7YEIw4Ke/O3J hn2aK8uw4XNt0W/2rT/0SyNLRjhjsjulXQeSHgKDmokdnT8/OghMdEjhctqzwHc/2Bq9 NEEMxVv0Tm9HLb7JclJjtK7ajCzOpGfkJBbbmnvN6zJdgAdjRli43JRwAe0UEur5YaQz PLyPlzwT2qKCZtlTp9ACx7C9gr3SQMbUWQOCzsn5XeAZAZsmjLE0YGZolxe5clhmgQai ApXKsaUF0tJ3DO1l4dvAR2ShckizUy3Kx0l0UbDbfUGuDwNt4YoUzsF2PXwbjOeQU/ON 6xMQ== X-Gm-Message-State: AO0yUKXwlkaU/JPomul/h6MnlFGR8RNe+UFMYEHLdPP4kBLnh8cCXJCY Hd+byopJi6AlAlMowFFflYoa7UE1CQo= X-Received: by 2002:a5d:6190:0:b0:2c7:a0b:e8d2 with SMTP id j16-20020a5d6190000000b002c70a0be8d2mr2569920wru.19.1677076759581; Wed, 22 Feb 2023 06:39:19 -0800 (PST) Received: from 127.0.0.1localhost (94.196.95.64.threembb.co.uk. [94.196.95.64]) by smtp.gmail.com with ESMTPSA id o2-20020a5d4742000000b002c59c6abc10sm8151735wrs.115.2023.02.22.06.39.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Feb 2023 06:39:19 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH for-next 4/4] io_uring/rsrc: optimise registered huge pages Date: Wed, 22 Feb 2023 14:36:51 +0000 Message-Id: <757a0c399774f23df5dbfe2e0cc79f7ea432b04c.1677041932.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758542641853280034?= X-GMAIL-MSGID: =?utf-8?q?1758542641853280034?= When registering huge pages, internally io_uring will split them into many PAGE_SIZE bvec entries. That's bad for performance as drivers need to eventually dma-map the data and will do it individually for each bvec entry. Coalesce huge pages into one large bvec. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 38 ++++++++++++++++++++++++++++++++------ 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index ebbd2cea7582..aab1bc6883e9 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1210,6 +1210,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, unsigned long off; size_t size; int ret, nr_pages, i; + struct folio *folio; *pimu = ctx->dummy_ubuf; if (!iov->iov_base) @@ -1224,6 +1225,21 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, goto done; } + /* If it's a huge page, try to coalesce them into a single bvec entry */ + if (nr_pages > 1) { + folio = page_folio(pages[0]); + for (i = 1; i < nr_pages; i++) { + if (page_folio(pages[i]) != folio) { + folio = NULL; + break; + } + } + if (folio) { + folio_put_refs(folio, nr_pages - 1); + nr_pages = 1; + } + } + imu = kvmalloc(struct_size(imu, bvec, nr_pages), GFP_KERNEL); if (!imu) goto done; @@ -1236,6 +1252,17 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, off = (unsigned long) iov->iov_base & ~PAGE_MASK; size = iov->iov_len; + /* store original address for later verification */ + imu->ubuf = (unsigned long) iov->iov_base; + imu->ubuf_end = imu->ubuf + iov->iov_len; + imu->nr_bvecs = nr_pages; + *pimu = imu; + ret = 0; + + if (folio) { + bvec_set_page(&imu->bvec[0], pages[0], size, off); + goto done; + } for (i = 0; i < nr_pages; i++) { size_t vec_len; @@ -1244,12 +1271,6 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, off = 0; size -= vec_len; } - /* store original address for later verification */ - imu->ubuf = (unsigned long) iov->iov_base; - imu->ubuf_end = imu->ubuf + iov->iov_len; - imu->nr_bvecs = nr_pages; - *pimu = imu; - ret = 0; done: if (ret) kvfree(imu); @@ -1364,6 +1385,11 @@ int io_import_fixed(int ddir, struct iov_iter *iter, const struct bio_vec *bvec = imu->bvec; if (offset <= bvec->bv_len) { + /* + * Note, huge pages buffers consists of one large + * bvec entry and should always go this way. The other + * branch doesn't expect non PAGE_SIZE'd chunks. + */ iter->bvec = bvec; iter->nr_segs = bvec->bv_len; iter->count -= offset;