From patchwork Wed Jul 12 16:20:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 11943 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp1295298vqm; Wed, 12 Jul 2023 10:14:29 -0700 (PDT) X-Google-Smtp-Source: APBJJlEinQnmg4ksIg26hg3ddvEtt6ZbkTH6EEqNSX2nqO24PFY1KTY9G4YefGSl1e/ss3nZGjY3 X-Received: by 2002:a17:902:d4c7:b0:1b6:a99b:ceb1 with SMTP id o7-20020a170902d4c700b001b6a99bceb1mr18786706plg.50.1689182069118; Wed, 12 Jul 2023 10:14:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689182069; cv=none; d=google.com; s=arc-20160816; b=MkZNg2G/Ca7Nk9h58k47SUoZJckbCqxp5/jLFPmqvipwIOe1UZyOl17aekpnp7LGbr /kMeLtmuoMpHm78fSwaH22HDvXCLUDnY2y4QUqxIC0GiPTJ5/fpwT1GLmDbfoCZvl42V iQMl8oH5v5lgWd7ReL/oFswX06zVlneqMDgGlKDfjxxOCfim4ihaYfd72Ot1sP2Nviz1 /58zvor+SX0Ee+8S1YAVXbbL0qz8W2gK3fgTgnFSzVp3xdfMo1+uB75KYpcbBkvt1Kqx 97+lYCnilF+3K0tbkJodnIo0bUoqZod0ML3IXln8QzujiKK9rS0s10Gpz654gosBuF9s 7qLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=GB+TVOcEAGohMo/9qjG0tcL6VxFJuBiixHc0IQwbD98=; fh=Yc1Pq4SO4cp2SMWZFqHWk5jCqhOcy7pio0TsAPBbUW4=; b=s2HDfIBO4O44KYYf08ZAEGYMXDesp1aojf9X+VkQD07U3MxhTZGnIOb6XSpkPDvw0c cCNEB+vQuJduvzTgt0oMfu9KLlIkZDAa7IQE/uzKIlz/kY/F1h42T0dJu75lOs9EqIRr BIeSmUkrLkmLM7dRab2utd9O72GurrtkD/VanVoheaO+V/epNyvq33HRgytXaf8dRvrI /vM6EWA/XUUSebNh99oeF0u7Y4Ufn7lwSr30Pn6YrpZBDDZokjsplHqI+eOl31TI3z3H 3DSWlMRLt5T0yJQSR5IGV3eVD+eLqfPJLSdz35UxDgUP1dHnFA7Og8iTf0c5uHKs/svZ lMvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=JqNSJM5Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c19-20020a170902f31300b001b8c4168e20si3517579ple.58.2023.07.12.10.14.14; Wed, 12 Jul 2023 10:14:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=JqNSJM5Q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232292AbjGLQVo (ORCPT + 99 others); Wed, 12 Jul 2023 12:21:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232963AbjGLQV3 (ORCPT ); Wed, 12 Jul 2023 12:21:29 -0400 Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 674622736 for ; Wed, 12 Jul 2023 09:20:23 -0700 (PDT) Received: by mail-io1-xd31.google.com with SMTP id ca18e2360f4ac-78706966220so32743139f.1 for ; Wed, 12 Jul 2023 09:20:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1689178822; x=1691770822; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=GB+TVOcEAGohMo/9qjG0tcL6VxFJuBiixHc0IQwbD98=; b=JqNSJM5QCDkw+R1ww+DDY6XYeGWUbAsCxQxhbbvdBaY2g7Yo9mq1Utaza/2CKVSM8B yDUYnnK6aVmrxP9r7gx+AZAaNJ1K4hxf2piEFU+6mlK1+eDms+kicmruEfTrHp0ik2Tt Vm+ktb+O8l708FmhVBKRZMaPK84pe3A7EU5PKUaNKk1VKGdOjQpM1eH+d+KC1BnPFvt6 LUxcTf90msPzRfcZqu0dh/LdCvPF+veHMiBnP5SgE1py/1HsKWqogpgveuIjiLxf1NZq M15munixc7UyORn1PlwYnauNOx1mIl1XABPGBcoCcjS+zit6CSyO2n9iVqMNJ3wR1riJ bYfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689178822; x=1691770822; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GB+TVOcEAGohMo/9qjG0tcL6VxFJuBiixHc0IQwbD98=; b=bfBZU5xspXefhGt5ggQGe9uqsRkYj94/N1UdoCh4x2OTbpShvPSdVP4GY2RNtjD3n/ eKFlrBbTYgxE7G8t0ULz7J1UBWmtzOSYtsG1IXjM+yZaGRnQu5HozUKMzsVLnuWByenW ZQ52meUz+m+oUo5wnRe2X+K3VvRl5WTgFwD3MkFkB74KoF9t6EEpPQ2HN3imJLXS1yJz B2CGULB+T2PtloMSl8vvy2sS2Od0EnHmO1TsBsUHuXHJdAvY+0Asw+y+Vt9j3ufGgaVI EwzQG6Hy2ZswSlNlsCGowFjjfbhcq4prRoKY7wsDhm/oaeAeIQj6gyjrjpHZFr/Vcvj2 Y+aQ== X-Gm-Message-State: ABy/qLbiEDHFg4XUH4CPCDipG+Kk3rDvK0ZEUAmkoqYwNEBUgm8nGdIm bVpIu+Oh0igOw/MSFsFesXS9MQ== X-Received: by 2002:a6b:1489:0:b0:780:cde6:3e22 with SMTP id 131-20020a6b1489000000b00780cde63e22mr18243438iou.0.1689178822177; Wed, 12 Jul 2023 09:20:22 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id x24-20020a029718000000b0042aec33bc26sm1328775jai.18.2023.07.12.09.20.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 09:20:21 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: tglx@linutronix.de, mingo@redhat.com, peterz@infradead.org, andres@anarazel.de Subject: [PATCHSET v2 0/8] Add io_uring futex/futexv support Date: Wed, 12 Jul 2023 10:20:09 -0600 Message-Id: <20230712162017.391843-1-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771235777427932910 X-GMAIL-MSGID: 1771235777427932910 Hi, This patchset adds support for first futex wake and wait, and then futexv. Patches 1..3 are just prep patches, patch 4 adds the wait and wake support for io_uring, and then patches 5..7 are again prep patches to end up with futexv support in patch 8. For both wait/wake/waitv, we support the bitset variant, as the "normal" variants can be easily implemented on top of that. PI and requeue are not supported through io_uring, just the above mentioned parts. This may change in the future, but in the spirit of keeping this small (and based on what people have been asking for), this is what we currently have. When I did these patches, I forgot that Pavel had previously posted a futex variant for io_uring. The major thing that had been holding me back from people asking about futexes and io_uring, is that I wanted to do this what I consider the right way - no usage of io-wq or thread offload, an actually async implementation that is efficient to use and don't rely on a blocking thread for futex wait/waitv. This is what this patchset attempts to do, while being minimally invasive on the futex side. I believe the diffstat reflects that. As far as I can recall, the first request for futex support with io_uring came from Andres Freund, working on postgres. His aio rework of postgres was one of the early adopters of io_uring, and futex support was a natural extension for that. This is relevant from both a usability point of view, as well as for effiency and performance. In Andres's words, for the former: "Futex wait support in io_uring makes it a lot easier to avoid deadlocks in concurrent programs that have their own buffer pool: Obviously pages in the application buffer pool have to be locked during IO. If the initiator of IO A needs to wait for a held lock B, the holder of lock B might wait for the IO A to complete. The ability to wait for a lock and IO completions at the same time provides an efficient way to avoid such deadlocks." and in terms of effiency, even without unlocking the full potential yet, Andres says: "Futex wake support in io_uring is useful because it allows for more efficient directed wakeups. For some "locks" postgres has queues implemented in userspace, with wakeup logic that cannot easily be implemented with FUTEX_WAKE_BITSET on a single "futex word" (imagine waiting for journal flushes to have completed up to a certain point). Thus a "lock release" sometimes need to wake up many processes in a row. A quick-and-dirty conversion to doing these wakeups via io_uring lead to a 3% throughput increase, with 12% fewer context switches, albeit in a fairly extreme workload." Some basic io_uring futex support and test cases are available in the liburing 'futex' branch: https://git.kernel.dk/cgit/liburing/log/?h=futex testing all of the variants. I originally wrote this code about a month ago and Andres has been using it with postgres, and I'm not aware of any bugs in it. That's not to say it's perfect, obviously, and I welcome some feedback so we can move this forward and hash out any potential issues. include/linux/io_uring_types.h | 3 + include/uapi/linux/io_uring.h | 4 + io_uring/Makefile | 4 +- io_uring/cancel.c | 5 + io_uring/cancel.h | 4 + io_uring/futex.c | 376 +++++++++++++++++++++++++++++++++ io_uring/futex.h | 36 ++++ io_uring/io_uring.c | 5 + io_uring/opdef.c | 35 ++- kernel/futex/futex.h | 33 +++ kernel/futex/requeue.c | 3 +- kernel/futex/syscalls.c | 27 ++- kernel/futex/waitwake.c | 49 +++-- 13 files changed, 548 insertions(+), 36 deletions(-) You can also find the code here: https://git.kernel.dk/cgit/linux/log/?h=io_uring-futex V2: - Abstract out __futex_wake_mark() helper. Use it both in the futex and io_uring code. This also fixes a missing WARN_ON on the io_uring side. - Have futex_op_to_flags() unconditionally clear flags to zero rather than do that in both callers. - Remove comment on needing to open-code futex_queue(), and associated hunk doing that. This was a leftover from an earlier version. - Expand the commit message logs in various patches.