From patchwork Thu Sep 21 18:29:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 143506 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5666405vqi; Fri, 22 Sep 2023 08:30:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFGjYC1Fs8r69qt14IPZzuanfFNRtEcks327vP8L5WbckHKQ+C4M0cRGv0VUY23JYCmMPig X-Received: by 2002:a05:6a21:3295:b0:15d:d250:d249 with SMTP id yt21-20020a056a21329500b0015dd250d249mr2127292pzb.37.1695396599748; Fri, 22 Sep 2023 08:29:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695396599; cv=none; d=google.com; s=arc-20160816; b=MFaqFdoHBF8CVCsqxwcDf0anSPMnudxKewcIPPyxSVn6rqnLewTXbn5MiOgi0zBvWW /jYo5YhhroFp1yvqzxwGFAB3fLmYf+rFkaK/ku1j5NhWKOblTWqvcgZJREe8pbfikKwM XqdBRjJGZFcNr+Jy15upbk5dAgmIxKJmeTFVEjid+IOzrhYYzp/mx3f/yYG/i/oSzhgR 6HEvIdeiq60TOksQnNm3zRt57MFSVt9UOpuIp2keYgrWPuPYNhDj4BscxTYbPytBsKlQ bXcMHXa72V9biYSpFewmCQrrweYq2DKLCfkBoAMohm1hPQ+R2RpG4uDl3+F7A+v2dIxI i1ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wwcbO19kHoooGXVXpd3ZWHbtBSLBsnmpj+7juMc3wkU=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=tL4zRE2tFzBhjr28jyMWLUrYFR1f+drKXPUbZOJ4oCH36hmHYIV70JdotR96cc1iGb dDSsE2Gq8SaGfbv+56VL2Nqils6va9oGuJBspzXH6cUB7n+T4ZWPvLYyRjm6tf4TjTXI Qn8DeJPn+06u0sprwjG7PeE0G67de366gaeSWQoE6w/Gemi4K7vrmne3/oB6wB6yVcgX 6Z2S9L5tAxBROK+RuttVrzaBfk2NQHsXMsXCI1hfCyREC0Ho0OwX8XJkizk24nhyJmdT eu4VZwVjtpN9lvhQgw5J/eci6hORCozGqWSBetblwqfU5q674BF1zN6VZo7ZLRXnrsRP c4gg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=3PZrgb1n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id y15-20020a17090322cf00b001bb792749a2si4413198plg.146.2023.09.22.08.29.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 22 Sep 2023 08:29:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=3PZrgb1n; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 0BD93807FCF2; Thu, 21 Sep 2023 12:55:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231230AbjIUTzl (ORCPT + 28 others); Thu, 21 Sep 2023 15:55:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231195AbjIUTzU (ORCPT ); Thu, 21 Sep 2023 15:55:20 -0400 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B0BBD7DAE for ; Thu, 21 Sep 2023 11:29:15 -0700 (PDT) Received: by mail-io1-xd2c.google.com with SMTP id ca18e2360f4ac-760dff4b701so17914239f.0 for ; Thu, 21 Sep 2023 11:29:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320955; x=1695925755; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wwcbO19kHoooGXVXpd3ZWHbtBSLBsnmpj+7juMc3wkU=; b=3PZrgb1nRUzH+q34WXjmHZtiDYbauRH6ZureE/sLaQKEcv8oVAXwj1eWrHEhRvjm1h 9d+JC16iwfuIoilg/Yr118CqhDgJ3UPADorZJFI2Wgc4ggtUzRxFBvfaLh4pFBAPz7DD y6cOLDNAtnFpbpjtuArdh5ZZwukRny0qkl+xlZODxqowZHqYb1FvYNzKDgYfQ/NrqVUM y7x7RJ9d1UM62rcDjPYri4+4azKb3znmMxJDNlLMe6pkHe0EKksZRODiCJiBEwRCZKfx /X9rNKLAFdx1sqZOhUnmlJVBCkleJzXdSmOHiKHacWG/Z2yXt2gSJjz3XE318ySbMDxD jxtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320955; x=1695925755; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wwcbO19kHoooGXVXpd3ZWHbtBSLBsnmpj+7juMc3wkU=; b=UGn0S/D3pOE7Nj9wf+0AmUVASx8puxbxTO0REnryAF3bvgW6dP/u9S9oyctz5buHVn RrAq6DwaKjDqYH381yz1hYD6BzUuo1u1PxbI2+A71UYafN7ouQ7VFtscsw8BkoyoLYLs 1Kzl+RW3YPbSm51z31lVD7gVLTLoU69nTegOPNbfsqDNdJ2TEK9Lbab0uxnRZZ/osfuH 998UMTd9rvc/qp/7oEKZxXAXo4Al8Iky/jcDSnW3RMzxOTYTVMK3K5JzM9a2Ht2BOWl1 oPnRYiCEZ/Sckz5dJeOcOXvgWnJ4O8U5vUAbTyIwSXetoCYVPZlaXQicyq0DKHClxzhy ZIKw== X-Gm-Message-State: AOJu0YxqRTwoLp3IDV1MxFzyx1fqO/Q6ptpXsuIasFbp3FVVUg+bfP9j r9s8cNo2yUpwXIRl3iNE0bQKdg== X-Received: by 2002:a05:6602:2c0a:b0:790:958e:a667 with SMTP id w10-20020a0566022c0a00b00790958ea667mr8177590iov.2.1695320954925; Thu, 21 Sep 2023 11:29:14 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:13 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 1/8] futex: move FUTEX2_VALID_MASK to futex.h Date: Thu, 21 Sep 2023 12:29:01 -0600 Message-Id: <20230921182908.160080-2-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Thu, 21 Sep 2023 12:55:50 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777752184830608238 X-GMAIL-MSGID: 1777752184830608238 We need this for validating the futex2 flags outside of the normal futex syscalls. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 2 ++ kernel/futex/syscalls.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index a06030a1a27b..a173a9d501e1 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -52,6 +52,8 @@ static inline unsigned int futex_to_flags(unsigned int op) return flags; } +#define FUTEX2_VALID_MASK (FUTEX2_SIZE_MASK | FUTEX2_PRIVATE) + /* FUTEX2_ to FLAGS_ */ static inline unsigned int futex2_to_flags(unsigned int flags2) { diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index 8200d86d30e1..2b5cafdfdc50 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -179,8 +179,6 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val, return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3); } -#define FUTEX2_VALID_MASK (FUTEX2_SIZE_MASK | FUTEX2_PRIVATE) - /** * futex_parse_waitv - Parse a waitv array from userspace * @futexv: Kernel side list of waiters to be filled From patchwork Thu Sep 21 18:29:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 142980 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5188648vqi; Thu, 21 Sep 2023 15:51:36 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFhMdMS7eDqWW9LwjyR1TT0y6UD0DNmOZzIkkjwOkQPwX3xwZQMobV3EHEIyf6KCPVnI8rr X-Received: by 2002:a05:6a20:9381:b0:14b:8023:33cb with SMTP id x1-20020a056a20938100b0014b802333cbmr7394651pzh.11.1695336696592; Thu, 21 Sep 2023 15:51:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695336696; cv=none; d=google.com; s=arc-20160816; b=GKnyfuEqStvpBwlWAstv9Es4ouWcL+IQ1/ouonzKu/Ze0TKHze6dOgfpuOaz4p/G4m 0VjwWpqBlnLHFD4sc2+Ks0tkEfbkCDrEoTfLzJIBIpmdAMezx6PYfGcyvoplfEapJ6CP 5ulVMyCl2O1JSrBjI0ROygS3Iv+PzZ3jkho0Pm0LBOaeolQplakvyMNqzZSOb6wFDLPW 8/zyvjRbdLPgOrjTeRo9QY2BSD9eMy6HDj8itRfdgCuy94tAah6pFEOF0YgcbYjHmGPI OmhnBnvZIuOSUuxLCIs2nfSzvLYHJUIej7qNUI50Ml+EX09luyPQNKAXRdHxomsdqhaj iSpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=V5m39pK9Sy+s98BPtXb5MPOzqgjCaZT2iuuWI0t83mc=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=Gg3rJq+ywRx9XPCEyTN4W2nc+NQSuwXjfIGeGPuOMQC0srD2Jcj5hbEAIcsOlrPHT/ gjdAxSYzoLQ6yTBOmxiCD+5M93ItKs02w/sXRELCnFpSq+o5EY5wlgL3HgI0bcJi4EDB Tq1e1HJRhW1XvzcsIbKBJC+FmdK5l756dXYB66Dov+X/YS/HQHBZX1zR635pEnqS50NY tj6x6ws/WntMA/Cdjl7lCgoQBfaa/ROi5ruGSMVsFkihSmFZwWz4hhCV3HavHolUX92/ ilspxBPA/FSwv/RbOgT/0YQxdPBR9j+8s5MOKAhofePOTOz/9fk6lE1LZtMgf5H1zy6c yiHg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=pbeT1sbp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id i12-20020a633c4c000000b00578a30162c8si2389864pgn.537.2023.09.21.15.51.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 15:51:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=pbeT1sbp; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id CD4C5837E9B0; Thu, 21 Sep 2023 11:37:55 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229625AbjIUShp (ORCPT + 29 others); Thu, 21 Sep 2023 14:37:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230053AbjIUShR (ORCPT ); Thu, 21 Sep 2023 14:37:17 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5A86D7DB5 for ; Thu, 21 Sep 2023 11:29:16 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id ca18e2360f4ac-79293ebfaf2so9846539f.1 for ; Thu, 21 Sep 2023 11:29:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320956; x=1695925756; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V5m39pK9Sy+s98BPtXb5MPOzqgjCaZT2iuuWI0t83mc=; b=pbeT1sbpkNh70vmMi3c2YLE6X/cRjVCPHP3VLnlyOgV6DtqthEfXH5xxiCUWNyq/aO Uev+MjFXqt/oSR+daovj2eWyBPsl3J/V3e7S4RjbmQ0mEYo65rVU1Faks5HvQjJ8yxm2 wjApNCrR206ws+T0WdYaGkPQoFRu59KCbABM+wp9dRYj2m6dPNeOsW/kFJyYjtoHE6Td 8chLc/WsroHdM7R3Nwb+37BgSrs1iijFll/osPXPuPQQsxBMY3bTCFNqjpuxqmLndUGW JgH/SYxLnTbYMyePJU1yjHhAmuhxwZSJiPznc9pjH3UKrnHOZDgsnnMXfPgxTLf2a+wH +jjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320956; x=1695925756; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V5m39pK9Sy+s98BPtXb5MPOzqgjCaZT2iuuWI0t83mc=; b=c12tllwlV10GCD1c6zZ0QAzHmHtLm/OMdhORZrYdyhqiHjdgcjTWkHolDXHhpTsKQP OHl7Lj7ZsQ2v5lRHtM4r+H+bq3PPphFiCvIDGQv6yEy1FTb+6g3xvbPkYWfpzGqVHLhx r+aVLwF6aHF7aYRAjIhVbRhb/4sMB+0IXAFqdRS3025s6KpCW8JuMtWzV/mg5v9wklsw lPN5amWOQIJd/+4HHqEuJUKYjXbx/xDiJIj/JCfBdcqtwAbmjPv7tLezd9+XodwXytI1 k//jHFiS7mIm4sm8FR7mj4QlHUUwc/chIuqd8NAbyDx6fypzMlbi+aOQvZcfUEbQpZQU WmXQ== X-Gm-Message-State: AOJu0YzIvSUfuf8O1xQmGRpFd5peZhGNPQxxFF+RPQfPdrJWtUJgvA8a 5eP8l2uiYA3BpH2I5jTGFl13mg== X-Received: by 2002:a05:6602:4996:b0:79a:c487:2711 with SMTP id eg22-20020a056602499600b0079ac4872711mr7316666iob.0.1695320956283; Thu, 21 Sep 2023 11:29:16 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:15 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 2/8] futex: factor out the futex wake handling Date: Thu, 21 Sep 2023 12:29:02 -0600 Message-Id: <20230921182908.160080-3-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:37:55 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777689371778929050 X-GMAIL-MSGID: 1777689371778929050 In preparation for having another waker that isn't futex_wake_mark(), add a wake handler in futex_q. No extra data is associated with the handler outside of struct futex_q itself. futex_wake_mark() is defined as the standard wakeup helper, now set through futex_q_init like other defaults. Normal sync futex waiting relies on wake_q holding tasks that should be woken up. This is what futex_wake_mark() does, it'll unqueue the futex and add the associated task to the wake queue. For async usage of futex waiting, rather than having tasks sleeping on the futex, we'll need to deal with a futex wake differently. For the planned io_uring case, that means posting a completion event for the task in question. Having a definable wake handler can help support that use case. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 5 +++++ kernel/futex/requeue.c | 3 ++- kernel/futex/waitwake.c | 6 +++--- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index a173a9d501e1..547f509b2c87 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -139,11 +139,15 @@ struct futex_pi_state { union futex_key key; } __randomize_layout; +struct futex_q; +typedef void (futex_wake_fn)(struct wake_q_head *wake_q, struct futex_q *q); + /** * struct futex_q - The hashed futex queue entry, one per waiting task * @list: priority-sorted list of tasks waiting on this futex * @task: the task waiting on the futex * @lock_ptr: the hash bucket lock + * @wake: the wake handler for this queue * @key: the key the futex is hashed on * @pi_state: optional priority inheritance state * @rt_waiter: rt_waiter storage for use with requeue_pi @@ -168,6 +172,7 @@ struct futex_q { struct task_struct *task; spinlock_t *lock_ptr; + futex_wake_fn *wake; union futex_key key; struct futex_pi_state *pi_state; struct rt_mutex_waiter *rt_waiter; diff --git a/kernel/futex/requeue.c b/kernel/futex/requeue.c index a0a79954f506..9dc789399a1a 100644 --- a/kernel/futex/requeue.c +++ b/kernel/futex/requeue.c @@ -58,6 +58,7 @@ enum { const struct futex_q futex_q_init = { /* list gets initialized in futex_queue()*/ + .wake = futex_wake_mark, .key = FUTEX_KEY_INIT, .bitset = FUTEX_BITSET_MATCH_ANY, .requeue_state = ATOMIC_INIT(Q_REQUEUE_PI_NONE), @@ -593,7 +594,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags1, /* Plain futexes just wake or requeue and are done */ if (!requeue_pi) { if (++task_count <= nr_wake) - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); else requeue_futex(this, hb1, hb2, &key2); continue; diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 37860f794bf7..35c6a637a4bb 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -177,7 +177,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) if (!(this->bitset & bitset)) continue; - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); if (++ret >= nr_wake) break; } @@ -292,7 +292,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, ret = -EINVAL; goto out_unlock; } - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); if (++ret >= nr_wake) break; } @@ -306,7 +306,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, ret = -EINVAL; goto out_unlock; } - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); if (++op_ret >= nr_wake2) break; } From patchwork Thu Sep 21 18:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 142863 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5092070vqi; Thu, 21 Sep 2023 12:38:43 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF5J9QnPI0l5/pqSpsUO9Am56U0Cr9S1oWTfmdN5vQ7BcfqFZwVJl9FA6x1UhDEc1K4eVxE X-Received: by 2002:a05:6a20:428e:b0:157:b453:dbb9 with SMTP id o14-20020a056a20428e00b00157b453dbb9mr7773727pzj.6.1695325123429; Thu, 21 Sep 2023 12:38:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695325123; cv=none; d=google.com; s=arc-20160816; b=KnPcKBYEXCsXXTlwMeMaGUZkOzmSCUk2C3FGe0XxDDm1xYltfsuKWgzL0HZs97HdOi aj2hToH1X36yjlfyu5Avyury1+tq2OP/9rm5nFUH1DkoMVW61sfkCEHubcdbYpfZaDbj 3WVHXg9Vru2q7G67CGmAzDQ4dGp63Jr0Wy1M2DegvRoNWmzU0jPeZlDiVzkJZYVJj5i9 XZrTzoYONZl5y692NHk/tMZhAESAf90jlrrY8JYynH1YOqRxj1o4vLhnbNp8v8FXYXqo 8uA0z7xBudaMgMQ6sCxGOKKbvPWV5G+DeLpFBsnItW9b08NTV4P6qxIY5HZj9HrM2T+4 hrdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aeITMYfcld3mteb+NSEcBXXoBeyRBDS9X8Wlurs1dfY=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=GJ8a5ORTkzUjrRII+nW69FlOdtuS1ARZ4o62KDXV2ZVAAayJak5QLcapwah8RZX7QP ju+qr0vAYLF3nxfwJHph8O8F40sRO1GUtnsEt+S7VdSzsjRSJHTA39BsNCfgyWX8N+4K bk5bHp1DhCIU6EaJiOVrnOvmlqvL+mIS9eQlZLSDlgNLHLIFE+Hir2GoPdQMEUX5hUEW Haz+YCNfOh2O/Z7v+C5KoZ0LNczgRjt55E3B7w8lar+c7r65W3q6y95nidI97G/Bfc0s R1TT6DvfWhcIgpk1u77qw8pRP3yuOZuoi70ZGYr4Mm2p/f4oZCwu2WjKVDiH9q1Qy9Us b8gQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=elkX0bbQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id p13-20020a170902e74d00b001b53a3fbcc3si2245849plf.328.2023.09.21.12.38.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 12:38:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=elkX0bbQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 44DE483F426D; Thu, 21 Sep 2023 11:37:50 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229968AbjIUShs (ORCPT + 29 others); Thu, 21 Sep 2023 14:37:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229931AbjIUShR (ORCPT ); Thu, 21 Sep 2023 14:37:17 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60085D7DB8 for ; Thu, 21 Sep 2023 11:29:18 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id ca18e2360f4ac-79f8df47bfbso10251639f.0 for ; Thu, 21 Sep 2023 11:29:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320957; x=1695925757; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aeITMYfcld3mteb+NSEcBXXoBeyRBDS9X8Wlurs1dfY=; b=elkX0bbQnyNGl1I50jhGbi+pWcQeuY5xLC5VLpCo+FVlOp7UwN6P0Qjig8a5k6gTkQ IL7gyU0LgEPpo9Y/PpIaiUVN0MVHLzLecVmUCsHkN14IwxkjxIkhNo9w2VQxBKO2Gpyr T3n2Z7dN+Z75R5u4IwI8PyaftN/9UYXl9GqRtbYkA4PM7NrFR1UmBTFreO0O11jsr1a4 DbXsv1worfom++ih99/XxocYLChcdRN0DHTxZPIdcbChypfX1OaaNmOSAeXQr2AkEGZZ H8ULrGx+Y3Ie51DabPn9XxVnN6c6uFlFfjOXjZgyG9q3PPDHxGudBcFECoeA6MCZe4Eq jFCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320957; x=1695925757; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aeITMYfcld3mteb+NSEcBXXoBeyRBDS9X8Wlurs1dfY=; b=mp60dpQUBWae6B3KnFos1kxEK5ImNSPW115+wO3EzzpbSHtZ6znzvLd4T6fvGm2AXU YvEwLBSwen77cSU5/5O6Ny88nVnWhmUVlrg+UHi19mm5QwQ95L8sTxKVABxlVgIjEQYp BHF6pQDWHd+0zrJBD7byDV690uFcRM4hePteNigkld9R3/D5uFCnS8+Q8kkmt/xCjFRR amoH5c5q+mk2DD6nrG/W2tC5aLqUuE2Kp4uTld4G1UWmzVEZCNmJvYPsrNUicsE8JlUc /zWalBt9kId2baNjN1BTnEHj95O0LRNI0QKNHMMiuq9jGvVtAQfSjxiGGD8IFoC9t417 maRQ== X-Gm-Message-State: AOJu0YxGNEAMk/uFvk3ohDfWDGgXj1pFpLQZn1SJh2qT3E5WyYnQ6Smm xLZT9UrScN8qtbg5NZlcbxMuabolKxp6IXSiluIBXA== X-Received: by 2002:a92:d903:0:b0:350:f353:4017 with SMTP id s3-20020a92d903000000b00350f3534017mr6601502iln.0.1695320957710; Thu, 21 Sep 2023 11:29:17 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:16 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 3/8] futex: abstract out a __futex_wake_mark() helper Date: Thu, 21 Sep 2023 12:29:03 -0600 Message-Id: <20230921182908.160080-4-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:37:50 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777677236311152193 X-GMAIL-MSGID: 1777677236311152193 Move the unqueue and lock_ptr clear into a helper that futex_wake_mark() calls. Add it to the public functions as well, in preparation for using it outside the core futex code. Suggested-by: Peter Zijlstra Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 1 + kernel/futex/waitwake.c | 33 ++++++++++++++++++++++----------- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 547f509b2c87..33835b81e0c3 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -219,6 +219,7 @@ extern int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags, struct futex_q *q, struct futex_hash_bucket **hb); extern void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, struct hrtimer_sleeper *timeout); +extern bool __futex_wake_mark(struct futex_q *q); extern void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q); extern int fault_in_user_writeable(u32 __user *uaddr); diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 35c6a637a4bb..6fcf5f723719 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -106,20 +106,11 @@ * double_lock_hb() and double_unlock_hb(), respectively. */ -/* - * The hash bucket lock must be held when this is called. - * Afterwards, the futex_q must not be accessed. Callers - * must ensure to later call wake_up_q() for the actual - * wakeups to occur. - */ -void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) +bool __futex_wake_mark(struct futex_q *q) { - struct task_struct *p = q->task; - if (WARN(q->pi_state || q->rt_waiter, "refusing to wake PI futex\n")) - return; + return false; - get_task_struct(p); __futex_unqueue(q); /* * The waiting task can free the futex_q as soon as q->lock_ptr = NULL @@ -130,6 +121,26 @@ void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) */ smp_store_release(&q->lock_ptr, NULL); + return true; +} + +/* + * The hash bucket lock must be held when this is called. + * Afterwards, the futex_q must not be accessed. Callers + * must ensure to later call wake_up_q() for the actual + * wakeups to occur. + */ +void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) +{ + struct task_struct *p = q->task; + + get_task_struct(p); + + if (!__futex_wake_mark(q)) { + put_task_struct(p); + return; + } + /* * Queue the task for later wakeup for after we've released * the hb->lock. From patchwork Thu Sep 21 18:29:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 143020 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5221459vqi; Thu, 21 Sep 2023 17:07:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH8ppgTJ0RwfBljtXv0VN0hX6EpyynI5y/3NrPwgNKCtjy4oXyvs16+CeaKjawpCG+CNKed X-Received: by 2002:a05:6870:b28c:b0:1d6:4c63:7bb0 with SMTP id c12-20020a056870b28c00b001d64c637bb0mr8441264oao.17.1695341237032; Thu, 21 Sep 2023 17:07:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695341237; cv=none; d=google.com; s=arc-20160816; b=Tjh1hxnTWnXRqGBbraCsO1tgul+YLDxytIYdCZc2KlQMIqMPWPtLwOv8PFlFvH6L5i 08Vm5pIUWkvtW6cXwkSz6Rn+jmRfSBlj3/gIuRvQT7a4oWGOthtHU9MceraFqRhKBbVW 7ZhRFjgsrxBXcSQkhh+1Ahry4yIixBruYMLaot3+TugF2pzjK/5jEFXVZcHCdDofshu5 ojIhx8gKSEYwPd+XnZi2dLT856R3qbXPeWp3djnT67ced2byk37a8JcHc8J0ouf7EUGg hfuKvQORhDFeqLbXr6a8bhMUF39SXCFZYSe+lTwqZmv/PbHKNNQzCAeAYhDWbEdL50SY xW8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7k8ejLYif2IyrgBYm50aE+8b9yDEF7uyTbKhg5k4iOU=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=RWhq7f9KTcgJ7WtdDi5NzV3+5ilUo2vDtLGEELJGXVUSZKLp7AGLVt+DrqLGbzsJCi Pi7TCUHBsJDtFVTFkrV8XakFcB/pyechaxSwEikm69jZHSWsh1gzOtrVsdLdOK99HANs mlas2qtXyBoTH74DJZPI3bR6kLyGAWnprSuYtycuyTA0mEXy8zDTjNeUs6tRl3bu+EzX UTzA2bsDo7pW2SUkvg+wWe8gI/MOZzf4p+1oeZjlvoukMeOQSrB0uJ5ctJxcTuxN5Zag FMV1sRuHoEmHPLT26mmNIDEF8mp6nf00IvvWSb7B6ARrFTkwis7IjSCbTc1BKTtaSmek tanA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=pwijKD9h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id bz14-20020a056a02060e00b00578e7a37c44si3060738pgb.42.2023.09.21.17.07.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 17:07:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=pwijKD9h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 1D7EC834F488; Thu, 21 Sep 2023 11:38:53 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230073AbjIUShw (ORCPT + 29 others); Thu, 21 Sep 2023 14:37:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230301AbjIUShR (ORCPT ); Thu, 21 Sep 2023 14:37:17 -0400 Received: from mail-io1-xd34.google.com (mail-io1-xd34.google.com [IPv6:2607:f8b0:4864:20::d34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCD7DD7DBB for ; Thu, 21 Sep 2023 11:29:19 -0700 (PDT) Received: by mail-io1-xd34.google.com with SMTP id ca18e2360f4ac-77dcff76e35so17714939f.1 for ; Thu, 21 Sep 2023 11:29:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320959; x=1695925759; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7k8ejLYif2IyrgBYm50aE+8b9yDEF7uyTbKhg5k4iOU=; b=pwijKD9hdirFOef0c5Z21XQag4QOl6ce9O1OwytB7awhWDi0kaceNEgjsuTEXuOskZ JEa5aOylzELWipInx3NTX4rur5OSJ4m5QExMAGf8ZNzpwlOnm4+2HM0KFu5FKWRVQ4gq 8yCmWhzAGd341ICgyelAUNYIImVZxO/f6P8dLSboiOj6cZbVlR92GrnYYGyGjjKPgx2S 5WNvKCvk7vP0S7FYp3WTZDIrD2eHcPzZ12WThAaUZZT6nubaL5VfSbhBYLD9b9DDJ0mC DujFrkpUBpoStHXRlPBnW8sgmoa7M5WURKH9Lz+ZodvAAM6AsS0Mn3UGpitOqklXv9Uo 9gzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320959; x=1695925759; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7k8ejLYif2IyrgBYm50aE+8b9yDEF7uyTbKhg5k4iOU=; b=sEk4feHMX37TpJ5G6uG21TmVokk9uHRgzEdNr7Q6Y8g+UPCYJH+IyrbTtmu4tWgt2u 6qNClVvfHI98QQcfp/7O65P23BKTHcbTEek31CZGiTmhYRfQrY8G9i5RvWOj2572h6Dy 6bGCJCLs9SF6kQsqGo2YK/UycD8hSnyqvvCrJ1UQqPJStir2NpwsbqK75eTRzkbSeTEM 5XfrQ3/G4eo6VpTRHovB5mrGcfUr4ZMl0B7ZrNMw7/phQumHKylDEa7tcvBALsWdtaju lO0lLewtzFaYOBYlHrzJZdjEL31gU2rQP8lYzYv6lZzNuqtM6W3MHij2gGT5lNC6qS4u pK6w== X-Gm-Message-State: AOJu0YzDvuetaAnWbCbvDCpjHKFt70D/mdsZK1yLBKZw4WlbpCdlRlf+ Tc+xM3BX8FOdfRwm7BfQjq4pVQ== X-Received: by 2002:a05:6602:2e15:b0:792:6dd8:a65f with SMTP id o21-20020a0566022e1500b007926dd8a65fmr8609274iow.0.1695320958988; Thu, 21 Sep 2023 11:29:18 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:18 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 4/8] io_uring: add support for futex wake and wait Date: Thu, 21 Sep 2023 12:29:04 -0600 Message-Id: <20230921182908.160080-5-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:38:53 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777694133003151940 X-GMAIL-MSGID: 1777694133003151940 Add support for FUTEX_WAKE/WAIT primitives. IORING_OP_FUTEX_WAKE is mix of FUTEX_WAKE and FUTEX_WAKE_BITSET, as it does support passing in a bitset. Similary, IORING_OP_FUTEX_WAIT is a mix of FUTEX_WAIT and FUTEX_WAIT_BITSET. FUTEX_WAKE is straight forward, as those can always be done directly from the io_uring submission without needing async handling. For FUTEX_WAIT, things are a bit more complicated. If the futex isn't ready, then we rely on a callback via futex_queue->wake() when someone wakes up the futex. From that calback, we queue up task_work with the original task, which will post a CQE and wake it, if necessary. Cancelations are supported, both from the application point-of-view, but also to be able to cancel pending waits if the ring exits before all events have occurred. The return value of futex_unqueue() is used to gate who wins the potential race between cancelation and futex wakeups. Whomever gets a 'ret == 1' return from that claims ownership of the io_uring futex request. This is just the barebones wait/wake support. PI or REQUEUE support is not added at this point, unclear if we might look into that later. Likewise, explicit timeouts are not supported either. It is expected that users that need timeouts would do so via the usual io_uring mechanism to do that using linked timeouts. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 5 + include/uapi/linux/io_uring.h | 3 + io_uring/Makefile | 1 + io_uring/cancel.c | 5 + io_uring/cancel.h | 4 + io_uring/futex.c | 230 +++++++++++++++++++++++++++++++++ io_uring/futex.h | 34 +++++ io_uring/io_uring.c | 7 + io_uring/opdef.c | 23 ++++ 9 files changed, 312 insertions(+) create mode 100644 io_uring/futex.c create mode 100644 io_uring/futex.h diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index fe1c5d4ec56c..4fc68cd6b043 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -315,6 +315,11 @@ struct io_ring_ctx { struct hlist_head waitid_list; +#ifdef CONFIG_FUTEX + struct hlist_head futex_list; + struct io_alloc_cache futex_cache; +#endif + const struct cred *sq_creds; /* cred used for __io_sq_thread() */ struct io_sq_data *sq_data; /* if using sq thread polling */ diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 683ac2b74721..4ddd7bdbbfb8 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -66,6 +66,7 @@ struct io_uring_sqe { __u32 msg_ring_flags; __u32 uring_cmd_flags; __u32 waitid_flags; + __u32 futex_flags; }; __u64 user_data; /* data to be passed back at completion time */ /* pack this to avoid bogus arm OABI complaints */ @@ -243,6 +244,8 @@ enum io_uring_op { IORING_OP_SENDMSG_ZC, IORING_OP_READ_MULTISHOT, IORING_OP_WAITID, + IORING_OP_FUTEX_WAIT, + IORING_OP_FUTEX_WAKE, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/Makefile b/io_uring/Makefile index 7bd64e442567..e5be47e4fc3b 100644 --- a/io_uring/Makefile +++ b/io_uring/Makefile @@ -10,3 +10,4 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ cancel.o kbuf.o rsrc.o rw.o opdef.o \ notif.o waitid.o obj-$(CONFIG_IO_WQ) += io-wq.o +obj-$(CONFIG_FUTEX) += futex.o diff --git a/io_uring/cancel.c b/io_uring/cancel.c index eb77a51c5a79..3c19cccb1aec 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -16,6 +16,7 @@ #include "poll.h" #include "timeout.h" #include "waitid.h" +#include "futex.h" #include "cancel.h" struct io_cancel { @@ -124,6 +125,10 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd, if (ret != -ENOENT) return ret; + ret = io_futex_cancel(ctx, cd, issue_flags); + if (ret != -ENOENT) + return ret; + spin_lock(&ctx->completion_lock); if (!(cd->flags & IORING_ASYNC_CANCEL_FD)) ret = io_timeout_cancel(ctx, cd); diff --git a/io_uring/cancel.h b/io_uring/cancel.h index fc98622e6166..c0a8e7c520b6 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -1,4 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 +#ifndef IORING_CANCEL_H +#define IORING_CANCEL_H #include @@ -22,3 +24,5 @@ void init_hash_table(struct io_hash_table *table, unsigned size); int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg); bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd); + +#endif diff --git a/io_uring/futex.c b/io_uring/futex.c new file mode 100644 index 000000000000..4278302d212c --- /dev/null +++ b/io_uring/futex.c @@ -0,0 +1,230 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include + +#include + +#include "../kernel/futex/futex.h" +#include "io_uring.h" +#include "rsrc.h" +#include "futex.h" + +struct io_futex { + struct file *file; + u32 __user *uaddr; + unsigned long futex_val; + unsigned long futex_mask; + u32 futex_flags; +}; + +struct io_futex_data { + union { + struct futex_q q; + struct io_cache_entry cache; + }; + struct io_kiocb *req; +}; + +void io_futex_cache_init(struct io_ring_ctx *ctx) +{ + io_alloc_cache_init(&ctx->futex_cache, IO_NODE_ALLOC_CACHE_MAX, + sizeof(struct io_futex_data)); +} + +static void io_futex_cache_entry_free(struct io_cache_entry *entry) +{ + kfree(container_of(entry, struct io_futex_data, cache)); +} + +void io_futex_cache_free(struct io_ring_ctx *ctx) +{ + io_alloc_cache_free(&ctx->futex_cache, io_futex_cache_entry_free); +} + +static void io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) +{ + struct io_futex_data *ifd = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + + io_tw_lock(ctx, ts); + if (!io_alloc_cache_put(&ctx->futex_cache, &ifd->cache)) + kfree(ifd); + req->async_data = NULL; + hlist_del_init(&req->hash_node); + io_req_task_complete(req, ts); +} + +static bool __io_futex_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) +{ + struct io_futex_data *ifd = req->async_data; + + /* futex wake already done or in progress */ + if (!futex_unqueue(&ifd->q)) + return false; + + hlist_del_init(&req->hash_node); + io_req_set_res(req, -ECANCELED, 0); + req->io_task_work.func = io_futex_complete; + io_req_task_work_add(req); + return true; +} + +int io_futex_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned int issue_flags) +{ + struct hlist_node *tmp; + struct io_kiocb *req; + int nr = 0; + + if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_FD_FIXED)) + return -ENOENT; + + io_ring_submit_lock(ctx, issue_flags); + hlist_for_each_entry_safe(req, tmp, &ctx->futex_list, hash_node) { + if (req->cqe.user_data != cd->data && + !(cd->flags & IORING_ASYNC_CANCEL_ANY)) + continue; + if (__io_futex_cancel(ctx, req)) + nr++; + if (!(cd->flags & IORING_ASYNC_CANCEL_ALL)) + break; + } + io_ring_submit_unlock(ctx, issue_flags); + + if (nr) + return nr; + + return -ENOENT; +} + +bool io_futex_remove_all(struct io_ring_ctx *ctx, struct task_struct *task, + bool cancel_all) +{ + struct hlist_node *tmp; + struct io_kiocb *req; + bool found = false; + + lockdep_assert_held(&ctx->uring_lock); + + hlist_for_each_entry_safe(req, tmp, &ctx->futex_list, hash_node) { + if (!io_match_task_safe(req, task, cancel_all)) + continue; + __io_futex_cancel(ctx, req); + found = true; + } + + return found; +} + +int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + u32 flags; + + if (unlikely(sqe->fd || sqe->len || sqe->buf_index || sqe->file_index)) + return -EINVAL; + + iof->uaddr = u64_to_user_ptr(READ_ONCE(sqe->addr)); + iof->futex_val = READ_ONCE(sqe->addr2); + iof->futex_mask = READ_ONCE(sqe->addr3); + flags = READ_ONCE(sqe->futex_flags); + + if (flags & ~FUTEX2_VALID_MASK) + return -EINVAL; + + iof->futex_flags = futex2_to_flags(flags); + if (!futex_flags_valid(iof->futex_flags)) + return -EINVAL; + + if (!futex_validate_input(iof->futex_flags, iof->futex_val) || + !futex_validate_input(iof->futex_flags, iof->futex_mask)) + return -EINVAL; + + return 0; +} + +static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q) +{ + struct io_futex_data *ifd = container_of(q, struct io_futex_data, q); + struct io_kiocb *req = ifd->req; + + if (unlikely(!__futex_wake_mark(q))) + return; + + io_req_set_res(req, 0, 0); + req->io_task_work.func = io_futex_complete; + io_req_task_work_add(req); +} + +static struct io_futex_data *io_alloc_ifd(struct io_ring_ctx *ctx) +{ + struct io_cache_entry *entry; + + entry = io_alloc_cache_get(&ctx->futex_cache); + if (entry) + return container_of(entry, struct io_futex_data, cache); + + return kmalloc(sizeof(struct io_futex_data), GFP_NOWAIT); +} + +int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct io_ring_ctx *ctx = req->ctx; + struct io_futex_data *ifd = NULL; + struct futex_hash_bucket *hb; + int ret; + + if (!iof->futex_mask) { + ret = -EINVAL; + goto done; + } + + io_ring_submit_lock(ctx, issue_flags); + ifd = io_alloc_ifd(ctx); + if (!ifd) { + ret = -ENOMEM; + goto done_unlock; + } + + req->async_data = ifd; + ifd->q = futex_q_init; + ifd->q.bitset = iof->futex_mask; + ifd->q.wake = io_futex_wake_fn; + ifd->req = req; + + ret = futex_wait_setup(iof->uaddr, iof->futex_val, + futex2_to_flags(iof->futex_flags), &ifd->q, &hb); + if (!ret) { + hlist_add_head(&req->hash_node, &ctx->futex_list); + io_ring_submit_unlock(ctx, issue_flags); + + futex_queue(&ifd->q, hb); + return IOU_ISSUE_SKIP_COMPLETE; + } + +done_unlock: + io_ring_submit_unlock(ctx, issue_flags); +done: + if (ret < 0) + req_set_fail(req); + io_req_set_res(req, ret, 0); + kfree(ifd); + return IOU_OK; +} + +int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + int ret; + + ret = futex_wake(iof->uaddr, futex2_to_flags(iof->futex_flags), + iof->futex_val, iof->futex_mask); + if (ret < 0) + req_set_fail(req); + io_req_set_res(req, ret, 0); + return IOU_OK; +} diff --git a/io_uring/futex.h b/io_uring/futex.h new file mode 100644 index 000000000000..ddc9e0d73c52 --- /dev/null +++ b/io_uring/futex.h @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "cancel.h" + +int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags); +int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags); + +#if defined(CONFIG_FUTEX) +int io_futex_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned int issue_flags); +bool io_futex_remove_all(struct io_ring_ctx *ctx, struct task_struct *task, + bool cancel_all); +void io_futex_cache_init(struct io_ring_ctx *ctx); +void io_futex_cache_free(struct io_ring_ctx *ctx); +#else +static inline int io_futex_cancel(struct io_ring_ctx *ctx, + struct io_cancel_data *cd, + unsigned int issue_flags) +{ + return 0; +} +static inline bool io_futex_remove_all(struct io_ring_ctx *ctx, + struct task_struct *task, bool cancel_all) +{ + return false; +} +static inline void io_futex_cache_init(struct io_ring_ctx *ctx) +{ +} +static inline void io_futex_cache_free(struct io_ring_ctx *ctx) +{ +} +#endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2dff4772bf14..f4198258159d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -93,6 +93,7 @@ #include "net.h" #include "notif.h" #include "waitid.h" +#include "futex.h" #include "timeout.h" #include "poll.h" @@ -330,6 +331,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) sizeof(struct async_poll)); io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, sizeof(struct io_async_msghdr)); + io_futex_cache_init(ctx); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); @@ -350,6 +352,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) ctx->submit_state.free_list.next = NULL; INIT_WQ_LIST(&ctx->locked_free_list); INIT_HLIST_HEAD(&ctx->waitid_list); +#ifdef CONFIG_FUTEX + INIT_HLIST_HEAD(&ctx->futex_list); +#endif INIT_DELAYED_WORK(&ctx->fallback_work, io_fallback_req_func); INIT_WQ_LIST(&ctx->submit_state.compl_reqs); return ctx; @@ -2894,6 +2899,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_eventfd_unregister(ctx); io_alloc_cache_free(&ctx->apoll_cache, io_apoll_cache_free); io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); + io_futex_cache_free(ctx); io_destroy_buffers(ctx); mutex_unlock(&ctx->uring_lock); if (ctx->sq_creds) @@ -3306,6 +3312,7 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, mutex_lock(&ctx->uring_lock); ret |= io_poll_remove_all(ctx, task, cancel_all); ret |= io_waitid_remove_all(ctx, task, cancel_all); + ret |= io_futex_remove_all(ctx, task, cancel_all); mutex_unlock(&ctx->uring_lock); ret |= io_kill_timeouts(ctx, task, cancel_all); if (task) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index aadcbf7136b0..31a3a421e94d 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -34,6 +34,7 @@ #include "cancel.h" #include "rw.h" #include "waitid.h" +#include "futex.h" static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags) { @@ -444,6 +445,22 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_waitid_prep, .issue = io_waitid, }, + [IORING_OP_FUTEX_WAIT] = { +#if defined(CONFIG_FUTEX) + .prep = io_futex_prep, + .issue = io_futex_wait, +#else + .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_FUTEX_WAKE] = { +#if defined(CONFIG_FUTEX) + .prep = io_futex_prep, + .issue = io_futex_wake, +#else + .prep = io_eopnotsupp_prep, +#endif + }, }; const struct io_cold_def io_cold_defs[] = { @@ -670,6 +687,12 @@ const struct io_cold_def io_cold_defs[] = { .name = "WAITID", .async_size = sizeof(struct io_waitid_async), }, + [IORING_OP_FUTEX_WAIT] = { + .name = "FUTEX_WAIT", + }, + [IORING_OP_FUTEX_WAKE] = { + .name = "FUTEX_WAKE", + }, }; const char *io_uring_get_opcode(u8 opcode) From patchwork Thu Sep 21 18:29:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 143058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5251265vqi; Thu, 21 Sep 2023 18:18:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG+x38hkqUz0o4wvzhc3FwlqwysFZpcmUwNYT8qvpFoWziPkdo3w7aPxuumyeU/spxDYFi3 X-Received: by 2002:a17:903:120b:b0:1c3:1167:26e4 with SMTP id l11-20020a170903120b00b001c3116726e4mr7621483plh.60.1695345526122; Thu, 21 Sep 2023 18:18:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695345526; cv=none; d=google.com; s=arc-20160816; b=CeapFWAa+Eo44dK4g8/toPyXV85lfBMIVUH9AN0jYWEUGpJqSfXxNCchj7zo4LHuHZ bdmDWnZ8ZRQEG+t28+YxBovXlusMqFwSeie5ekgFEqx1QXlT7vpJzlBWaUvztExJ8Ung gwCg4C0BtIu0zfp+XF0DowPDQeD9bS5BzrTAyYsvlD7SYZyvY+Xxz+iIZO59yGWqvMCf NFRcyfPStZ5R/TfKOMyACHwCpNPd0P/22RieRVrHbN4vdh/Cbu1fyUMjasPoqF1gIvBr VLqDjEKbuSw/yCWPEmYaNRnT8ddRRHbD+48MtFduk33BJGtWwl7pM8peFK0rUGczdNRB KqJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=v/TGT+v2onwhR3DWI/C2XC90jEj/DLLYrXKmnR2IfXs=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=GlgdNnU/ix4Jkvkp9ohuuOqq8I3+t82vG3z26deSOkPKxPISvwF+DGjth8FPn8n7OT iQWPd4lXbTrARe2Er5Rwrl6zr7kb8ESekbZyV0Bj6M1P/T2PeyQQS6Nq50+yj3qQSBji V05Wopcxqh0nVx3dnsk4fpwVhWCIbXJ5EyW/gvyZO1H+Nd4eOAv03KmiTR31y/0oa976 u/wNY/4tzEiFRgDD9EpugPagSSdnvjrsOY/fA2FEjzaidElVWzEbb6gDSEKvYlEhS0PX A/Nba2FUbqujGBIhfM7C7bunlazU/TwR9b84Roc3bR8HEP7OsOhQMhwhuWU6DlFOFp99 Ct4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=OkyarrJg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id u16-20020a170902e81000b001b9e82a6beesi2833803plg.548.2023.09.21.18.18.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 18:18:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=OkyarrJg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id E55948311B5C; Thu, 21 Sep 2023 11:41:17 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229783AbjIUSiE (ORCPT + 29 others); Thu, 21 Sep 2023 14:38:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35358 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229720AbjIUShT (ORCPT ); Thu, 21 Sep 2023 14:37:19 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4730BD8681 for ; Thu, 21 Sep 2023 11:29:21 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id ca18e2360f4ac-797ea09af91so18715439f.1 for ; Thu, 21 Sep 2023 11:29:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320960; x=1695925760; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=v/TGT+v2onwhR3DWI/C2XC90jEj/DLLYrXKmnR2IfXs=; b=OkyarrJghc8lGO3QyBOffqxkuM6y4VS1FNrUywgbd6eLH2gBeuptfipWNb37WvyKRl 69hRMuQOTTrrBPt4/pHvw6Hkw9+Spqdcw7S2Nmy84ielPmjMsUmO7fzrvam2kz45Utwj PEaxUZzV8Rir9QoJYcxCYRP2RIYfoff6GZkeexm33xiWiqUYfo3T4n8JqQeg1PYWpB6g pE8fjNTk3k/VxvzSB+AzWlQ3uKEaa1kxp5MVxuZHOLcHuJcopOWRbORGw/TL+zouKcsV laQdtc5JHhG33+uJxwrLblO1S8aikB2D0Z1ld4VLhaAUJmehbJgGUPW1G2nNYz0hTmGH I5tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320960; x=1695925760; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v/TGT+v2onwhR3DWI/C2XC90jEj/DLLYrXKmnR2IfXs=; b=vwt514YaCCVPZ/ZNSbEYd0mBCO5N/Xgg4jJ9GZjrEgxhvJijDoUvFhrDy1eAfZQMVQ HvgqmvRcTgieg82XMYyiYBSNKITyZ2w7vXBDQh9V/M+MUY3nct/g54ONnKmlLVTUirO4 Coqz6myrY6Dd4gKvwxfCNOygskUYVhfY3CUegTpARYa52OzU/wx4ZqL1LEmwRO8xQB9Y RtFMpo7juOf7gcOCrJ1WsMjB0BrF5AuBp9atvzTsRtPmhZZtUKekwCI9hrgtzZjybPHA Y9J5wPS7jnKcgkZ8LPuyuXT1vjPCqioA5iCzTBAztvl7Y9WL5FiCM/4YTMXuxysr6YFL KdIg== X-Gm-Message-State: AOJu0YxJvNjDhZPcxCWOxER6rSJtnFKYuPxXGRAOtx9NUhdk+RsHaPgS 6LM9HIO5jv7Aa91NbmoFkHBvzA== X-Received: by 2002:a6b:c885:0:b0:79f:8cd3:fd0e with SMTP id y127-20020a6bc885000000b0079f8cd3fd0emr2093182iof.1.1695320960509; Thu, 21 Sep 2023 11:29:20 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:19 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 5/8] futex: add wake_data to struct futex_q Date: Thu, 21 Sep 2023 12:29:05 -0600 Message-Id: <20230921182908.160080-6-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:41:18 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777698630261275177 X-GMAIL-MSGID: 1777698630261275177 With handling multiple futex_q for waitv, we cannot easily go from the futex_q to data related to that request or queue. Add a wake_data argument that belongs to the wake handler assigned. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 33835b81e0c3..76f6c2e0f539 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -148,6 +148,7 @@ typedef void (futex_wake_fn)(struct wake_q_head *wake_q, struct futex_q *q); * @task: the task waiting on the futex * @lock_ptr: the hash bucket lock * @wake: the wake handler for this queue + * @wake_data: data associated with the wake handler * @key: the key the futex is hashed on * @pi_state: optional priority inheritance state * @rt_waiter: rt_waiter storage for use with requeue_pi @@ -173,6 +174,7 @@ struct futex_q { struct task_struct *task; spinlock_t *lock_ptr; futex_wake_fn *wake; + void *wake_data; union futex_key key; struct futex_pi_state *pi_state; struct rt_mutex_waiter *rt_waiter; From patchwork Thu Sep 21 18:29:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 142837 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5063111vqi; Thu, 21 Sep 2023 11:46:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IER2cckFu/LsIYdD3A/veVLEzpwx0M649MdEvQC5WRRnW2gZ/1KuHjaFzztAnS3fvwLy1Un X-Received: by 2002:a17:90b:88e:b0:262:d6cb:3567 with SMTP id bj14-20020a17090b088e00b00262d6cb3567mr6510334pjb.26.1695321984953; Thu, 21 Sep 2023 11:46:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695321984; cv=none; d=google.com; s=arc-20160816; b=jtnbLTPDjj+2nObP1YDQmqvoTv3Q1yrAq0mqpqAxAlSQh6CCKVQtK1PG2Fqoq06JCs esmKQPycn1f9WB9db1ryuFd4gfY5hn7dkTGEF7b2n/oe5wJ6RCL4aDt0N92rofPNqtFu XMmFQDqYEhdsa7qCxIgezNEnKMcXkfESOfQzGB4c7VbJ382LeTxsxLHOfduYdXOO2a8P TIEh9gi6REbmIYviRvj2SnfcEa/5+EhFqrMgfDrRaexPoweaPImn3+ccyT5BZR7EJ2ys Jf4b3uivKAVe/yrDiYavyuUUzGFrA1PtDiV+D9nNJUG6N7ixFYklUHGLAecXRC05KfgM 198A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8/OGVEqmKZ+YrEmaYsT23Mong0SvETBYb//Qn3SwQyE=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=0+GDpDHTz0hBbYIjbYHLwnEJ9bSSfq8/atXgweNpmY79ZUCVhBla6m/X/uCTSyRe5Q onVr4S8xXqZm0zRZ867O1zh3ErH7S9gMynfmLQ+NKrYA+8FneE4EawYEbbvn4RMYuPuL KNa1Rfj/7py3cpo4RuRjxLsX8daNdFDb5fO0VqLGBAke2zDhPMLJVLZn48SR56vJ1r4G JQluZ+7Dgo2HWHuIv8jSOaq8PWc9xejd1rAMNCIQk2ZKeKwrZFeHKWpJo2R7sG6qnAwD vU8AA0yJTd0g61L0tGvnznh9UP3mZjgtE0lrVBsQW+j3B+E5NkgtFTpdvTnErvMxQK9a 5wdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=ouP89geP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id fy22-20020a17090b021600b0027660aac61asi4266067pjb.114.2023.09.21.11.46.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:46:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=ouP89geP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id B4D8982B6B6C; Thu, 21 Sep 2023 11:38:55 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230303AbjIUShz (ORCPT + 29 others); Thu, 21 Sep 2023 14:37:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229636AbjIUShZ (ORCPT ); Thu, 21 Sep 2023 14:37:25 -0400 Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77CB4D8685 for ; Thu, 21 Sep 2023 11:29:22 -0700 (PDT) Received: by mail-il1-x136.google.com with SMTP id e9e14a558f8ab-34fcb08d1d5so1009155ab.1 for ; Thu, 21 Sep 2023 11:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320962; x=1695925762; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8/OGVEqmKZ+YrEmaYsT23Mong0SvETBYb//Qn3SwQyE=; b=ouP89gePGLkXltybMGwkkustlfFAplUbJ4vP8U3TPcA3954wsu06QRtfnzwyvbYSYs QTVHkUdd32WVcNpJRncn7PZABjuOL6kSBVZA5/n2nwrpfc7UspeBjKI2sCPmEM3Kt5zW wHrwgAgk5ePJk4X5bYIyrevgGO5xLjj+P3TuE6Qn0T6aZvpAevB2QEv5ucKh70Y5YtCh /pwmK27WAxolaya57PNG4z/YSDrVzW8q3s7bm6oHwrL/c2etyw+ELu4SDQrAtVbwrzqd XjerkcEb5mbjB9u9WLAcyoW3KquEhxUKzGQ3PT9ROfv2DZiDl01XgicGqolS6PWTRO1d XnWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320962; x=1695925762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8/OGVEqmKZ+YrEmaYsT23Mong0SvETBYb//Qn3SwQyE=; b=D8EmFImjnAgIi1Ucf3oaqHoszP8dD2ggW2d7BUBI2MRguwV8f9mqKGEV9fNV36COPo pCxxCoNX2TfRFFrja1B9xIOUJxfWL7d6oGFZ1BY+DLWZtcHXducSh9nYZt22gORvq9or 4vdw5o3fIT6WyzVWMh/1+mvYf/P9EXvQzx8LnvtKocRyb3LgVOtBFAsNXWMiNfCPXUQ+ uG2JphHY14+rDOSJ2mpnmaV9zkwtg5bZP0KejdXNlHFQMxC1FWSGzqJa6gLZ7CEzA4I4 3nfrP35VqksvGHl2JvL68YkuqA48rasUY82uEuWDs+BduN2J+NcttQCZKdTbeIkGlTON VkqQ== X-Gm-Message-State: AOJu0YzADdMIom8ckhNSDzF+aACwHRIaj8XTZ+zQ+uKIekIhDUg64gMH 0wt34Me5XAzqOmP/tjthJw/koA2VwnYxtw7tiArfkw== X-Received: by 2002:a92:c52d:0:b0:34f:7ba2:50e8 with SMTP id m13-20020a92c52d000000b0034f7ba250e8mr6203745ili.2.1695320961817; Thu, 21 Sep 2023 11:29:21 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:21 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 6/8] futex: make futex_parse_waitv() available as a helper Date: Thu, 21 Sep 2023 12:29:06 -0600 Message-Id: <20230921182908.160080-7-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:38:55 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777673945650217185 X-GMAIL-MSGID: 1777673945650217185 To make it more generically useful, augment it with allowing the caller to pass in the wake handler and wake data. Convert the futex_waitv() syscall, passing in the default handlers. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 5 +++++ kernel/futex/syscalls.c | 16 +++++++++++----- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 76f6c2e0f539..6b6a6b3da103 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -361,6 +361,11 @@ struct futex_vector { struct futex_q q; }; +extern int futex_parse_waitv(struct futex_vector *futexv, + struct futex_waitv __user *uwaitv, + unsigned int nr_futexes, futex_wake_fn *wake, + void *wake_data); + extern int futex_wait_multiple(struct futex_vector *vs, unsigned int count, struct hrtimer_sleeper *to); diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index 2b5cafdfdc50..4b6da9116aa6 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -184,12 +184,15 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val, * @futexv: Kernel side list of waiters to be filled * @uwaitv: Userspace list to be parsed * @nr_futexes: Length of futexv + * @wake: Wake to call when futex is woken + * @wake_data: Data for the wake handler * * Return: Error code on failure, 0 on success */ -static int futex_parse_waitv(struct futex_vector *futexv, - struct futex_waitv __user *uwaitv, - unsigned int nr_futexes) +int futex_parse_waitv(struct futex_vector *futexv, + struct futex_waitv __user *uwaitv, + unsigned int nr_futexes, futex_wake_fn *wake, + void *wake_data) { struct futex_waitv aux; unsigned int i; @@ -214,6 +217,8 @@ static int futex_parse_waitv(struct futex_vector *futexv, futexv[i].w.val = aux.val; futexv[i].w.uaddr = aux.uaddr; futexv[i].q = futex_q_init; + futexv[i].q.wake = wake; + futexv[i].q.wake_data = wake_data; } return 0; @@ -306,7 +311,8 @@ SYSCALL_DEFINE5(futex_waitv, struct futex_waitv __user *, waiters, goto destroy_timer; } - ret = futex_parse_waitv(futexv, waiters, nr_futexes); + ret = futex_parse_waitv(futexv, waiters, nr_futexes, futex_wake_mark, + NULL); if (!ret) ret = futex_wait_multiple(futexv, nr_futexes, timeout ? &to : NULL); @@ -421,7 +427,7 @@ SYSCALL_DEFINE4(futex_requeue, if (!waiters) return -EINVAL; - ret = futex_parse_waitv(futexes, waiters, 2); + ret = futex_parse_waitv(futexes, waiters, 2, futex_wake_mark, NULL); if (ret) return ret; From patchwork Thu Sep 21 18:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 143129 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5288424vqi; Thu, 21 Sep 2023 20:07:12 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGjnsV5Fb3NgMjix9L4ABZvFHUiCsNA+EnywIs4KEcXrEZJubWLScVnvqpk/RozDPckgiZC X-Received: by 2002:a05:6870:c088:b0:1be:dfdf:cb1 with SMTP id c8-20020a056870c08800b001bedfdf0cb1mr8397064oad.46.1695352032398; Thu, 21 Sep 2023 20:07:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695352032; cv=none; d=google.com; s=arc-20160816; b=ukyeBGocCwaqbVBfjSExAqhX0pV3z4BYc3lZQYP0LT5IjaofomKqArGUHh1Rm5W9AA qtP/Yws+lTWddqXfSTtfXdnLkzQZ9Ekt62skDgpqgmir/nA0ye7ZO3nIVwnvU9n6qNcd h9DWYVOBrF13auKLsK0BtMAwHyadWRSOymwjxYajs05/28dYw4b/xVd96otc53OPO6IC ZznTjAFqW121Ftkzwz3APbqu4EBFYdR5/+ShbTeBQg9b6QwX8PlN2K6Nfshn2ARaE1is RtNDc+RWhAyEoPV5lRZXS5OZqtexJXZD6+01scHUJVoAGBGaJNgyydcjTGtWmEUr7YEs 8Phw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dMx5R6E0R0h9+pd+QZi7bIoYwQS8EWnvSyRr5gNA3m0=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=lozmJeN18IcOibLyIyzUN3zO/mEZzRuAUWgMUXOpJ+Ryw52QEEuhEaxUdqtRnYV4nH 7z5YGB9PXEwZhBj+Fonw5RXMjdst5YNea9W+/evAzc5eV56Qlnnv23SZVHXNzIdxUMn+ xFBmUhUiOYKYGU082po5/eDneBo2swOwqGXzrSkO3DiY5+41vm/GuH31BEyHXYvY5gp5 texgpoBYHuuCWtTOKoyWOTNGyP2WV7R3gWd0ZSic5QeDErQ1/0oocDP1QpJTNRANPp6q S9eUl/nml/V8AqLOecUeFZwgkWC9vZXMlY9PvjZn0JIC8JtzCNYME6hPI3gEIXu6YTm7 w4wA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=h5zvPxbW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id m127-20020a632685000000b00577461296a9si2988993pgm.338.2023.09.21.20.07.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 20:07:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=h5zvPxbW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 2F2D08275E7E; Thu, 21 Sep 2023 11:37:59 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230227AbjIUSh6 (ORCPT + 29 others); Thu, 21 Sep 2023 14:37:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229834AbjIUSh0 (ORCPT ); Thu, 21 Sep 2023 14:37:26 -0400 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7C0DD8687 for ; Thu, 21 Sep 2023 11:29:23 -0700 (PDT) Received: by mail-io1-xd2d.google.com with SMTP id ca18e2360f4ac-79f96830e4dso3976939f.1 for ; Thu, 21 Sep 2023 11:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320963; x=1695925763; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dMx5R6E0R0h9+pd+QZi7bIoYwQS8EWnvSyRr5gNA3m0=; b=h5zvPxbWi+KBRxy2l33P4Ql1YrEkI5Vx8JkaX3wnEgJgrTip2d0QcU61NZKFXB/ZT4 +bilLkvNep0yTxPjbCHKLlY5A6YkiwfKVWsjTCN7ksJCuRSiqvnfLqwvppLCoYEGJumu vuCHLIx2ifF+PsvY37r0fd4azPnN+TqB0WDk7j8Ihho6HeE5uWIJWsTkjqFQWxL2gWij 7Uts5c0xhdZOb0cKv63m1Xo+Sbkz1cEZaftBB+TDUcnxnj9XXp/JM4wEU87WAGG1HeNv MU/aMdT3NIEBJh+ofW+2cCCCpQUh/fgqQoD0xZQdHisLwUt1HeO9K4ncQph9yhGJNFP7 6ZYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320963; x=1695925763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dMx5R6E0R0h9+pd+QZi7bIoYwQS8EWnvSyRr5gNA3m0=; b=whuWQJxnJHIYDaUx0aZxNZyXZRLxE4WPqXR7tzgpd2o1bKmzRSMIf+zYVRfES78C6a F5AfE4jaaokBZFQREBofEzClZ9Td/0yUvKVCX8hTZ5frULb+uiwqisxd3nOMxxAgpEpf NaZbHpfUKJQe3xchWu3FDekGYDiI8Jxyj0PrpgkUskXUF8xLiifofb2DwQtTUJAX13L+ zgV7R3fowwkuMQcbclfAs2g+3h63YJaC8BegtRAxMGfh1p8Vd/VIVJdxrRbsIGOBzAS6 gM3P1BUrhLMw3xRGFLWYkjXpt66lWDiBxEVbPQNQLtZsXz93Qv6zypzdglMkBg8ZkNBv IByw== X-Gm-Message-State: AOJu0YzKEuD1FsHcjv9gtIxuvAHnfwr3kUjo7ViChV/q/aGRVDmZDm4U jqvSkBOcys+07PHw+1fIhW7Lyg== X-Received: by 2002:a05:6602:3788:b0:792:7c78:55be with SMTP id be8-20020a056602378800b007927c7855bemr6834507iob.0.1695320963064; Thu, 21 Sep 2023 11:29:23 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:22 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 7/8] futex: make the vectored futex operations available Date: Thu, 21 Sep 2023 12:29:07 -0600 Message-Id: <20230921182908.160080-8-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:37:59 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777705452877063137 X-GMAIL-MSGID: 1777705452877063137 Rename unqueue_multiple() as futex_unqueue_multiple(), and make both that and futex_wait_multiple_setup() available for external users. This is in preparation for wiring up vectored waits in io_uring. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 5 +++++ kernel/futex/waitwake.c | 10 +++++----- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 6b6a6b3da103..8b195d06f4e8 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -366,6 +366,11 @@ extern int futex_parse_waitv(struct futex_vector *futexv, unsigned int nr_futexes, futex_wake_fn *wake, void *wake_data); +extern int futex_wait_multiple_setup(struct futex_vector *vs, int count, + int *woken); + +extern int futex_unqueue_multiple(struct futex_vector *v, int count); + extern int futex_wait_multiple(struct futex_vector *vs, unsigned int count, struct hrtimer_sleeper *to); diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 6fcf5f723719..61b112897a84 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -372,7 +372,7 @@ void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, } /** - * unqueue_multiple - Remove various futexes from their hash bucket + * futex_unqueue_multiple - Remove various futexes from their hash bucket * @v: The list of futexes to unqueue * @count: Number of futexes in the list * @@ -382,7 +382,7 @@ void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, * - >=0 - Index of the last futex that was awoken; * - -1 - No futex was awoken */ -static int unqueue_multiple(struct futex_vector *v, int count) +int futex_unqueue_multiple(struct futex_vector *v, int count) { int ret = -1, i; @@ -410,7 +410,7 @@ static int unqueue_multiple(struct futex_vector *v, int count) * - 0 - Success * - <0 - -EFAULT, -EWOULDBLOCK or -EINVAL */ -static int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) +int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) { struct futex_hash_bucket *hb; bool retry = false; @@ -472,7 +472,7 @@ static int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *wo * was woken, we don't return error and return this index to * userspace */ - *woken = unqueue_multiple(vs, i); + *woken = futex_unqueue_multiple(vs, i); if (*woken >= 0) return 1; @@ -557,7 +557,7 @@ int futex_wait_multiple(struct futex_vector *vs, unsigned int count, __set_current_state(TASK_RUNNING); - ret = unqueue_multiple(vs, count); + ret = futex_unqueue_multiple(vs, count); if (ret >= 0) return ret; From patchwork Thu Sep 21 18:29:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 142865 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp5092633vqi; Thu, 21 Sep 2023 12:39:49 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFE/YvBx07Qb2EN7vmXDFwoHnELY/aK/pI9qAqvtkExG7lQgfo36t+DbNhdXh3PPdEZAXmI X-Received: by 2002:a17:90b:1002:b0:25d:eca9:1621 with SMTP id gm2-20020a17090b100200b0025deca91621mr6176578pjb.6.1695325189603; Thu, 21 Sep 2023 12:39:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695325189; cv=none; d=google.com; s=arc-20160816; b=j7dv1eoSoqWuv/ggYwXLgsaxkt2XW55GBxmG+kNWVqjzUIojJDcCYM4r7TMTuh84xh GIluwOUGLiHCoyfe9Pj0WFH7aSQm2fu3+NK4NiLjV6UzsEb7XjWVBAzPcs2KKvJFO7Du jGAQPi9ajmb5Xk9or5oShHnFlY7r77nHOcOu2cayMeaHCVJ+9eB+lgEoLqM2u8+AdSPf +h6jbTAOeZUAiCBWluGRobqr/Tq8cQkClCusQcrO+ELyDikXXFSpOElNeCZwHKJYsHRm Pzr4cnBNCZGNLecU4ffIqghYJqs791Ka6foYmJJIet1iNLnZsNDK/OfAkHfPJQ93gRsn kLOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T07+FrfCPjZBo72MwSk6fDllH6IdPm+PDxNrxWagj38=; fh=McWMhBXbZKPJudkA+aRuAP3u2dQeraI6k1jPJeJNhbk=; b=0c0djg2YLkmP4AlMhwjrSWWLTi6OdaaMkq/quQGwwVmR2gdv5PQShk9591h+Mu66Ro 1yOAj+sW5eNxHVsfZQAYeh33W+L7GcTJbkCoMkM5rzRMyv50F4OXYWehjKFFTHpSd388 p4hShG4c0anvJKFN+/VEx8EMs0NHPrGt/NLdC936SI5mavwnpc9WMjKB2+w629oUBmA0 tJOdOtGE2Aw1yWViHVGuWYNdy80cmUSsCodblMujBOZChOD0IIqkyI4WbBLvfBG/rKyY kT1pRDpSmTYRaHmwcc5uvR10esvYiwFXgigZCokjVg+Rk5bgb8sC/2jCCQc+sdBBUvuS vSZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=XjA2LfKP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id r60-20020a17090a43c200b002534f4ce2b6si4345287pjg.125.2023.09.21.12.39.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 12:39:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=XjA2LfKP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 724C7837E9BF; Thu, 21 Sep 2023 11:38:07 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230024AbjIUSiA (ORCPT + 29 others); Thu, 21 Sep 2023 14:38:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53344 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230059AbjIUSh0 (ORCPT ); Thu, 21 Sep 2023 14:37:26 -0400 Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30C46D868E for ; Thu, 21 Sep 2023 11:29:25 -0700 (PDT) Received: by mail-io1-xd31.google.com with SMTP id ca18e2360f4ac-79f96830e4dso3977839f.1 for ; Thu, 21 Sep 2023 11:29:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1695320964; x=1695925764; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T07+FrfCPjZBo72MwSk6fDllH6IdPm+PDxNrxWagj38=; b=XjA2LfKPRX+SU4/H3+AzQrupIHosj67NBFlTc+tdPDTcz80R3VLJdTkJbxnt7UBDUQ vUX1tgdxThvBw1zkH5+1zRqOkiW4z0wlx0ktvE6DHny6BykYkJAAhvWA+ubOtrpuL/8u cybB1u1BzyVqBz3TRKvMJmNixNnUZ9XEuariQQY00UzjikcFKxckzQjfqrD61y+fCmrl f4nRBls5SBjvucuEynYf1z4b2BR8c+FR071nsUQ+sDX752TYzPIPKeJGS4nxQtQZml7u E/54/7GOVAbRqLb74PFGj9B71Phh+fb9NMX+WJgyfCItr4EJTsHAGf4D+VxxHFzuTo9j vHXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695320964; x=1695925764; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T07+FrfCPjZBo72MwSk6fDllH6IdPm+PDxNrxWagj38=; b=O6arO8B9TDS89sWzmqx6APaQjdQC1LdYurIO8+eFEfR8G+0Wgl9Ub3U7u4aonGehzl rkoM8pLyhaUsQuuvFuAeFVGshp1CGAVNTGAXMLzr7HkMjcEGPElHZswcXxSYoqcwMyhN dnvQkQVhSCSSIWyflFHUylyDqPEJ9vtKs5IATmZ0pvoxZqbb45mSxrb2q9O5NcUHqEj+ p+J+rCxjOGYuwLMcCufzPymAy3ftBh9gbJmcpvvEi0lGb303f6GwsWNhEN0GG3c3wgm3 5rhRHX1okstFEmusPf8b7yubCTn7+gET02BYYay1R8iYd8rX6O0ZpVwgGc3r17fs5WGq yVsw== X-Gm-Message-State: AOJu0YxX18Hmmi1u18zPGIP/EGvi9VInzHy8RY7gmgpAtRHKy8l+hxYV iQgeqtFA9/bWQtVWTz1pATUNlQ== X-Received: by 2002:a05:6602:13c2:b0:79d:1c65:9bde with SMTP id o2-20020a05660213c200b0079d1c659bdemr8647351iov.1.1695320964457; Thu, 21 Sep 2023 11:29:24 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id o25-20020a02c6b9000000b0042b227eb1ddsm500441jan.55.2023.09.21.11.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Sep 2023 11:29:23 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 8/8] io_uring: add support for vectored futex waits Date: Thu, 21 Sep 2023 12:29:08 -0600 Message-Id: <20230921182908.160080-9-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230921182908.160080-1-axboe@kernel.dk> References: <20230921182908.160080-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 21 Sep 2023 11:38:07 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777677305557267220 X-GMAIL-MSGID: 1777677305557267220 This adds support for IORING_OP_FUTEX_WAITV, which allows registering a notification for a number of futexes at once. If one of the futexes are woken, then the request will complete with the index of the futex that got woken as the result. This is identical to what the normal vectored futex waitv operation does. Use like IORING_OP_FUTEX_WAIT, except sqe->addr must now contain the a pointer to a struct futex_waitv array, and sqe->off must now contain the number of elements in that array. For cancelations, FUTEX_WAITV does not rely on the futex_unqueue() return value as we're dealing with multiple futexes. Instead, a separate per io_uring request atomic is used to claim ownership of the request. Waiting on N futexes could be done with IORING_OP_FUTEX_WAIT as well, but that punts a lot of the work to the application: 1) Application would need to submit N IORING_OP_FUTEX_WAIT requests, rather than just a single IORING_OP_FUTEX_WAITV. 2) When one futex is woken, application would need to cancel the remaining N-1 requests that didn't trigger. While this is of course doable, having a single vectored futex wait makes for much simpler application code. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 1 + io_uring/futex.c | 164 ++++++++++++++++++++++++++++++++-- io_uring/futex.h | 2 + io_uring/opdef.c | 11 +++ 4 files changed, 169 insertions(+), 9 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 4ddd7bdbbfb8..172472626f5b 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -246,6 +246,7 @@ enum io_uring_op { IORING_OP_WAITID, IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAKE, + IORING_OP_FUTEX_WAITV, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/futex.c b/io_uring/futex.c index 4278302d212c..0c07df8668aa 100644 --- a/io_uring/futex.c +++ b/io_uring/futex.c @@ -14,10 +14,15 @@ struct io_futex { struct file *file; - u32 __user *uaddr; + union { + u32 __user *uaddr; + struct futex_waitv __user *uwaitv; + }; unsigned long futex_val; unsigned long futex_mask; + unsigned long futexv_owned; u32 futex_flags; + unsigned int futex_nr; }; struct io_futex_data { @@ -44,6 +49,13 @@ void io_futex_cache_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->futex_cache, io_futex_cache_entry_free); } +static void __io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) +{ + req->async_data = NULL; + hlist_del_init(&req->hash_node); + io_req_task_complete(req, ts); +} + static void io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) { struct io_futex_data *ifd = req->async_data; @@ -52,22 +64,59 @@ static void io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) io_tw_lock(ctx, ts); if (!io_alloc_cache_put(&ctx->futex_cache, &ifd->cache)) kfree(ifd); - req->async_data = NULL; - hlist_del_init(&req->hash_node); - io_req_task_complete(req, ts); + __io_futex_complete(req, ts); } -static bool __io_futex_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) +static void io_futexv_complete(struct io_kiocb *req, struct io_tw_state *ts) { - struct io_futex_data *ifd = req->async_data; + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct futex_vector *futexv = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + int res = 0; - /* futex wake already done or in progress */ - if (!futex_unqueue(&ifd->q)) + io_tw_lock(ctx, ts); + + res = futex_unqueue_multiple(futexv, iof->futex_nr); + if (res != -1) + io_req_set_res(req, res, 0); + + kfree(req->async_data); + req->flags &= ~REQ_F_ASYNC_DATA; + __io_futex_complete(req, ts); +} + +static bool io_futexv_claimed(struct io_futex *iof) +{ + return test_bit(0, &iof->futexv_owned); +} + +static bool io_futexv_claim(struct io_futex *iof) +{ + if (test_bit(0, &iof->futexv_owned) || + test_and_set_bit(0, &iof->futexv_owned)) return false; + return true; +} + +static bool __io_futex_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) +{ + /* futex wake already done or in progress */ + if (req->opcode == IORING_OP_FUTEX_WAIT) { + struct io_futex_data *ifd = req->async_data; + + if (!futex_unqueue(&ifd->q)) + return false; + req->io_task_work.func = io_futex_complete; + } else { + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + + if (!io_futexv_claim(iof)) + return false; + req->io_task_work.func = io_futexv_complete; + } hlist_del_init(&req->hash_node); io_req_set_res(req, -ECANCELED, 0); - req->io_task_work.func = io_futex_complete; io_req_task_work_add(req); return true; } @@ -146,6 +195,54 @@ int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static void io_futex_wakev_fn(struct wake_q_head *wake_q, struct futex_q *q) +{ + struct io_kiocb *req = q->wake_data; + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + + if (!io_futexv_claim(iof)) + return; + if (unlikely(!__futex_wake_mark(q))) + return; + + io_req_set_res(req, 0, 0); + req->io_task_work.func = io_futexv_complete; + io_req_task_work_add(req); +} + +int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct futex_vector *futexv; + int ret; + + /* No flags or mask supported for waitv */ + if (unlikely(sqe->fd || sqe->buf_index || sqe->file_index || + sqe->addr2 || sqe->addr3)) + return -EINVAL; + + iof->uaddr = u64_to_user_ptr(READ_ONCE(sqe->addr)); + iof->futex_nr = READ_ONCE(sqe->len); + if (!iof->futex_nr || iof->futex_nr > FUTEX_WAITV_MAX) + return -EINVAL; + + futexv = kcalloc(iof->futex_nr, sizeof(*futexv), GFP_KERNEL); + if (!futexv) + return -ENOMEM; + + ret = futex_parse_waitv(futexv, iof->uwaitv, iof->futex_nr, + io_futex_wakev_fn, req); + if (ret) { + kfree(futexv); + return ret; + } + + iof->futexv_owned = 0; + req->flags |= REQ_F_ASYNC_DATA; + req->async_data = futexv; + return 0; +} + static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q) { struct io_futex_data *ifd = container_of(q, struct io_futex_data, q); @@ -170,6 +267,55 @@ static struct io_futex_data *io_alloc_ifd(struct io_ring_ctx *ctx) return kmalloc(sizeof(struct io_futex_data), GFP_NOWAIT); } +int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct futex_vector *futexv = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + int ret, woken = -1; + + io_ring_submit_lock(ctx, issue_flags); + + ret = futex_wait_multiple_setup(futexv, iof->futex_nr, &woken); + + /* + * The above call leaves us potentially non-running. This is fine + * for the sync syscall as it'll be blocking unless we already got + * one of the futexes woken, but it obviously won't work for an async + * invocation. Mark us runnable again. + */ + __set_current_state(TASK_RUNNING); + + /* + * We got woken while setting up, let that side do the completion + */ + if (io_futexv_claimed(iof)) { +skip: + io_ring_submit_unlock(ctx, issue_flags); + return IOU_ISSUE_SKIP_COMPLETE; + } + + /* + * 0 return means that we successfully setup the waiters, and that + * nobody triggered a wakeup while we were doing so. < 0 or 1 return + * is either an error or we got a wakeup while setting up. + */ + if (!ret) { + hlist_add_head(&req->hash_node, &ctx->futex_list); + goto skip; + } + + io_ring_submit_unlock(ctx, issue_flags); + if (ret < 0) + req_set_fail(req); + else if (woken != -1) + ret = woken; + io_req_set_res(req, ret, 0); + kfree(futexv); + req->flags &= ~REQ_F_ASYNC_DATA; + return IOU_OK; +} + int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); diff --git a/io_uring/futex.h b/io_uring/futex.h index ddc9e0d73c52..0847e9e8a127 100644 --- a/io_uring/futex.h +++ b/io_uring/futex.h @@ -3,7 +3,9 @@ #include "cancel.h" int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags); +int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags); int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags); #if defined(CONFIG_FUTEX) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 31a3a421e94d..25a3515a177c 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -459,6 +459,14 @@ const struct io_issue_def io_issue_defs[] = { .issue = io_futex_wake, #else .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_FUTEX_WAITV] = { +#if defined(CONFIG_FUTEX) + .prep = io_futexv_prep, + .issue = io_futexv_wait, +#else + .prep = io_eopnotsupp_prep, #endif }, }; @@ -693,6 +701,9 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_FUTEX_WAKE] = { .name = "FUTEX_WAKE", }, + [IORING_OP_FUTEX_WAITV] = { + .name = "FUTEX_WAITV", + }, }; const char *io_uring_get_opcode(u8 opcode)