From patchwork Fri Jul 28 16:42:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127809 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp610823vqg; Fri, 28 Jul 2023 11:13:49 -0700 (PDT) X-Google-Smtp-Source: APBJJlEWS0KanCxeCpXSBG9SAwmVJC1HJ/UwSAzHe13EazWshKa7gqY9NZdzHrSWws/iW/3/JRSb X-Received: by 2002:a05:6a00:1989:b0:67e:6269:6ea8 with SMTP id d9-20020a056a00198900b0067e62696ea8mr3560932pfl.22.1690568029423; Fri, 28 Jul 2023 11:13:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690568029; cv=none; d=google.com; s=arc-20160816; b=yvJBZ3sPoClw32RMCXEIHeBon1Rkq3W4rzw/710hmJrzroAVD4pd7vavN/02G/6XUe mZo4kTM7p+7SO0eQ/jcZyo+Ls51j9FyGG1wNL0KNUBlYBCfvzYJpF/V6HJOPMYQlBeay wAEX3O9mkFQb7vMY4rcH0ZSTAp2Jyn5u07zE5egEJOPS+J2lj1bd6U1ZBBdewYD+70xA k+kaRAFtWazN51uZ5f0xkrcfJGvytu8nADDnS8QE+kQCZrKZxqnPosvkTbTmgsT3eu/H OPHnGpyPwyKTx98j/YucXrUy5GBI+whuAjAXhxwAhuwf7B3duyNc8YI4Xm+ef8ntMD0b hy1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qfXIoWZOPMTUuZ51sHKPtDeshXixERaT1oNJnhO0ex0=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=lfyHbIK3mEIoN3Aln0iFDSp+mDVmT/9qzmSzBWlFpHJ+aoXb1J2h31/BkQkAM9v+p7 wwf3xnpEX2UTdpO5gkMZo7KzfcEbUYkW+X0OshJp/HUO9qqznSFNsjEPrI/aA5XIPTSr XdKmCsG8lZspbtZMKCiZ5oGRZmwcuMvybri4WUiglFcYirVjRTza0G/gsh+I04c2LTQi 9rGPcdgH7fvZfsJmd1C4CKHcXJcLJxobzGLqPM7oCzZcIXYj6IvEQQjNfCMDdnmmlcVA qNMT/2CQsMYzWvFXiLhf5FUKiPQxEbl5+sl6PLxrVgCf4Cz522KoR6EJLlFhx2ydKOx7 kbSg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=It3a8uIR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 34-20020a631862000000b0053b928deb7asi3322762pgy.344.2023.07.28.11.13.36; Fri, 28 Jul 2023 11:13:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=It3a8uIR; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233128AbjG1Qms (ORCPT + 99 others); Fri, 28 Jul 2023 12:42:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229564AbjG1Qmm (ORCPT ); Fri, 28 Jul 2023 12:42:42 -0400 Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E952219A4 for ; Fri, 28 Jul 2023 09:42:40 -0700 (PDT) Received: by mail-io1-xd2d.google.com with SMTP id ca18e2360f4ac-78706966220so22787239f.1 for ; Fri, 28 Jul 2023 09:42:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562560; x=1691167360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qfXIoWZOPMTUuZ51sHKPtDeshXixERaT1oNJnhO0ex0=; b=It3a8uIRBWBxtFna+HZJERO/3/0f2r4nyuR/E+sjXiG9TQGNLD0TfIGecOO32rhhu/ nQ0XUBk8WmxIZY4qJQaI/VevgEsW/CwC4nLk8hoIU8YxvQl3AwGLhkgFzwIi4tKY+8su Oyl/hwKdVV/FUPnGD472zaZePQjaJGgeRkhyHLD/8mRuCDBX3cOvi9Y040oyMhHLRBrG RlhjY65j/7wF/feBfAD43VQKB3zPIAz3pDn5QXxTz+Cy61LdylA3/pmWHZ7uMGveK7ci pMkLlV0TBFxQdPM9H7ZjTdWt8u9Brkuy2GgG3dCpn0Vy2XHOmVX/Rz+RtnHNAt8sddmh 1lJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562560; x=1691167360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qfXIoWZOPMTUuZ51sHKPtDeshXixERaT1oNJnhO0ex0=; b=FiJcTD0YOS1G6jiYe9vwx1DdoMjemrU1Cb3eui4nSsVb5WaLDk6dcsWc9uXxEl2RzZ YaQTB+31mqQ36xBpnJxLAm9XdwIg18hRiTOBmFw6jk8JB6SzP9xCg5pOGA+X36MrELYW hMOTApe/qtelCXfKJEeVcoITugyuFje/DHKzXbJ1qyXax7v+Oc0AnrCozmRcMaYiwcTj E/jitCB8C0WjOYtDvOMsfa7xYz7DJ7aJJQYdR0qAdfbIsD7KRYxLoVHNJ620AlnyYA2I xVtA/axfo6LWYzkcCZQSeSN2nXIOjtnNtvW7gAkK5km0/0U2LKpEfvuSJA1i09c0IhGk Vm9Q== X-Gm-Message-State: ABy/qLaCmazsyTw5GYgOfP8XInk0LSNTpOq0shqYWAekrRKCI6AbLiJO UVhLY9aORmLMqI9NimOUSGmAiA== X-Received: by 2002:a05:6602:14ce:b0:787:16ec:2699 with SMTP id b14-20020a05660214ce00b0078716ec2699mr113690iow.2.1690562560150; Fri, 28 Jul 2023 09:42:40 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:39 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 01/12] futex: Clarify FUTEX2 flags Date: Fri, 28 Jul 2023 10:42:24 -0600 Message-Id: <20230728164235.1318118-2-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772689061880538732 X-GMAIL-MSGID: 1772689061880538732 From: Peter Zijlstra sys_futex_waitv() is part of the futex2 series (the first and only so far) of syscalls and has a flags field per futex (as opposed to flags being encoded in the futex op). This new flags field has a new namespace, which unfortunately isn't super explicit. Notably it currently takes FUTEX_32 and FUTEX_PRIVATE_FLAG. Introduce the FUTEX2 namespace to clarify this Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Jens Axboe --- include/uapi/linux/futex.h | 16 +++++++++++++--- kernel/futex/syscalls.c | 7 +++---- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/include/uapi/linux/futex.h b/include/uapi/linux/futex.h index 71a5df8d2689..0c5abb6aa8f8 100644 --- a/include/uapi/linux/futex.h +++ b/include/uapi/linux/futex.h @@ -44,10 +44,20 @@ FUTEX_PRIVATE_FLAG) /* - * Flags to specify the bit length of the futex word for futex2 syscalls. - * Currently, only 32 is supported. + * Flags for futex2 syscalls. */ -#define FUTEX_32 2 + /* 0x00 */ + /* 0x01 */ +#define FUTEX2_32 0x02 + /* 0x04 */ + /* 0x08 */ + /* 0x10 */ + /* 0x20 */ + /* 0x40 */ +#define FUTEX2_PRIVATE FUTEX_PRIVATE_FLAG + +/* do not use */ +#define FUTEX_32 FUTEX2_32 /* historical accident :-( */ /* * Max numbers of elements in a futex_waitv array diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index a8074079b09e..42b6c2fac7db 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -183,8 +183,7 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val, return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3); } -/* Mask of available flags for each futex in futex_waitv list */ -#define FUTEXV_WAITER_MASK (FUTEX_32 | FUTEX_PRIVATE_FLAG) +#define FUTEX2_MASK (FUTEX2_32 | FUTEX2_PRIVATE) /** * futex_parse_waitv - Parse a waitv array from userspace @@ -205,10 +204,10 @@ static int futex_parse_waitv(struct futex_vector *futexv, if (copy_from_user(&aux, &uwaitv[i], sizeof(aux))) return -EFAULT; - if ((aux.flags & ~FUTEXV_WAITER_MASK) || aux.__reserved) + if ((aux.flags & ~FUTEX2_MASK) || aux.__reserved) return -EINVAL; - if (!(aux.flags & FUTEX_32)) + if (!(aux.flags & FUTEX2_32)) return -EINVAL; futexv[i].w.flags = aux.flags; From patchwork Fri Jul 28 16:42:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127784 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp589648vqg; Fri, 28 Jul 2023 10:32:51 -0700 (PDT) X-Google-Smtp-Source: APBJJlHth9yoQkFrytxkbiFqm8Y/tmJ944sAcGBXys3OS6X2daNFipXEmeJoQe+7XCXXjOfr5oQh X-Received: by 2002:a05:6a00:21cc:b0:686:be77:4326 with SMTP id t12-20020a056a0021cc00b00686be774326mr3184488pfj.34.1690565571575; Fri, 28 Jul 2023 10:32:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690565571; cv=none; d=google.com; s=arc-20160816; b=UGJFCWZxr+dvG/F8VmDEZuXhb5NvTd18eivvlqXuo4+DONym/3qLTNqar1S4MlWRoz KDmMMaIz+mSpwvVLBewNzOBkSyLgyruqLJ0oDZmBrQjvw8KhEWfwyYkbiTpM5USOqL2s OEpNnKZo6eq0jmZYa3ua6SwUGP8LHDkP31QuYvC8hxkyKxugp5tRPytbtcavlu2YuIrX zsE7WU/y9I10/A7X6ClNC5h2DzpYJhvjPdZmSapwix8X5d6jtdoRP8NfZTv+rMMCWEsa cMNKtlh5EGLZMmapNl6Ot4HbrDr7e5oMh6y0uHzzsB4rzkA/aXhrkFkLpNlxJe9SoUQ5 oMwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mapQj7LodZYltGHkh10jKsE+2rfFqJFkkugF8pstoFg=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=EvDM20emnkNRS88TfNhZLkO8mbePzS23CwSANmerSdquwCsctpPkcR1/s48jhZ4e37 V1LjScnV7kUsz3YiF0H1FbGTDq4dJk3dxdktUQ8Wg10pAQrmoS3YBF+mbYNl3LhMMs6u AQZRnjdfjkIvA2vMllhT84mKil4qj+RMFINGweiQYCJFEvhVH/R+kGVGu51+uqBNk3Pt WhMmgRwW7KbfRHa9vu3IBV2MQy89JdT1scsVGYGVhir0FbCO5rSj9ZyEdd4XjnLQjbQF TDGDtN+qI4r6aj6vqc5FCMU/CnpedD7oL/VmZuAixFQy09Q+uUWQIcGuYZmBkHX7farZ 8J6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=a5mW98D1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ds13-20020a056a004acd00b00678e14c629bsi3390577pfb.401.2023.07.28.10.32.38; Fri, 28 Jul 2023 10:32:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=a5mW98D1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232862AbjG1Qmu (ORCPT + 99 others); Fri, 28 Jul 2023 12:42:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231842AbjG1Qmm (ORCPT ); Fri, 28 Jul 2023 12:42:42 -0400 Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9AB5E2 for ; Fri, 28 Jul 2023 09:42:41 -0700 (PDT) Received: by mail-io1-xd2f.google.com with SMTP id ca18e2360f4ac-7835bbeb6a0so33275739f.0 for ; Fri, 28 Jul 2023 09:42:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562561; x=1691167361; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mapQj7LodZYltGHkh10jKsE+2rfFqJFkkugF8pstoFg=; b=a5mW98D1BzTHh5iFih+TUp7Ky3zQGTmEErih779mOBmxyohbEapKqcyDSrGispTk21 jSoP4Uh17S731GmqrrDr/xbGQXHc1J0XtFq0HhSskJWNvzceY0d6aXiDqK9lFKTELDiK aO5YTfkQXivC74Siejd+JilC+/eIVGqXgzVA5tiYrcisNLEZfk94gycBdIg+2H/6wIxN CUNj4hAmpA2kcAugSwMEnJRZsrRiKlrAE7s21tJpbQpZ54JoyDDg19esxjVicrFHbvOz WSwlWkEnIUrfa8SqdB8I341LZwCNuy5X4c2uV46d4DVa4NXQfs5YbiRQXL/Hw6BBi83j SzqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562561; x=1691167361; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mapQj7LodZYltGHkh10jKsE+2rfFqJFkkugF8pstoFg=; b=BUPOnIjZYHQdcDTVXGTsMX64xVWv2s7IVAZq8YitVAhCrExajnZ79WY+YhgXYW86zz RD5W958a8T03WqB2VsXWI3zhNtP/HVM1YWU+756E+eTysCKHYJQkh+sDa6BIWTj8iTVw ozmEGXzkWKZofSY3HsfnTsEsHbS69RfB2pHjLPGWW3hXx8bkfZ6gXEfW8igEvFO/pTMI 781bSUkAYmr01rR2l27O52ZxRLDTDBatRmrz1juGUZzvzuZ0Dwijh8Fr2C0mVn0DpKzF GraVvgH1aydC+XDVEwc6HG7kF+0KA/IhL9u6eZV0uyV68qtYlTs26oXvL/b0JvERtKvF bZmA== X-Gm-Message-State: ABy/qLagmzMYH8Q889zYgLAVBVDxvumoo5EMQYE8rmJVZxgCH1Mr8NFh hDH7//hITmP3ltrr0tinXwQAeg== X-Received: by 2002:a05:6602:1543:b0:780:c6bb:ad8d with SMTP id h3-20020a056602154300b00780c6bbad8dmr149421iow.0.1690562561063; Fri, 28 Jul 2023 09:42:41 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:40 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 02/12] futex: Extend the FUTEX2 flags Date: Fri, 28 Jul 2023 10:42:25 -0600 Message-Id: <20230728164235.1318118-3-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772686484390336514 X-GMAIL-MSGID: 1772686484390336514 From: Peter Zijlstra Add the definition for the missing but always intended extra sizes, and add a NUMA flag for the planned numa extention. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Jens Axboe --- include/uapi/linux/futex.h | 7 ++++--- kernel/futex/syscalls.c | 9 +++++++-- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/uapi/linux/futex.h b/include/uapi/linux/futex.h index 0c5abb6aa8f8..0ed021acc1d9 100644 --- a/include/uapi/linux/futex.h +++ b/include/uapi/linux/futex.h @@ -46,10 +46,11 @@ /* * Flags for futex2 syscalls. */ - /* 0x00 */ - /* 0x01 */ +#define FUTEX2_8 0x00 +#define FUTEX2_16 0x01 #define FUTEX2_32 0x02 - /* 0x04 */ +#define FUTEX2_64 0x03 +#define FUTEX2_NUMA 0x04 /* 0x08 */ /* 0x10 */ /* 0x20 */ diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index 42b6c2fac7db..cfc8001e88ea 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -183,7 +183,7 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val, return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3); } -#define FUTEX2_MASK (FUTEX2_32 | FUTEX2_PRIVATE) +#define FUTEX2_MASK (FUTEX2_64 | FUTEX2_PRIVATE) /** * futex_parse_waitv - Parse a waitv array from userspace @@ -207,7 +207,12 @@ static int futex_parse_waitv(struct futex_vector *futexv, if ((aux.flags & ~FUTEX2_MASK) || aux.__reserved) return -EINVAL; - if (!(aux.flags & FUTEX2_32)) + if (!IS_ENABLED(CONFIG_64BIT) || in_compat_syscall()) { + if ((aux.flags & FUTEX2_64) == FUTEX2_64) + return -EINVAL; + } + + if ((aux.flags & FUTEX2_64) != FUTEX2_32) return -EINVAL; futexv[i].w.flags = aux.flags; From patchwork Fri Jul 28 16:42:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127808 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp608681vqg; Fri, 28 Jul 2023 11:09:49 -0700 (PDT) X-Google-Smtp-Source: APBJJlEh4HBq1gmU+jfWk5wUTB3GplAUlxHXtWpkTDJVQi2+mlFwQkKLiws0OD8tYFEzQImAKe9l X-Received: by 2002:a9d:7d85:0:b0:6bc:8aca:ae53 with SMTP id j5-20020a9d7d85000000b006bc8acaae53mr2112298otn.12.1690567789617; Fri, 28 Jul 2023 11:09:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690567789; cv=none; d=google.com; s=arc-20160816; b=jIL2mRndiaChqIzHXFyHynNWifVlj7yxdYhHLxRz1Jv2gIWH7Klxkfqt3Xyd/yV0Cq lahCo0hPruHfwzUK3h27xaQLsLepLhx7jdVEITQSRHSKsDk5Kqzn1Cl+OvGhIi2OVec9 hNpjDfI+F52Ef0e4Ol5XhZpdZvofZQsIU4l1RdCN36nQksNGI6WiO4HMqQWfWBnJR2zs CFzAo0GXyTV6cwFSaSqJbWoK/WjAW3EkdRdiKuyM9IOceBtNyMv2WmnB7a0stibVnssz vhBESD9qq1mM1z79Z3TBzXPk3ub5TCpo9asA1EUXk/pUtMTGN41SjyOEeHZZs2QvVwfS EhHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=30eD9fAtYTDVQhSBmDtrnQoUWRd4/P+cYE3Me4cIvhU=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=Wbgxx06S3z9+PW/GfElvyUh63iv5pI44SNFUW4Y6ydi9gVL7dRfP4uIBUOBXlIoABb xRHpeYiopJfOOdxH+09l5LDakw34RSTi0jBkdBGGrRkHIo33TfYJTF74s6cHTgseyKQn i+Leg4ycGcZNOrb5s8+73uRQbSfAFnNYDLCOgtKmR0ncIeBY/cWRL/d7uNhsP2CPdceS egdhLQ3kl7KvtCqanLY8b/wgdtgrr+WgVI32RuKTVC/UtCuIR5QtthMJASgBDvhvQEAi 8yLhgsaUIHn0ot0RBn8V+cjx17ZrkMbXBQNCgMryE9K1NPIlyH4ardeQYiXU+8NLBhBh vWKA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=DKvcWZja; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h123-20020a636c81000000b0056406432f69si2645361pgc.132.2023.07.28.11.09.36; Fri, 28 Jul 2023 11:09:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=DKvcWZja; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233236AbjG1Qmz (ORCPT + 99 others); Fri, 28 Jul 2023 12:42:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231183AbjG1Qmr (ORCPT ); Fri, 28 Jul 2023 12:42:47 -0400 Received: from mail-il1-x12a.google.com (mail-il1-x12a.google.com [IPv6:2607:f8b0:4864:20::12a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 85EB910F9 for ; Fri, 28 Jul 2023 09:42:42 -0700 (PDT) Received: by mail-il1-x12a.google.com with SMTP id e9e14a558f8ab-348d1c94fdaso1755685ab.1 for ; Fri, 28 Jul 2023 09:42:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562562; x=1691167362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=30eD9fAtYTDVQhSBmDtrnQoUWRd4/P+cYE3Me4cIvhU=; b=DKvcWZja9ydTXsU5tYZC8ODhETIbn5KR9hRed/c2NdhL4yY3uygRbDb5dDpGPIZmNR BlNzu5WNQYhZkw7Vlb+FJoMxxXJp2Rs9l5K32ZlR+xTZxOdARQ2RlK7VCgor7RzgPYFL tBNfXpcdQakAIi1W+bqEbixou2yk9uRPNjHpe9XZhRAGRiB9SGwK6GhDpm1IAzvAp2SA /xrm5m8vmVTqcRlgSiq84QIj6f/DaB7sXCbDtCd/wccBuIU80cWHRxPzjuALtRcngYb3 48Ut2CMMr0NQruQqc+pA4QQuKwIgyCIGJXIbowoUi0VKb4Pzgac8YwfH+U1niBycADs0 eQpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562562; x=1691167362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=30eD9fAtYTDVQhSBmDtrnQoUWRd4/P+cYE3Me4cIvhU=; b=IBjEHReF0w1HzPWcaDnZSaLY/vXNsewM7tRVOyiM3id5hNeVELKL/j3mF89cicYaHE Kw9C4Umq8wgZA85yUkkTQCWp26g8M11IPWXXq3zStqbD0PGEBMUdnBP8XrkIXrit0ceb kinkNV/PF0bz0H0kkqnH3fccKtXs5AK/RfgEotH9YFKNe36BE9fXxgbhZ2AZ5NjZFpAW bEvIakteYycenJ2aFzbjoWjjZdCZMiE8Z+tDucYFDk4h9+4xIt9w8xldB0R7ZD5NGtFm OyKmg4pbTdhAsjjoYAF/q73hyFnA2fi2z6jjt++ALWxKHZl6KDyVmnB2aFoSezWnjVSF vUNw== X-Gm-Message-State: ABy/qLammGZ15/wy4CPb42nzqmChtVnfYL3cqZdL+hBU5vnLZC5n4J2p w8+OcNFErQmJabXfKUamO5pz1mr+y3zveKJagK4= X-Received: by 2002:a92:c64f:0:b0:346:10c5:2949 with SMTP id 15-20020a92c64f000000b0034610c52949mr66624ill.1.1690562561891; Fri, 28 Jul 2023 09:42:41 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:41 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 03/12] futex: Flag conversion Date: Fri, 28 Jul 2023 10:42:26 -0600 Message-Id: <20230728164235.1318118-4-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772688810965133076 X-GMAIL-MSGID: 1772688810965133076 From: Peter Zijlstra Futex has 3 sets of flags: - legacy futex op bits - futex2 flags - internal flags Add a few helpers to convert from the API flags into the internal flags. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 64 +++++++++++++++++++++++++++++++++++++++-- kernel/futex/syscalls.c | 24 ++++++---------- kernel/futex/waitwake.c | 4 +-- 3 files changed, 72 insertions(+), 20 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index b5379c0e6d6d..c0e04599904a 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef CONFIG_PREEMPT_RT #include @@ -16,8 +17,15 @@ * Futex flags used to encode options to functions and preserve them across * restarts. */ +#define FLAGS_SIZE_8 0x00 +#define FLAGS_SIZE_16 0x01 +#define FLAGS_SIZE_32 0x02 +#define FLAGS_SIZE_64 0x03 + +#define FLAGS_SIZE_MASK 0x03 + #ifdef CONFIG_MMU -# define FLAGS_SHARED 0x01 +# define FLAGS_SHARED 0x10 #else /* * NOMMU does not have per process address space. Let the compiler optimize @@ -25,8 +33,58 @@ */ # define FLAGS_SHARED 0x00 #endif -#define FLAGS_CLOCKRT 0x02 -#define FLAGS_HAS_TIMEOUT 0x04 +#define FLAGS_CLOCKRT 0x20 +#define FLAGS_HAS_TIMEOUT 0x40 +#define FLAGS_NUMA 0x80 + +/* FUTEX_ to FLAGS_ */ +static inline unsigned int futex_to_flags(unsigned int op) +{ + unsigned int flags = FLAGS_SIZE_32; + + if (!(op & FUTEX_PRIVATE_FLAG)) + flags |= FLAGS_SHARED; + + if (op & FUTEX_CLOCK_REALTIME) + flags |= FLAGS_CLOCKRT; + + return flags; +} + +/* FUTEX2_ to FLAGS_ */ +static inline unsigned int futex2_to_flags(unsigned int flags2) +{ + unsigned int flags = flags2 & FUTEX2_64; + + if (!(flags2 & FUTEX2_PRIVATE)) + flags |= FLAGS_SHARED; + + if (flags2 & FUTEX2_NUMA) + flags |= FLAGS_NUMA; + + return flags; +} + +static inline bool futex_flags_valid(unsigned int flags) +{ + /* Only 64bit futexes for 64bit code */ + if (!IS_ENABLED(CONFIG_64BIT) || in_compat_syscall()) { + if ((flags & FLAGS_SIZE_MASK) == FLAGS_SIZE_64) + return false; + } + + /* Only 32bit futexes are implemented -- for now */ + if ((flags & FLAGS_SIZE_MASK) != FLAGS_SIZE_32) + return false; + + return true; +} + +static inline unsigned int futex_size(unsigned int flags) +{ + unsigned int size = flags & FLAGS_SIZE_MASK; + return 1 << size; /* {0,1,2,3} -> {1,2,4,8} */ +} #ifdef CONFIG_FAIL_FUTEX extern bool should_fail_futex(bool fshared); diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index cfc8001e88ea..36824b64219a 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -1,6 +1,5 @@ // SPDX-License-Identifier: GPL-2.0-or-later -#include #include #include @@ -85,15 +84,12 @@ SYSCALL_DEFINE3(get_robust_list, int, pid, long do_futex(u32 __user *uaddr, int op, u32 val, ktime_t *timeout, u32 __user *uaddr2, u32 val2, u32 val3) { + unsigned int flags = futex_to_flags(op); int cmd = op & FUTEX_CMD_MASK; - unsigned int flags = 0; - if (!(op & FUTEX_PRIVATE_FLAG)) - flags |= FLAGS_SHARED; - - if (op & FUTEX_CLOCK_REALTIME) { - flags |= FLAGS_CLOCKRT; - if (cmd != FUTEX_WAIT_BITSET && cmd != FUTEX_WAIT_REQUEUE_PI && + if (flags & FLAGS_CLOCKRT) { + if (cmd != FUTEX_WAIT_BITSET && + cmd != FUTEX_WAIT_REQUEUE_PI && cmd != FUTEX_LOCK_PI2) return -ENOSYS; } @@ -201,21 +197,19 @@ static int futex_parse_waitv(struct futex_vector *futexv, unsigned int i; for (i = 0; i < nr_futexes; i++) { + unsigned int flags; + if (copy_from_user(&aux, &uwaitv[i], sizeof(aux))) return -EFAULT; if ((aux.flags & ~FUTEX2_MASK) || aux.__reserved) return -EINVAL; - if (!IS_ENABLED(CONFIG_64BIT) || in_compat_syscall()) { - if ((aux.flags & FUTEX2_64) == FUTEX2_64) - return -EINVAL; - } - - if ((aux.flags & FUTEX2_64) != FUTEX2_32) + flags = futex2_to_flags(aux.flags); + if (!futex_flags_valid(flags)) return -EINVAL; - futexv[i].w.flags = aux.flags; + futexv[i].w.flags = flags; futexv[i].w.val = aux.val; futexv[i].w.uaddr = aux.uaddr; futexv[i].q = futex_q_init; diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index ba01b9408203..fa9757766103 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -419,11 +419,11 @@ static int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *wo */ retry: for (i = 0; i < count; i++) { - if ((vs[i].w.flags & FUTEX_PRIVATE_FLAG) && retry) + if (!(vs[i].w.flags & FLAGS_SHARED) && retry) continue; ret = get_futex_key(u64_to_user_ptr(vs[i].w.uaddr), - !(vs[i].w.flags & FUTEX_PRIVATE_FLAG), + vs[i].w.flags & FLAGS_SHARED, &vs[i].q.key, FUTEX_READ); if (unlikely(ret)) From patchwork Fri Jul 28 16:42:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127780 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp589088vqg; Fri, 28 Jul 2023 10:31:58 -0700 (PDT) X-Google-Smtp-Source: APBJJlEHDjcfLgh2labSQ5LM0K/lHZUgo8+14WzPYB8QNww/DCLDZi9xMpOg8xLu/hNtGsxNjDqN X-Received: by 2002:a05:6a20:3206:b0:13b:b111:6d13 with SMTP id hl6-20020a056a20320600b0013bb1116d13mr2090930pzc.5.1690565518098; Fri, 28 Jul 2023 10:31:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690565518; cv=none; d=google.com; s=arc-20160816; b=khfG+YSqIqkurd6TKDrliDXQeSiF5aGJTH4MHw0SlUjad70p8hcF6VyJmdIXHkR+7e WDRO0/5yG/YPkNxVfEKqvWt06l3bK9TMHAucOR6GwXR7GIc6JGu2Lfwas8fMxfgnEK67 G0IExthFlTzXdUErOJtD6cZd8IagsGx0T1MifjjA4h6AHesyOkcDd2xC4FePfcBJHILg QCWpKo7qYkapRQXUjFzZkQz7tec9a+aQ4rA5ne13aUh49Kfxfnk3UB4DAwF3npl0WEIo SvXKhi3WM/l0iSyDxMDaa13vDLg2wFjVBwv1+dYfw7pZuc77lvFL2xPE2IjVZhvXnCpm 532A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zKpdtSS0oh3kDYJZ6F3Ca88AdtI49+W/nTkRsVyM1Ag=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=ogAYcKEKpmoIuxTt9sQttlNKeDgEEr8QX6Mv3+jw2MSaMy4Gqcc83hoT8l0uzyTeFb SYVOiaxyTR7xqZFfywg0fsducZk/KbrtIsn8xqngyAzc01zf/WnwFbEX/wrhKUdy1QxK nz0iK3Smbvef34G6o15j4o17PibUiyJW4EsPKXObjfNj69dYSYrmuWQvohbm2GoY0YTq 7fyFsKOtLcCMDtaTZX4Vt8OZBzzcmsU1QF3v04QOsk7enfgokFwvjaFyhLh+W8WtvLDE ls8kDRxZqzfFIhHbijklkFxTqyrTGZA5rAmPTrTDlPLh2UgsXGZKIiTIUabLpzdUv354 ADcw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=zGoEYSJZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f7-20020a056a00238700b0066df86b9458si3507288pfc.223.2023.07.28.10.31.44; Fri, 28 Jul 2023 10:31:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=zGoEYSJZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233628AbjG1Qmx (ORCPT + 99 others); Fri, 28 Jul 2023 12:42:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232982AbjG1Qmr (ORCPT ); Fri, 28 Jul 2023 12:42:47 -0400 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EBFA173F for ; Fri, 28 Jul 2023 09:42:43 -0700 (PDT) Received: by mail-io1-xd2c.google.com with SMTP id ca18e2360f4ac-785ccd731a7so26646739f.0 for ; Fri, 28 Jul 2023 09:42:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562562; x=1691167362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zKpdtSS0oh3kDYJZ6F3Ca88AdtI49+W/nTkRsVyM1Ag=; b=zGoEYSJZYOD4AuwgEazidpIZwFo55y4OjaJiMaXAFFHq3iGLUh6WtqT6+T0597AiIH iM4ad5dZSzqSmj5Ycw0/cAhAzOR2HYX3ABHE9LA/9M133adNJ2o9B2FcpGA9BNg9Za/M 1zDTorroC8j+40qoe0G0UXaXZONGtOAONO7I49IT8gw37YYYSMmGUevAOzb/klC53eO2 3df5poDz8gYK/wFg/IhwQ9blj3CGXAwc4OVuUe5s5IM2Vs9Bl7OF3rIMw3no4MpuyNRq HFgJ0YFREspmzzaJx5p6rcaZZ/at3qcoDvFUFzlLXjw/acoOJLVNof2pkkIvV06PiuTr ukLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562562; x=1691167362; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zKpdtSS0oh3kDYJZ6F3Ca88AdtI49+W/nTkRsVyM1Ag=; b=Dy6yZ623n7Uggf/iqJYsUl6I8vO4PJb4ka18bfI0uiIoRIjIhOqHgUpLB4ftigN8DO xFoK6YTlCFbKKjzhMv+b7YdeC0Yo/r1cHWCxLJ2yUlTVDsq8vzla2VfqckKPFKbFS7V+ 6vYfwZWJOOQWHFh7bZleJgsZ7Z7YDA2Nui3IxF1bPfWicgu6fkw5ag94bIXmlUYWiN5U KNVVOT2GGB6yqVUthiPWlX4/Ts8+SiOROCssH/4xcwKup+aSWS1Cppoh1ozGKDQy35M9 5/29gO6N1jWveni4zeR1Kp2AFnT2wC4vUdqaptnhwHciyJSWiqXm56ywn1JyP3d54rsW QHHw== X-Gm-Message-State: ABy/qLYOA/4R8/VubruE/w6VWdnAp/SqyOLxs03NDXs0kXKe4W4GJFOY TC0dl95KeQHqLHmh+Ayy0X8hJA== X-Received: by 2002:a05:6602:3d5:b0:780:d65c:d78f with SMTP id g21-20020a05660203d500b00780d65cd78fmr98287iov.2.1690562562667; Fri, 28 Jul 2023 09:42:42 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:42 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 04/12] futex: Validate futex value against futex size Date: Fri, 28 Jul 2023 10:42:27 -0600 Message-Id: <20230728164235.1318118-5-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772686428900762534 X-GMAIL-MSGID: 1772686428900762534 From: Peter Zijlstra Ensure the futex value fits in the given futex size. Since this adds a constraint to an existing syscall, it might possibly change behaviour. Currently the value would be truncated to a u32 and any high bits would get silently lost. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 8 ++++++++ kernel/futex/syscalls.c | 3 +++ 2 files changed, 11 insertions(+) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index c0e04599904a..d0a43d751e30 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -86,6 +86,14 @@ static inline unsigned int futex_size(unsigned int flags) return 1 << size; /* {0,1,2,3} -> {1,2,4,8} */ } +static inline bool futex_validate_input(unsigned int flags, u64 val) +{ + int bits = 8 * futex_size(flags); + if (bits < 64 && (val >> bits)) + return false; + return true; +} + #ifdef CONFIG_FAIL_FUTEX extern bool should_fail_futex(bool fshared); #else diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index 36824b64219a..d2b2bcf2a665 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -209,6 +209,9 @@ static int futex_parse_waitv(struct futex_vector *futexv, if (!futex_flags_valid(flags)) return -EINVAL; + if (!futex_validate_input(flags, aux.val)) + return -EINVAL; + futexv[i].w.flags = flags; futexv[i].w.val = aux.val; futexv[i].w.uaddr = aux.uaddr; From patchwork Fri Jul 28 16:42:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127811 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp615308vqg; Fri, 28 Jul 2023 11:22:33 -0700 (PDT) X-Google-Smtp-Source: APBJJlFxM1jsNxu7NY1uirytWbqumgMUUwv3UnsXJM8p/HIUhbMpnpDc0tj2Ht4CmtBZOVPSvLoq X-Received: by 2002:a05:6512:280d:b0:4e0:a426:6ddc with SMTP id cf13-20020a056512280d00b004e0a4266ddcmr2875838lfb.0.1690568552869; Fri, 28 Jul 2023 11:22:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690568552; cv=none; d=google.com; s=arc-20160816; b=SuOdu+GcaeefxOoOtyk+e1zprj0kc/sYRWHiIclf8vFxrKhsxqFMJvQLnxXDIICyEI 6SnkO3fdlwA7jc9uPK8df38IuvSrrZcLZrpu70vQvuSzzwuG+5kwhJ5+1FXn9fuq6ub/ d6RfHoQqiUnLmaRPyyCyNqgZEB5qR5e2TeruiH1qz0IbWb7Tedm4Y08ZpLYar7pEgiZD bpRSnRBXCxNiSAnhhNTS5jzil3J/vQ6CNfFCovdAVVb8IUDHNuysx5RN/0JaJDIvDSgI INlgHq7ytisqVxbxE+d+L4de8INf5ZkoJ5f+lpeeThw3dtrxF+JAS1fWAAVwR5z4rXKY 3zXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YUJiOK1ym7lvCvvQLvWy2I//CZyKSdCVkDmBm3xzfUg=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=qa9nJ3qTVMPk+v7G1jIFFGxj14MoChCWX5plVy9cwyoFr0gQTFr1/Xgv9FHULtWbCi uMOjJZh1VUevSNjO6dndwHZariBYIDu/2Lft/s9aYzHTcZXlrKB8MgzNk7VdOVBaaITI LtDGGMVXM+GQN7cgF5aEvSOwVUCb5qg3GrUCBlmWDNFXam5e4Fv1uwf9okRt3Sr9rp1p +cubWgM3RIXfk2+vPuXEo9dyooh/lKgY3QG8m0bBfKyRAWPvVyhplR0IMi/SuT5V+iCb Ipg5xKfCb+JhTuGQEtCaPl2eFqCsEQErEG2tpFyZsN9u3UnYWB1UBYGVSwrUG/BsZSZX Xkmw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=2uarFxq6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d14-20020aa7d5ce000000b005189fadd2e6si2872331eds.413.2023.07.28.11.22.08; Fri, 28 Jul 2023 11:22:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=2uarFxq6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233953AbjG1Qng (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233429AbjG1Qmt (ORCPT ); Fri, 28 Jul 2023 12:42:49 -0400 Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10AA53C01 for ; Fri, 28 Jul 2023 09:42:44 -0700 (PDT) Received: by mail-io1-xd33.google.com with SMTP id ca18e2360f4ac-780c89d1998so33117839f.1 for ; Fri, 28 Jul 2023 09:42:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562563; x=1691167363; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YUJiOK1ym7lvCvvQLvWy2I//CZyKSdCVkDmBm3xzfUg=; b=2uarFxq6l4LF0iYUmrIR5R6ld0C1sNTQloeFP9NZniGqCpXDfNqN4kdzp/v9lF71rn Cc6pIudy/N5dwUpMAUkOrYWVqlv5x8QGrmAVBfLP4bCrWLEZVk2Ci8dtsp5QhN5QIX86 KvfKq8uqWqElwGg917IDsdSfuAOPtp6iab9ti6lD3ZYupxuX5lFMNcz7xrz0pH43pMLt 0aLjkSR1CxXmgDjVh95jyJJApBy5x4LCsyMIOZJr6VC9HFTSt+0s+fsD05lHJAPm/InS RDEN0jkwXwiMajW+0Z2mpJdFp7yFOrBrGq+R5UaUQAxORXizXlLHqnTGnis+lHa0keSb FD0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562563; x=1691167363; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YUJiOK1ym7lvCvvQLvWy2I//CZyKSdCVkDmBm3xzfUg=; b=ZiLCV8FFmSFxPNM0WtSxgV4/g8c51AOOPk3SXaIy+Ir1qSYq9HWWUU4HkwTl960Xs7 lK9EMXlJrHijMCEifaHtB5WLvYOFqJx7q6Xlg/Ieuy/ldCx1h3JWTorFDTc4ia8F6ewn Skg31IGxnyDYHLtRNJ9Sz4pvVZW24Kg0abQ11Ny6o1rWhbXBO9Mcvqb1uXcc4cNmVUiu Duv3H97rt5XsrehbMZ9Hutvro4+oFR9/xhGsl4aIzuoiy1DDRvdIA7An1OJXQoejbPf5 D1H0FQpN+UOF+zTjhByyZeEEwA7++FSb8sRyVcPmYebgml/ot9X74Z7ChpEtpFrMCumh BX9Q== X-Gm-Message-State: ABy/qLanQbajzeUCmsvCzk7th4rYXXaQzybUDXBktWE0w2Sfz8G52Xrb 2bAJehT5H6OJaaEFMExYcLzQ/A== X-Received: by 2002:a6b:c30f:0:b0:783:6e76:6bc7 with SMTP id t15-20020a6bc30f000000b007836e766bc7mr60982iof.2.1690562563452; Fri, 28 Jul 2023 09:42:43 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:43 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 05/12] futex: move FUTEX2_MASK to futex.h Date: Fri, 28 Jul 2023 10:42:28 -0600 Message-Id: <20230728164235.1318118-6-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772689610914364305 X-GMAIL-MSGID: 1772689610914364305 We need this for validating the futex2 flags outside of the normal futex syscalls. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 2 ++ kernel/futex/syscalls.c | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index d0a43d751e30..2f8deaabc9bc 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -51,6 +51,8 @@ static inline unsigned int futex_to_flags(unsigned int op) return flags; } +#define FUTEX2_MASK (FUTEX2_64 | FUTEX2_PRIVATE) + /* FUTEX2_ to FLAGS_ */ static inline unsigned int futex2_to_flags(unsigned int flags2) { diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index d2b2bcf2a665..221c49797de9 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -179,8 +179,6 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val, return do_futex(uaddr, op, val, tp, uaddr2, (unsigned long)utime, val3); } -#define FUTEX2_MASK (FUTEX2_64 | FUTEX2_PRIVATE) - /** * futex_parse_waitv - Parse a waitv array from userspace * @futexv: Kernel side list of waiters to be filled From patchwork Fri Jul 28 16:42:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127757 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp574192vqg; Fri, 28 Jul 2023 10:06:48 -0700 (PDT) X-Google-Smtp-Source: APBJJlHR8osZ2JwQ5OQ++648wacxtSjqiBabHImv3aSZBRCz5gJervCfytaufElcNI/pLucHgu8/ X-Received: by 2002:a17:903:25c5:b0:1bb:4861:d39e with SMTP id jc5-20020a17090325c500b001bb4861d39emr2240532plb.12.1690564008408; Fri, 28 Jul 2023 10:06:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690564008; cv=none; d=google.com; s=arc-20160816; b=ZW/c/fvVfZnW5QxKhMyQuTw3jyCUiDQ8Y3tY+RACAn2xFA1IH7la1q8euLMRv0hfl5 m1TGlAZsL/70GTbyNbfwYRkfR7ScaVNqVb1OdhCo4DDeRkFwy4+O+t/baDnWSAp/84eM hz/DLWheq+JaMf3Bz8nobX53TmzqmfJv92tCZjuCil4R6moJbJ6sp/8zfL0G4fCKoLNO xMlg+27tcBF3nS+C1bddqSdgdkjuA5TosFbsa6YyMYxsEk+rlqf2afWDTHRi9f2rX+Ei WRdB2tPn4qGnLRgEy5meYbwV8rReOplCYjoRwsipgMRqPft/xhlzKPxkvlKumU2y+YXH Ty7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1MuKSt6kQlqZtSb725uIc8Jh1HnDCZ6mt+Wvn40Ha5c=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=qzRoOlACDWEGHo288DJwQiTaLdOUQIcVTPujRkegzTr4k0O0iVK09F+DYk+BPCWCPe xPyrTe2o4K6Ka6Os3dFwW4RnJm1O9q11pkguL9p5u2ImmXRwgs5QN/HxZd4iTpKKZf3E a+dB6T0h/DKj3pq+qKjeR1n1rx+dqsQGWEsMFqfXpqVKX0lNOI1XWPWJxdSRkHGzNA5K qVDgaV2JkKXgjHjmd7pWYQHJ1GmQNly/cJphCf7dnCqfayr6W4v6WFhNmDvvNphCpnis 5l4Zp52XCYGyd3afWTSlUvB3h/FVQn17G7JFP0kioN/UjX49PLi94mguo3qbrlHnJGRC cyJw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=fAIW8K+Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n4-20020a637204000000b005641ddd0316si1231816pgc.6.2023.07.28.10.06.34; Fri, 28 Jul 2023 10:06:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=fAIW8K+Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234059AbjG1Qnj (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233361AbjG1Qmt (ORCPT ); Fri, 28 Jul 2023 12:42:49 -0400 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B29B3C3C for ; Fri, 28 Jul 2023 09:42:45 -0700 (PDT) Received: by mail-io1-xd2e.google.com with SMTP id ca18e2360f4ac-78706966220so22789339f.1 for ; Fri, 28 Jul 2023 09:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562564; x=1691167364; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1MuKSt6kQlqZtSb725uIc8Jh1HnDCZ6mt+Wvn40Ha5c=; b=fAIW8K+ZNnNqacGBmi1BV/Ct8dW/b2LtaoE7V4bEt8sG5on/ONzNQ59wRJeKSkFcpG 45NEPqZwbglWlHr0BH4sVi7EeuXg/sv8zLaDoB8d27rcLeJGyhFobtChC2gYCkz9Rsf2 lYozv4QRpmeWwmz2lifSNOOvgbTkIxIzRrITWMUv4/Fky9EFcb0suOEW4qDqaCiLg/Oz jNtTVSn7ol3l/MSwEhxvPltOTmBVok3UdLwSLKUV6H8Aynqvno484Rc2vJ5YyPWg+M/n A6YIQnFHhj+K4cNxAduvjC1BeDF+c5HiwCi4l4SWDWbRvEe32FHoqfGcnS4LKxK/Hyah L7Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562564; x=1691167364; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1MuKSt6kQlqZtSb725uIc8Jh1HnDCZ6mt+Wvn40Ha5c=; b=dcqXOX89XRck6OnzoBOKJGMZlKrZgA8XnLTmlLJxBuHel/JlJi/RTG2kd+HlZG57Jr KyxZ0VD1cO3ntCeZ7HgLvHOeYT9arnmz6NpSpa8qGGBy++6eOmNUHWncLB3QnoKP7ae4 UAU0vCJwKoKvcF9muwaGLnS84qrl2UMTO0VpSNTPepdmqZhci1N87fH6bccTL5dhBg0C 7skX/dHntjPS+KmBjc9seO8zl3auko+mm82PMf3GLYEOC6z174WzOVzzxsyYbWXOW/9q 2J6c897uG7/8hv5NG52wb6/UOgrqjiJ68/H3qge33jqgDI6G828sXFNeEGz8aahLoL9L 2q1Q== X-Gm-Message-State: ABy/qLbHkpK5XNAOsVzgm4HIB8ZwEw3+o3/PDSY3djSvFEBqd6oUrAOW HUd++ulxbxJi9tiUGdWUpIMJtQ== X-Received: by 2002:a05:6602:2b91:b0:77a:ee79:652 with SMTP id r17-20020a0566022b9100b0077aee790652mr119454iov.1.1690562564241; Fri, 28 Jul 2023 09:42:44 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:43 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 06/12] futex: factor out the futex wake handling Date: Fri, 28 Jul 2023 10:42:29 -0600 Message-Id: <20230728164235.1318118-7-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772684845210718053 X-GMAIL-MSGID: 1772684845210718053 In preparation for having another waker that isn't futex_wake_mark(), add a wake handler in futex_q. No extra data is associated with the handler outside of struct futex_q itself. futex_wake_mark() is defined as the standard wakeup helper, now set through futex_q_init like other defaults. Normal sync futex waiting relies on wake_q holding tasks that should be woken up. This is what futex_wake_mark() does, it'll unqueue the futex and add the associated task to the wake queue. For async usage of futex waiting, rather than having tasks sleeping on the futex, we'll need to deal with a futex wake differently. For the planned io_uring case, that means posting a completion event for the task in question. Having a definable wake handler can help support that use case. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 5 +++++ kernel/futex/requeue.c | 3 ++- kernel/futex/waitwake.c | 6 +++--- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 2f8deaabc9bc..bfc1e3c260b0 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -137,11 +137,15 @@ struct futex_pi_state { union futex_key key; } __randomize_layout; +struct futex_q; +typedef void (futex_wake_fn)(struct wake_q_head *wake_q, struct futex_q *q); + /** * struct futex_q - The hashed futex queue entry, one per waiting task * @list: priority-sorted list of tasks waiting on this futex * @task: the task waiting on the futex * @lock_ptr: the hash bucket lock + * @wake: the wake handler for this queue * @key: the key the futex is hashed on * @pi_state: optional priority inheritance state * @rt_waiter: rt_waiter storage for use with requeue_pi @@ -166,6 +170,7 @@ struct futex_q { struct task_struct *task; spinlock_t *lock_ptr; + futex_wake_fn *wake; union futex_key key; struct futex_pi_state *pi_state; struct rt_mutex_waiter *rt_waiter; diff --git a/kernel/futex/requeue.c b/kernel/futex/requeue.c index cba8b1a6a4cc..e892bc6c41d8 100644 --- a/kernel/futex/requeue.c +++ b/kernel/futex/requeue.c @@ -58,6 +58,7 @@ enum { const struct futex_q futex_q_init = { /* list gets initialized in futex_queue()*/ + .wake = futex_wake_mark, .key = FUTEX_KEY_INIT, .bitset = FUTEX_BITSET_MATCH_ANY, .requeue_state = ATOMIC_INIT(Q_REQUEUE_PI_NONE), @@ -591,7 +592,7 @@ int futex_requeue(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, /* Plain futexes just wake or requeue and are done */ if (!requeue_pi) { if (++task_count <= nr_wake) - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); else requeue_futex(this, hb1, hb2, &key2); continue; diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index fa9757766103..0272b8c3b132 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -174,7 +174,7 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset) if (!(this->bitset & bitset)) continue; - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); if (++ret >= nr_wake) break; } @@ -289,7 +289,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, ret = -EINVAL; goto out_unlock; } - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); if (++ret >= nr_wake) break; } @@ -303,7 +303,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2, ret = -EINVAL; goto out_unlock; } - futex_wake_mark(&wake_q, this); + this->wake(&wake_q, this); if (++op_ret >= nr_wake2) break; } From patchwork Fri Jul 28 16:42:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127760 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp575473vqg; Fri, 28 Jul 2023 10:08:33 -0700 (PDT) X-Google-Smtp-Source: APBJJlG5YqtxQ9WZ7ftP+ZKNvuUJQFVMcps5pv6lxUSzvYdNWw00Z90fgvqPOQ473t633Wj9HMTb X-Received: by 2002:a05:6a20:7da6:b0:138:c:ed04 with SMTP id v38-20020a056a207da600b00138000ced04mr2643661pzj.39.1690564113234; Fri, 28 Jul 2023 10:08:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690564113; cv=none; d=google.com; s=arc-20160816; b=eXca0OFP59y2T9r3I0D4Mj4SzQInoHqLTAVnTPGFXfCHb5na+YVVexzwLSchTH9WOQ QUu0qKt2wEAT0cRe7uRMRmE2NfKueFnRIYpLL3wBHE3PJOjoC285M1EMRmPYRGgry53I L4PYbtdQh/hOqtIpXEaSTc+afxa7jgPA1JXqB6D6aJRwHHFnzR9yuWKqXHlDkuqS2GJq 0u8evAZu8aFYwHBWTFsxCsVnINGC5WAOsGlewQyyjJtldtsRU1x9XGOGb53rVnggwE68 lfbhdb+q4/y8DfEvAFPEuXwjtAoyo7qTiA/44KcQ1Ob273b1g5QY+duEBjE4yGBXwOwx gYLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=jgmPORLNso3KhGjKAf8ri1EDar6yb/Ua+Lm2vp9NjUk=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=cRnknrY3kFZ3PtIy6+g2bD3/ZVM7G02VC5AvtbaDsL8hsd96lcegYOyQ9xw6l5lj/Y Un5UTAmffGFgp1AJrMv71PNefMs4a5VBl1BidEfdWq7yKmWIGCO+dcS4k+JjNdDlNwNE YdROMn8Ty9RPRMceyg474LLmmqWDcxvwcMBQrcx+TUdgGHPRwyagncpxc231+bvGXuRR 6toGocFduVEioYmE9kl28JGYJWbBjrHI/XnU/mdwJEdyKIePN9etYSM9/23p+qGIMjSC 7xtSiijoNnAu6/I6o+5i8WlBWDtVzS6FDhkxHiQYQca6DAdI7twIBaulgdzhUQD7r1fl Zmyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=Ur0yn05v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l62-20020a639141000000b005641e8b52basi1124359pge.302.2023.07.28.10.08.19; Fri, 28 Jul 2023 10:08:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=Ur0yn05v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234560AbjG1Qnn (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233680AbjG1Qmy (ORCPT ); Fri, 28 Jul 2023 12:42:54 -0400 Received: from mail-io1-xd31.google.com (mail-io1-xd31.google.com [IPv6:2607:f8b0:4864:20::d31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C781512C for ; Fri, 28 Jul 2023 09:42:45 -0700 (PDT) Received: by mail-io1-xd31.google.com with SMTP id ca18e2360f4ac-78706966220so22789939f.1 for ; Fri, 28 Jul 2023 09:42:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562565; x=1691167365; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jgmPORLNso3KhGjKAf8ri1EDar6yb/Ua+Lm2vp9NjUk=; b=Ur0yn05vA4g6T1oMAgp5S9eOXhQtpLAoIUiHwtUeZg7WO3bUolj2hC6IlLND4XUMQk 1f6OAMKnGzk4YG7fQyrPaVOicySAWV6S/Uppg1jacyTOFCo0Jg3Ci/zeD2fpcWM+Wacy fQXHgBS5CGBszY2iiZR+K4mhDOLA+caxiY8mRZ9YkvIEC+UdvmAejzvXVylbd5FW88sw YvtNlixuhvlm/8yVwYJfR6N9j4H0qPCMSeouDkwUC91RSldd+8w2qTixfwoIAyYhQ/OZ iHx0excWbAyl4KxIih6EFokslp1PTrvrkYZcLTIMotZrIGaDpyOyTFCWsp9BkQQHkb+u vkog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562565; x=1691167365; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jgmPORLNso3KhGjKAf8ri1EDar6yb/Ua+Lm2vp9NjUk=; b=AzMn590AAbIpzB6Pr8scNPilyXH95H2tscmrd+WmLF5NrMksK9UQOpxig9L6aCeE2X xzaydcvL43feWR8s0Xsqn/0jF2nsCTrh33qCTdTJknKZtLK9lAj80BE4N78hV9iSRW3a 69/ZNi/AUHOlf9u4b1qAoHzFXJtBYjRH61sCtsrD9VeyMThc/S0+hcmt/AGKCTCtdrL2 taCDZBPASm+Gc+KXYzEfQ+tYawwk8Z6v4xneOW2xqqaO4sXi8omecUZcDvG/XEj+zzNF eU2wQ+hZ7FafOv+gZ7C/07ii2yoD6UVSNxlTv8nlMme6pESPEKPermbYsrGzPJ7amq3m zIcQ== X-Gm-Message-State: ABy/qLaGShRfiTYdBmWVyLUn3FSFo4SHuXxKOX7CXXceICSF855dPZsT mlgzJlHLQ8fm40R0j4fJtzBIwg== X-Received: by 2002:a6b:c9d3:0:b0:788:2d78:813c with SMTP id z202-20020a6bc9d3000000b007882d78813cmr110344iof.0.1690562565051; Fri, 28 Jul 2023 09:42:45 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:44 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 07/12] futex: abstract out a __futex_wake_mark() helper Date: Fri, 28 Jul 2023 10:42:30 -0600 Message-Id: <20230728164235.1318118-8-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772684955443631488 X-GMAIL-MSGID: 1772684955443631488 Move the unqueue and lock_ptr clear into a helper that futex_wake_mark() calls. Add it to the public functions as well, in preparation for using it outside the core futex code. Suggested-by: Peter Zijlstra Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 1 + kernel/futex/waitwake.c | 33 ++++++++++++++++++++++----------- 2 files changed, 23 insertions(+), 11 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index bfc1e3c260b0..e04c74a34832 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -217,6 +217,7 @@ extern int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags, struct futex_q *q, struct futex_hash_bucket **hb); extern void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, struct hrtimer_sleeper *timeout); +extern bool __futex_wake_mark(struct futex_q *q); extern void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q); extern int fault_in_user_writeable(u32 __user *uaddr); diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 0272b8c3b132..86f67f652b95 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -106,20 +106,11 @@ * double_lock_hb() and double_unlock_hb(), respectively. */ -/* - * The hash bucket lock must be held when this is called. - * Afterwards, the futex_q must not be accessed. Callers - * must ensure to later call wake_up_q() for the actual - * wakeups to occur. - */ -void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) +bool __futex_wake_mark(struct futex_q *q) { - struct task_struct *p = q->task; - if (WARN(q->pi_state || q->rt_waiter, "refusing to wake PI futex\n")) - return; + return false; - get_task_struct(p); __futex_unqueue(q); /* * The waiting task can free the futex_q as soon as q->lock_ptr = NULL @@ -130,6 +121,26 @@ void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) */ smp_store_release(&q->lock_ptr, NULL); + return true; +} + +/* + * The hash bucket lock must be held when this is called. + * Afterwards, the futex_q must not be accessed. Callers + * must ensure to later call wake_up_q() for the actual + * wakeups to occur. + */ +void futex_wake_mark(struct wake_q_head *wake_q, struct futex_q *q) +{ + struct task_struct *p = q->task; + + get_task_struct(p); + + if (!__futex_wake_mark(q)) { + put_task_struct(p); + return; + } + /* * Queue the task for later wakeup for after we've released * the hb->lock. From patchwork Fri Jul 28 16:42:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127817 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp616238vqg; Fri, 28 Jul 2023 11:24:36 -0700 (PDT) X-Google-Smtp-Source: APBJJlEeBGxZPVe0gWuxj8R5mPpXw7ojrPZ7apgJ06rPpMqWXHmsCrA72sDWWJHnn9rmyb+rB9L3 X-Received: by 2002:aca:2414:0:b0:3a1:e222:97db with SMTP id n20-20020aca2414000000b003a1e22297dbmr3606862oic.30.1690568675869; Fri, 28 Jul 2023 11:24:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690568675; cv=none; d=google.com; s=arc-20160816; b=Bj1vMPLLZQqE0Vp7XJxIkkTqwlLfeSeyPLkcjzolG1TGbwChhECftgxU42ZjX94PG8 p2Wts+KH+XNECfRAhdIzKctyazzAYOT1DYnRfuczBXu4c5udyyJIn/IIY3xZ1pdzJIPY fj2p7D8kYECLOndbBYY+szM/nemxqn4Yq3g/tnx6pHghMSPrWARRuPR3C+0ZtFIA1/9u NyFoKFl1uG5zDi+gDzxZC3XcinEwv/yKzwKZwwpCKw/aUo6ZjcbscUv5DLU4FnkJv2kf 9hjxcz2wrlqBAjOk786rJRkSiuBqr/NDQ+smmDITG2ny++m2qpmYMPC+RXQ/rzgvZD1Q N3Zw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KxccK53kF0FsYbBmNLwXJRJproEpiVcxC/RyBoM24d8=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=k4RqHVWYY4SEJBRmqbHbE/DRXVrnth3q2uHVN9JvWJw0mTR+YxezYLdJihcy55BzjQ OmFPTHI8sQaSzrWmqK1kRdqypOhUyzvtAuMEy2w+5ippflaZowihwtIWvzoprbA05Uyv Moc7GZTsRT5DxajyBkksZGOCZ9zEDLmhTpkITc0iFoioBU+751RtbY7oNhctEu3kTlbL fZzMFKpoQH5xomZB13+lgb9CYe9rudX1qpGXauatnqGtZeWeLeqcNPbcSg+UJY3YlGvj 8FGK6wJhSIF66TfmIk05e1T+svuofa3INS/g/LOUNYSe8jH6yItzm+6J6RI5ni/l+hJh 0Cnw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=u0RbFeid; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f10-20020a63100a000000b00563de9d4ed5si3420256pgl.357.2023.07.28.11.24.22; Fri, 28 Jul 2023 11:24:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=u0RbFeid; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234675AbjG1Qnt (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233852AbjG1Qm6 (ORCPT ); Fri, 28 Jul 2023 12:42:58 -0400 Received: from mail-io1-xd35.google.com (mail-io1-xd35.google.com [IPv6:2607:f8b0:4864:20::d35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3EFC420F for ; Fri, 28 Jul 2023 09:42:46 -0700 (PDT) Received: by mail-io1-xd35.google.com with SMTP id ca18e2360f4ac-7835bbeb6a0so33278039f.0 for ; Fri, 28 Jul 2023 09:42:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562566; x=1691167366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KxccK53kF0FsYbBmNLwXJRJproEpiVcxC/RyBoM24d8=; b=u0RbFeidp9UhOJK1a3Hb1TuJ7h0f8PXpaMtwzvFaeZT9PIlq3REg/9saBdqEhrt4+A JW1hUymVr7UIr6oNywNAU2EuBgfAvpkHlWrCGHIbzGnYzd+vq4ZvAfAEcpmA2/vm1M1X 9HYJ2iUL6j0AtyhN+HXxvXQSGLjv/oDulfSIywaU7DODMoX9iWa75mjPRZSpDvwil0k4 ++GUwJQ6KIMV674BIuIq3srumNa03p8blvzaWeublUaNB26ke2s3L9Ywl/0J5ZuHIvlp xdy37+Td7k+MYtkayv07/uPMm2wLqXOjQOCuNQshSWoGJYhc6HBgW9MrbkcwxzuBKApt sGNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562566; x=1691167366; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KxccK53kF0FsYbBmNLwXJRJproEpiVcxC/RyBoM24d8=; b=O+1INYS612UVY06q6F9UUKHqJQH/fmIdGqe9tW5ARuEeq6G6gc2VlSTc1wPe8900YX 1KGnHA1Ia2nBjOAoGbTAg5o/KkgSD5IImC6tqSMK+XHTzxd1M41f+6c6aCrLn1yewG+G GGaEHe5Y03o6OgSpM/Gg0w/4Q34AM2z/yWSQsmIFnrj4JmEbDNrDzpOA3rNqGAysZ/lm zISKGCdOZcln/TTK8OMMTRQaXCVAOIJlbpvIr7rOpus8Hw3z2jC/iuSPrwFpovsZFwJ3 2qE29pfuZq9GOo2N5BNvgHegoVL3+YCWMtByevixGfOcGjNe3QpWwqRtmkVUi/JOOGyB 3gGg== X-Gm-Message-State: ABy/qLYNBwdvhgx2GTTL6TbzwgMbEb1RbBQh7dRfGKbSExsn1Yp4Bo5v Ayk0F6KNdbo+vYNPxvMh4bSdSw== X-Received: by 2002:a6b:b214:0:b0:780:d6ef:160 with SMTP id b20-20020a6bb214000000b00780d6ef0160mr90008iof.1.1690562565890; Fri, 28 Jul 2023 09:42:45 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:45 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 08/12] io_uring: add support for futex wake and wait Date: Fri, 28 Jul 2023 10:42:31 -0600 Message-Id: <20230728164235.1318118-9-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772689739421929564 X-GMAIL-MSGID: 1772689739421929564 Add support for FUTEX_WAKE/WAIT primitives. IORING_OP_FUTEX_WAKE is mix of FUTEX_WAKE and FUTEX_WAKE_BITSET, as it does support passing in a bitset. Similary, IORING_OP_FUTEX_WAIT is a mix of FUTEX_WAIT and FUTEX_WAIT_BITSET. FUTEX_WAKE is straight forward, as those can always be done directly from the io_uring submission without needing async handling. For FUTEX_WAIT, things are a bit more complicated. If the futex isn't ready, then we rely on a callback via futex_queue->wake() when someone wakes up the futex. From that calback, we queue up task_work with the original task, which will post a CQE and wake it, if necessary. Cancelations are supported, both from the application point-of-view, but also to be able to cancel pending waits if the ring exits before all events have occurred. This is just the barebones wait/wake support. PI or REQUEUE support is not added at this point, unclear if we might look into that later. Likewise, explicit timeouts are not supported either. It is expected that users that need timeouts would do so via the usual io_uring mechanism to do that using linked timeouts. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 + include/uapi/linux/io_uring.h | 3 + io_uring/Makefile | 4 +- io_uring/cancel.c | 5 + io_uring/cancel.h | 4 + io_uring/futex.c | 230 +++++++++++++++++++++++++++++++++ io_uring/futex.h | 34 +++++ io_uring/io_uring.c | 5 + io_uring/opdef.c | 24 +++- 9 files changed, 310 insertions(+), 2 deletions(-) create mode 100644 io_uring/futex.c create mode 100644 io_uring/futex.h diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f04ce513fadb..a7f03d8d879f 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -273,6 +273,9 @@ struct io_ring_ctx { struct io_wq_work_list locked_free_list; unsigned int locked_free_nr; + struct hlist_head futex_list; + struct io_alloc_cache futex_cache; + const struct cred *sq_creds; /* cred used for __io_sq_thread() */ struct io_sq_data *sq_data; /* if using sq thread polling */ diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 36f9c73082de..3bd2d765f593 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -65,6 +65,7 @@ struct io_uring_sqe { __u32 xattr_flags; __u32 msg_ring_flags; __u32 uring_cmd_flags; + __u32 futex_flags; }; __u64 user_data; /* data to be passed back at completion time */ /* pack this to avoid bogus arm OABI complaints */ @@ -235,6 +236,8 @@ enum io_uring_op { IORING_OP_URING_CMD, IORING_OP_SEND_ZC, IORING_OP_SENDMSG_ZC, + IORING_OP_FUTEX_WAIT, + IORING_OP_FUTEX_WAKE, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/Makefile b/io_uring/Makefile index 8cc8e5387a75..2e4779bc550c 100644 --- a/io_uring/Makefile +++ b/io_uring/Makefile @@ -7,5 +7,7 @@ obj-$(CONFIG_IO_URING) += io_uring.o xattr.o nop.o fs.o splice.o \ openclose.o uring_cmd.o epoll.o \ statx.o net.o msg_ring.o timeout.o \ sqpoll.o fdinfo.o tctx.o poll.o \ - cancel.o kbuf.o rsrc.o rw.o opdef.o notif.o + cancel.o kbuf.o rsrc.o rw.o opdef.o \ + notif.o obj-$(CONFIG_IO_WQ) += io-wq.o +obj-$(CONFIG_FUTEX) += futex.o diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 7b23607cf4af..3dba8ccb1cd8 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -15,6 +15,7 @@ #include "tctx.h" #include "poll.h" #include "timeout.h" +#include "futex.h" #include "cancel.h" struct io_cancel { @@ -119,6 +120,10 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd, if (ret != -ENOENT) return ret; + ret = io_futex_cancel(ctx, cd, issue_flags); + if (ret != -ENOENT) + return ret; + spin_lock(&ctx->completion_lock); if (!(cd->flags & IORING_ASYNC_CANCEL_FD)) ret = io_timeout_cancel(ctx, cd); diff --git a/io_uring/cancel.h b/io_uring/cancel.h index fc98622e6166..c0a8e7c520b6 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -1,4 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 +#ifndef IORING_CANCEL_H +#define IORING_CANCEL_H #include @@ -22,3 +24,5 @@ void init_hash_table(struct io_hash_table *table, unsigned size); int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg); bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd); + +#endif diff --git a/io_uring/futex.c b/io_uring/futex.c new file mode 100644 index 000000000000..f58ab33bfb32 --- /dev/null +++ b/io_uring/futex.c @@ -0,0 +1,230 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include + +#include + +#include "../kernel/futex/futex.h" +#include "io_uring.h" +#include "rsrc.h" +#include "futex.h" + +struct io_futex { + struct file *file; + u32 __user *uaddr; + unsigned long futex_val; + unsigned long futex_mask; + u32 futex_flags; +}; + +struct io_futex_data { + union { + struct futex_q q; + struct io_cache_entry cache; + }; + struct io_kiocb *req; +}; + +void io_futex_cache_init(struct io_ring_ctx *ctx) +{ + io_alloc_cache_init(&ctx->futex_cache, IO_NODE_ALLOC_CACHE_MAX, + sizeof(struct io_futex_data)); +} + +static void io_futex_cache_entry_free(struct io_cache_entry *entry) +{ + kfree(container_of(entry, struct io_futex_data, cache)); +} + +void io_futex_cache_free(struct io_ring_ctx *ctx) +{ + io_alloc_cache_free(&ctx->futex_cache, io_futex_cache_entry_free); +} + +static void io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) +{ + struct io_futex_data *ifd = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + + io_tw_lock(ctx, ts); + if (!io_alloc_cache_put(&ctx->futex_cache, &ifd->cache)) + kfree(ifd); + req->async_data = NULL; + hlist_del_init(&req->hash_node); + io_req_task_complete(req, ts); +} + +static bool __io_futex_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) +{ + struct io_futex_data *ifd = req->async_data; + + /* futex wake already done or in progress */ + if (!futex_unqueue(&ifd->q)) + return false; + + hlist_del_init(&req->hash_node); + io_req_set_res(req, -ECANCELED, 0); + req->io_task_work.func = io_futex_complete; + io_req_task_work_add(req); + return true; +} + +int io_futex_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned int issue_flags) +{ + struct hlist_node *tmp; + struct io_kiocb *req; + int nr = 0; + + if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_FD_FIXED)) + return -ENOENT; + + io_ring_submit_lock(ctx, issue_flags); + hlist_for_each_entry_safe(req, tmp, &ctx->futex_list, hash_node) { + if (req->cqe.user_data != cd->data && + !(cd->flags & IORING_ASYNC_CANCEL_ANY)) + continue; + if (__io_futex_cancel(ctx, req)) + nr++; + if (!(cd->flags & IORING_ASYNC_CANCEL_ALL)) + break; + } + io_ring_submit_unlock(ctx, issue_flags); + + if (nr) + return nr; + + return -ENOENT; +} + +bool io_futex_remove_all(struct io_ring_ctx *ctx, struct task_struct *task, + bool cancel_all) +{ + struct hlist_node *tmp; + struct io_kiocb *req; + bool found = false; + + lockdep_assert_held(&ctx->uring_lock); + + hlist_for_each_entry_safe(req, tmp, &ctx->futex_list, hash_node) { + if (!io_match_task_safe(req, task, cancel_all)) + continue; + __io_futex_cancel(ctx, req); + found = true; + } + + return found; +} + +int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + u32 flags; + + if (unlikely(sqe->fd || sqe->len || sqe->buf_index || sqe->file_index)) + return -EINVAL; + + iof->uaddr = u64_to_user_ptr(READ_ONCE(sqe->addr)); + iof->futex_val = READ_ONCE(sqe->addr2); + iof->futex_mask = READ_ONCE(sqe->addr3); + flags = READ_ONCE(sqe->futex_flags); + + if (flags & ~FUTEX2_MASK) + return -EINVAL; + + iof->futex_flags = futex2_to_flags(flags); + if (!futex_flags_valid(iof->futex_flags)) + return -EINVAL; + + if (!futex_validate_input(iof->futex_flags, iof->futex_val) || + !futex_validate_input(iof->futex_flags, iof->futex_mask)) + return -EINVAL; + + return 0; +} + +static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q) +{ + struct io_futex_data *ifd = container_of(q, struct io_futex_data, q); + struct io_kiocb *req = ifd->req; + + if (unlikely(!__futex_wake_mark(q))) + return; + + io_req_set_res(req, 0, 0); + req->io_task_work.func = io_futex_complete; + io_req_task_work_add(req); +} + +static struct io_futex_data *io_alloc_ifd(struct io_ring_ctx *ctx) +{ + struct io_cache_entry *entry; + + entry = io_alloc_cache_get(&ctx->futex_cache); + if (entry) + return container_of(entry, struct io_futex_data, cache); + + return kmalloc(sizeof(struct io_futex_data), GFP_NOWAIT); +} + +int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct io_ring_ctx *ctx = req->ctx; + struct io_futex_data *ifd = NULL; + struct futex_hash_bucket *hb; + int ret; + + if (!iof->futex_mask) { + ret = -EINVAL; + goto done; + } + + io_ring_submit_lock(ctx, issue_flags); + ifd = io_alloc_ifd(ctx); + if (!ifd) { + ret = -ENOMEM; + goto done_unlock; + } + + req->async_data = ifd; + ifd->q = futex_q_init; + ifd->q.bitset = iof->futex_mask; + ifd->q.wake = io_futex_wake_fn; + ifd->req = req; + + ret = futex_wait_setup(iof->uaddr, iof->futex_val, + futex2_to_flags(iof->futex_flags), &ifd->q, &hb); + if (!ret) { + hlist_add_head(&req->hash_node, &ctx->futex_list); + io_ring_submit_unlock(ctx, issue_flags); + + futex_queue(&ifd->q, hb); + return IOU_ISSUE_SKIP_COMPLETE; + } + +done_unlock: + io_ring_submit_unlock(ctx, issue_flags); +done: + if (ret < 0) + req_set_fail(req); + io_req_set_res(req, ret, 0); + kfree(ifd); + return IOU_OK; +} + +int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + int ret; + + ret = futex_wake(iof->uaddr, futex2_to_flags(iof->futex_flags), + iof->futex_val, iof->futex_mask); + if (ret < 0) + req_set_fail(req); + io_req_set_res(req, ret, 0); + return IOU_OK; +} diff --git a/io_uring/futex.h b/io_uring/futex.h new file mode 100644 index 000000000000..ddc9e0d73c52 --- /dev/null +++ b/io_uring/futex.h @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "cancel.h" + +int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags); +int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags); + +#if defined(CONFIG_FUTEX) +int io_futex_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned int issue_flags); +bool io_futex_remove_all(struct io_ring_ctx *ctx, struct task_struct *task, + bool cancel_all); +void io_futex_cache_init(struct io_ring_ctx *ctx); +void io_futex_cache_free(struct io_ring_ctx *ctx); +#else +static inline int io_futex_cancel(struct io_ring_ctx *ctx, + struct io_cancel_data *cd, + unsigned int issue_flags) +{ + return 0; +} +static inline bool io_futex_remove_all(struct io_ring_ctx *ctx, + struct task_struct *task, bool cancel_all) +{ + return false; +} +static inline void io_futex_cache_init(struct io_ring_ctx *ctx) +{ +} +static inline void io_futex_cache_free(struct io_ring_ctx *ctx) +{ +} +#endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 135da2fd0eda..e52cbdcb29b8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -92,6 +92,7 @@ #include "cancel.h" #include "net.h" #include "notif.h" +#include "futex.h" #include "timeout.h" #include "poll.h" @@ -330,6 +331,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) sizeof(struct async_poll)); io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, sizeof(struct io_async_msghdr)); + io_futex_cache_init(ctx); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); @@ -349,6 +351,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->tctx_list); ctx->submit_state.free_list.next = NULL; INIT_WQ_LIST(&ctx->locked_free_list); + INIT_HLIST_HEAD(&ctx->futex_list); INIT_DELAYED_WORK(&ctx->fallback_work, io_fallback_req_func); INIT_WQ_LIST(&ctx->submit_state.compl_reqs); return ctx; @@ -2869,6 +2872,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_eventfd_unregister(ctx); io_alloc_cache_free(&ctx->apoll_cache, io_apoll_cache_free); io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); + io_futex_cache_free(ctx); io_destroy_buffers(ctx); mutex_unlock(&ctx->uring_lock); if (ctx->sq_creds) @@ -3281,6 +3285,7 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, ret |= io_cancel_defer_files(ctx, task, cancel_all); mutex_lock(&ctx->uring_lock); ret |= io_poll_remove_all(ctx, task, cancel_all); + ret |= io_futex_remove_all(ctx, task, cancel_all); mutex_unlock(&ctx->uring_lock); ret |= io_kill_timeouts(ctx, task, cancel_all); if (task) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 3b9c6489b8b6..c9f23c21a031 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -33,6 +33,7 @@ #include "poll.h" #include "cancel.h" #include "rw.h" +#include "futex.h" static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags) { @@ -426,11 +427,26 @@ const struct io_issue_def io_issue_defs[] = { .issue = io_sendmsg_zc, #else .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_FUTEX_WAIT] = { +#if defined(CONFIG_FUTEX) + .prep = io_futex_prep, + .issue = io_futex_wait, +#else + .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_FUTEX_WAKE] = { +#if defined(CONFIG_FUTEX) + .prep = io_futex_prep, + .issue = io_futex_wake, +#else + .prep = io_eopnotsupp_prep, #endif }, }; - const struct io_cold_def io_cold_defs[] = { [IORING_OP_NOP] = { .name = "NOP", @@ -648,6 +664,12 @@ const struct io_cold_def io_cold_defs[] = { .fail = io_sendrecv_fail, #endif }, + [IORING_OP_FUTEX_WAIT] = { + .name = "FUTEX_WAIT", + }, + [IORING_OP_FUTEX_WAKE] = { + .name = "FUTEX_WAKE", + }, }; const char *io_uring_get_opcode(u8 opcode) From patchwork Fri Jul 28 16:42:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127763 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp575892vqg; Fri, 28 Jul 2023 10:09:11 -0700 (PDT) X-Google-Smtp-Source: APBJJlFmzxu2TtuI2njpz5NaTZRKSr57yee10q3+iakADS2QxDdF+qX4aZ2o+i4W7mLKcRPzQLCb X-Received: by 2002:a05:6870:c21f:b0:1bb:d808:da72 with SMTP id z31-20020a056870c21f00b001bbd808da72mr4165268oae.57.1690564151308; Fri, 28 Jul 2023 10:09:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690564151; cv=none; d=google.com; s=arc-20160816; b=tE4nSJZFGOBVg6e0ZkP/JrHb+h6Zqh2zcbdIoOSlsGrtXFjSpjCwTfRFcGnboWng8U 1kmPUCID4hHZq2+/hU/38CzZzoXLePuvhXf70Bo7RvFKBLwSNHnvLSFTMwLEOuRNbUDK bczAuQZkyz1lJ1yBYWV9WgSa1ZsGU9S5KiiWNfu6ayLeU2eD9STCHkHUaDKPwxT5TJe7 6gFH+EA+5uQZNmb8uJ7E3CJFcDqdj+oO8zYTuzohw1GMfidcs+im20DKv85c3tui+my7 ZDUa5GRwl1gCAE0BI+wKBFSDibdtaKAoR/WWCTkol4ddtfoh+n1dboZTL39Woq/sL/Fi 4kcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=OXSlh5iMF2cncsBboldJj3SWTlVFipNC9kJryix07S8=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=lWujfUA+6OM+570yGxZnb0oYdS1s20Ty62HZat21GPmQrfq1vv+Fwyyg6NmncBNncM Pa6KfycrKo2Z4hbf4EJFZACUubsK+9CJS+479lRQo6o1Ids3RS8eBrZp5oX6cOhrUdE+ 3kvdMCLDBFTRAKqZC6xgf8V9iL8DKDLXGt3OmgdFwwFna1ozA/tdw3OTBzYxIiOA9RET H1Emipo4wpqKOrFT0PWJP327zEaN68IT1M7pC0TVyVHDfNc5SSvYowSlmlJOQVZ8+Hld UTeDD+XWmFMhjK9s/iT45yoAana0l10NZS6ZaLT6+z6oz62H+zYMCtB94ad1Mt6/tE23 tQgg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=uR1ssBnj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w22-20020a63c116000000b0055bdcb7dd5csi3295485pgf.701.2023.07.28.10.08.57; Fri, 28 Jul 2023 10:09:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=uR1ssBnj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234631AbjG1Qnp (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233883AbjG1Qm6 (ORCPT ); Fri, 28 Jul 2023 12:42:58 -0400 Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73565421F for ; Fri, 28 Jul 2023 09:42:47 -0700 (PDT) Received: by mail-io1-xd2b.google.com with SMTP id ca18e2360f4ac-785d3a53ed6so26249139f.1 for ; Fri, 28 Jul 2023 09:42:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562567; x=1691167367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OXSlh5iMF2cncsBboldJj3SWTlVFipNC9kJryix07S8=; b=uR1ssBnjeuiTE5m+B1bv8/fVJi5l51pnVmG4/EL2HyEO/5GT3XvxP+aKESMKhKlEgX jZie60GP1Zs56RCJUEG7Fl3y6elluyJ1bBWTmu8gm4wxHC3Ht0X2XW8Iq0nSAqg4ZMDx L3jL+mrawF4MCUuHLxar6LxnHgBrJI4J3KXzU1IuiXv0ivgJuxsdxGJHyW/B3uwqOyiz vuWprkljIdUGjE/PFrQjkroizLVTWfApqq5YggJJCfCgmVzjxTs4LtwPy25jqwMknP7a G+1N8neLLNkHLV0jnpbkM3i88h4gcDlyQ8S2i/5tRA2Tk8Z7+F8S56ihdnb5n9oWTYAo 7rYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562567; x=1691167367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OXSlh5iMF2cncsBboldJj3SWTlVFipNC9kJryix07S8=; b=OrNeVbgmUKHOtR06ic0bRVPcFdNZMPdzQ3h4DMXhGTke2gCaFHbuyMmCktWUMjVGuZ +YPEaZ3eo0VfVTpN96Douaqw+ZqiQ1tygw0Dsz1r6JWbm+gVOVb954a6I7Y8v1AVIb4R 4uXD9FOGFSdKap+5aZ5Csc/TIIvHE0Ri3NcEFsxmKzJ+aMWrHa8SGxukeQR5ALTzOjTB AyDYf4pgB8kalZNNQ1pY/4GQVaZ80tKQbVIlUQ8Lw4/Az8+faBe98rLOPiilixoJg6IV 3O5r0spLnKjt4voAe9WPOemy0b9U0ASXG6LzAlk6dmHK9bhWvRxKvFJA9CqZtvQbjUS8 jLgg== X-Gm-Message-State: ABy/qLb8a1Wf7QJy8fwi0HdeecmzIT+6PfjOhr7Q2y84cm22RdmKu08B +8WYvrGkeHhbelMT3B+xXsbEMMuB5lizhYA0MJQ= X-Received: by 2002:a05:6602:3d5:b0:780:d65c:d78f with SMTP id g21-20020a05660203d500b00780d65cd78fmr98470iov.2.1690562566663; Fri, 28 Jul 2023 09:42:46 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:46 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 09/12] futex: add wake_data to struct futex_q Date: Fri, 28 Jul 2023 10:42:32 -0600 Message-Id: <20230728164235.1318118-10-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772684995176974817 X-GMAIL-MSGID: 1772684995176974817 With handling multiple futex_q for waitv, we cannot easily go from the futex_q to data related to that request or queue. Add a wake_data argument that belongs to the wake handler assigned. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index e04c74a34832..4633c99ea4b6 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -146,6 +146,7 @@ typedef void (futex_wake_fn)(struct wake_q_head *wake_q, struct futex_q *q); * @task: the task waiting on the futex * @lock_ptr: the hash bucket lock * @wake: the wake handler for this queue + * @wake_data: data associated with the wake handler * @key: the key the futex is hashed on * @pi_state: optional priority inheritance state * @rt_waiter: rt_waiter storage for use with requeue_pi @@ -171,6 +172,7 @@ struct futex_q { struct task_struct *task; spinlock_t *lock_ptr; futex_wake_fn *wake; + void *wake_data; union futex_key key; struct futex_pi_state *pi_state; struct rt_mutex_waiter *rt_waiter; From patchwork Fri Jul 28 16:42:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127820 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp616416vqg; Fri, 28 Jul 2023 11:24:54 -0700 (PDT) X-Google-Smtp-Source: APBJJlFB5zfpblMeAYrCHgsaPzhFmIXH2P1e3i6YJB8oC1abrM6tHnoaT3APkIeXFibLNXz1BI9O X-Received: by 2002:a05:6a00:218d:b0:63d:3339:e967 with SMTP id h13-20020a056a00218d00b0063d3339e967mr2354229pfi.19.1690568694587; Fri, 28 Jul 2023 11:24:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690568694; cv=none; d=google.com; s=arc-20160816; b=W9bpbsidw4FfYklaFxxAlUDMy4vbH6rgFItapqiO/dgGxzgqcTdx8Uha+OxebUiij2 nOCRwl2RWTC05vyFOy+VOxgdfLKYXbZCXMPA4A1P8HGPY1UdCA8gv4MpfDYPrxmBjxR1 mxZKUPTB2kjElxxPCVwDHQi55eMRiXhOqKCT5McoZpii3Rzhe71NAqtIcFG6zZtMN9S7 FPKsmZDNZfEKtCkap0zme7YMSHPpwwPx7qG6hQTbtBrMqE0o78xmjjdmTFNfOnvvKAaQ NuVgCALYOec+0MJCzBlsM1BciZvwl93o0gvKJGdVS1o9AIV3CcE4iCMTs+aii9AtT73J 9sWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=O15ZlJlMRRtfXuiAf9H5Y3NU4zBlFmFQB8XZUfTXeI4=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=wSoQOJpCciqrWbMyWTipDkKMuta0pT6uLUMxy0BxS1Eg8AhgvtjmC8kqDECBenjKy5 G9YQ756hQongoSo3/SFu/mFx2RdJDoqkmk6Xz03dGVpJnC8/A/duzduBkm3E3kBRm6gz MGWb2ZpUcpKHaC0HMyaN7cQb/CX7YkD4C0RrK54rT8oJYn4m8zFdL5GsjxyHd2gfqn7Z YPH2rTyPtmr5wMeOWaCDlJzvhCetDrMVG9dq337a7JhEU8WLEh8yVLdfVbeAbbwk5h7B HwuVyg6n/uEkstHZJRyBBVPXyofhlqAXL80BSqopuTRsROnuRBNujJaDM8NmSGINvxRr aDlA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=jVw7OGLE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u11-20020a6540cb000000b0056424625528si640010pgp.656.2023.07.28.11.24.41; Fri, 28 Jul 2023 11:24:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=jVw7OGLE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229743AbjG1Qnv (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233923AbjG1Qm7 (ORCPT ); Fri, 28 Jul 2023 12:42:59 -0400 Received: from mail-io1-xd36.google.com (mail-io1-xd36.google.com [IPv6:2607:f8b0:4864:20::d36]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13DD14223 for ; Fri, 28 Jul 2023 09:42:49 -0700 (PDT) Received: by mail-io1-xd36.google.com with SMTP id ca18e2360f4ac-77dcff76e35so28574539f.1 for ; Fri, 28 Jul 2023 09:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562568; x=1691167368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O15ZlJlMRRtfXuiAf9H5Y3NU4zBlFmFQB8XZUfTXeI4=; b=jVw7OGLEOwjjGVchaN6wX+a8JIucwVplUbPRiAybz25B+NvXLDyU7MPnQhfWGs38Or tly/xPZ68rh/HXSyZLyh3dB+QhVmCvxvAMdiAkX5xtV2698P8H6YoT3P6K7zlCkWg5Pe KSMOqbCNSDEgaXVZNh17+j4Kh8wTkzvWbeWeSYy0FtXgwM4clwYUefYE3YOZAuyyOl/s 4bjBIQnHWNdnJV45lXmC8uf5CAEFRkgEg2v6HUd/Pj5BwKCAlOJ3RC3gz879bN+YOl70 TFXw6nnIQOKHcQKkQX1en86DI89y3zVQKOzMra3/Na90JNMsKytK/54b8vOihTV9bhiE mI0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562568; x=1691167368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O15ZlJlMRRtfXuiAf9H5Y3NU4zBlFmFQB8XZUfTXeI4=; b=TXhOsfiJTu//JR9u7gODF+ZQm3FTzg9iZQxH/2jO2/jYTFA6tLm4sFhZMDjLOhfm+b jQP5ctbsuWtwi17mxQzJ/ytoDGbfZM1vgyYHLDrKrUDq+M69RRBEYfOMMgxJwXztBbY1 ++HergkKwM6Df0szDbUU+nPH1qLxOBcgN7nVFfOyWXmq8RScuRUI1rmMGv43s5JPUujE 01d9qQE3rM3pTpGpdiSgo++Vw6waQvhgNLWi7q8cdE5h0Yp0YDshQlysZqtjw+fCU0I8 YDPnKytfecpNGF5PiRg//Ke9clbJ9WZo+sWV/ycSQwzFJHZf6LFvVeAR6rZVLwcD34aR YzwA== X-Gm-Message-State: ABy/qLbS+rtXAk/oVe/VgL138++/CiKIDMACVT7xYhS5REEVxGnFQOrb m6SSX8GGvpV+5T77RMV88Smcgw== X-Received: by 2002:a6b:b797:0:b0:783:63e8:3bfc with SMTP id h145-20020a6bb797000000b0078363e83bfcmr116801iof.0.1690562567714; Fri, 28 Jul 2023 09:42:47 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:47 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 10/12] futex: make futex_parse_waitv() available as a helper Date: Fri, 28 Jul 2023 10:42:33 -0600 Message-Id: <20230728164235.1318118-11-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772689759279309369 X-GMAIL-MSGID: 1772689759279309369 To make it more generically useful, augment it with allowing the caller to pass in the wake handler and wake data. Convert the futex_waitv() syscall, passing in the default handlers. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 5 +++++ kernel/futex/syscalls.c | 14 ++++++++++---- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 4633c99ea4b6..6a13275ca231 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -355,6 +355,11 @@ struct futex_vector { struct futex_q q; }; +extern int futex_parse_waitv(struct futex_vector *futexv, + struct futex_waitv __user *uwaitv, + unsigned int nr_futexes, futex_wake_fn *wake, + void *wake_data); + extern int futex_wait_multiple(struct futex_vector *vs, unsigned int count, struct hrtimer_sleeper *to); diff --git a/kernel/futex/syscalls.c b/kernel/futex/syscalls.c index 221c49797de9..bbb3b3ceef51 100644 --- a/kernel/futex/syscalls.c +++ b/kernel/futex/syscalls.c @@ -184,12 +184,15 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val, * @futexv: Kernel side list of waiters to be filled * @uwaitv: Userspace list to be parsed * @nr_futexes: Length of futexv + * @wake: Wake to call when futex is woken + * @wake_data: Data for the wake handler * * Return: Error code on failure, 0 on success */ -static int futex_parse_waitv(struct futex_vector *futexv, - struct futex_waitv __user *uwaitv, - unsigned int nr_futexes) +int futex_parse_waitv(struct futex_vector *futexv, + struct futex_waitv __user *uwaitv, + unsigned int nr_futexes, futex_wake_fn *wake, + void *wake_data) { struct futex_waitv aux; unsigned int i; @@ -214,6 +217,8 @@ static int futex_parse_waitv(struct futex_vector *futexv, futexv[i].w.val = aux.val; futexv[i].w.uaddr = aux.uaddr; futexv[i].q = futex_q_init; + futexv[i].q.wake = wake; + futexv[i].q.wake_data = wake_data; } return 0; @@ -290,7 +295,8 @@ SYSCALL_DEFINE5(futex_waitv, struct futex_waitv __user *, waiters, goto destroy_timer; } - ret = futex_parse_waitv(futexv, waiters, nr_futexes); + ret = futex_parse_waitv(futexv, waiters, nr_futexes, futex_wake_mark, + NULL); if (!ret) ret = futex_wait_multiple(futexv, nr_futexes, timeout ? &to : NULL); From patchwork Fri Jul 28 16:42:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127762 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp575774vqg; Fri, 28 Jul 2023 10:09:00 -0700 (PDT) X-Google-Smtp-Source: APBJJlFcFpDCId8216mPdqdyP75ynCNAIs3DELOeznZzyzju3DNucMd1PpZmVjiiP0u4xrUNjRGL X-Received: by 2002:a17:90b:3b8d:b0:268:e3d:1251 with SMTP id pc13-20020a17090b3b8d00b002680e3d1251mr3155324pjb.20.1690564140648; Fri, 28 Jul 2023 10:09:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690564140; cv=none; d=google.com; s=arc-20160816; b=T+AS/oFOLKiScjK8/7sdIg2jg+WYyUl3s0lKQRUP7h9Kpjp+wP1K75u680LeRST9M3 62G9UxZA/4QQhkhci/etMFaAVHx0OpHaxS1EOLOUs/eYqPcLbO6zIeHH0lZJ82OiDfcs 5T3JBeh5KbkC25LvBwcdsfVbFEMQXwcUfQF/kRt7J3P/244fpkYW/ZlQDT1ylTAKUG2H agMvEoYv5Gl0ZLXgHQzvxyVMVk3ywh7BVdsJ4ci9buGCIOpffrKiLvYR7t0BhXclC/Bt SI0ktI6l0GDvx54YBe0XyQnnfSOtrMEjRDOJH+Ww3QSccCN0BrS5akMd8mMPUEKwn65U 43eQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+IWyuICOqdD99scZWz5YBmXli6kjkGIoPuZY8ce+o9s=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=sJSo5MXXQxNp5uJwaAE7cPNc5ZzcRJ98e1nEFgiLwqqVkCVspqQdUAUlzMYq7GywKo Tm10oyFJp5WGAMDvTcA1AatcxBbl7NSo4ixJk06+wloJ/b6wXlK3EQFAAip7LvlbR14h Ro4Vd1MidMbsPJjwPBZlOCQqHbX4qNCdxEMCueyH8cuveCRRhKOKNzCj85pQEaFtR3+M Cbci6fcJUuhjSAoojeEMxJ82jJbutJ8rYUXSCESuzx7/67k+mc3/uSpP/39rvA5PgGL0 jLKAsSzHV4ozWvpYdcZzkJG96VfknLhiuVSJEZt5LcwTIflkMt11Aee5coHz7IoJoBj3 xkmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=IhjMmRjW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n2-20020a17090ade8200b0025bec4468c2si3355586pjv.167.2023.07.28.10.08.46; Fri, 28 Jul 2023 10:09:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=IhjMmRjW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234771AbjG1Qn4 (ORCPT + 99 others); Fri, 28 Jul 2023 12:43:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233975AbjG1Qm7 (ORCPT ); Fri, 28 Jul 2023 12:42:59 -0400 Received: from mail-io1-xd2f.google.com (mail-io1-xd2f.google.com [IPv6:2607:f8b0:4864:20::d2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5EF910F9 for ; Fri, 28 Jul 2023 09:42:49 -0700 (PDT) Received: by mail-io1-xd2f.google.com with SMTP id ca18e2360f4ac-7748ca56133so22690939f.0 for ; Fri, 28 Jul 2023 09:42:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562569; x=1691167369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+IWyuICOqdD99scZWz5YBmXli6kjkGIoPuZY8ce+o9s=; b=IhjMmRjWzjaP+W+W8beG37/ey8xKQzl2UDPk8dq4Di6aPaecAKUbMF6tPz4poXAP/A dBalPK12CUy/nTOhDvhkq44mLP/imORnLSDPvct+yQTmL/3w8+/gCss5zvkrSsDkimkN G0Isr0ckjR5bGBZ15KgvigMk5TXxkb9luuF0sDPiTOmXLJ9AB3kYMCnZthEoMT7UqMvJ 5Mvqby+93gDIRseQdMraLEI+PQoi3R0znkV0qUEjT75Cwl6C8BJJJZ+DLYM3B/s19LPK WA1ok7IfO40y81DuOpIUTtrKmg2NI5U0aLBomyc2oHKUGxo02EvHT/asyNMG7jVIO4VY 9Ubg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562569; x=1691167369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+IWyuICOqdD99scZWz5YBmXli6kjkGIoPuZY8ce+o9s=; b=ENUy0Qa2MI8pjHMlRePvTx4vIm4qUjo8Kfs683Ack1WEysD/N56eJoszSstmipiNDp tPgJN2KY52c4wYC2r0mwckRT7mZ7TsI9XtJ9fgzGMorDEO3foA6MRNkThSvADYAb/MI9 Lq6S8uUPfGBZRSoFINeNkaazTV42ZW8zJMqf03dXcHArcXHMIu+c6e3HMoEztvLlpcW5 WYdK86GHKJSnPzXRWt7Zd1SfYWirlnEf8359JHPXcoc1UBzGZTiKo/r9Q3S9Xdy3cVe3 UrSDYLYv2VOEm6l2g84Uciv5ltVSaAlGTLcCP1XiLTonrN/Tie+Jpmi43QHBGulULSyT aYWQ== X-Gm-Message-State: ABy/qLaC2YZ3c3Ny1wIUjEgJnmPShybVcYfb0Nodvc6vUZ+sRbhiPC2c sfydgFsHm60eswurraRSI2EFkj3IPXGd8g5RcWk= X-Received: by 2002:a05:6602:2b91:b0:77a:ee79:652 with SMTP id r17-20020a0566022b9100b0077aee790652mr119853iov.1.1690562569351; Fri, 28 Jul 2023 09:42:49 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:48 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 11/12] futex: make the vectored futex operations available Date: Fri, 28 Jul 2023 10:42:34 -0600 Message-Id: <20230728164235.1318118-12-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772684984641259241 X-GMAIL-MSGID: 1772684984641259241 Rename unqueue_multiple() as futex_unqueue_multiple(), and make both that and futex_wait_multiple_setup() available for external users. This is in preparation for wiring up vectored waits in io_uring. Signed-off-by: Jens Axboe --- kernel/futex/futex.h | 5 +++++ kernel/futex/waitwake.c | 10 +++++----- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h index 6a13275ca231..b099b849aecb 100644 --- a/kernel/futex/futex.h +++ b/kernel/futex/futex.h @@ -360,6 +360,11 @@ extern int futex_parse_waitv(struct futex_vector *futexv, unsigned int nr_futexes, futex_wake_fn *wake, void *wake_data); +extern int futex_wait_multiple_setup(struct futex_vector *vs, int count, + int *woken); + +extern int futex_unqueue_multiple(struct futex_vector *v, int count); + extern int futex_wait_multiple(struct futex_vector *vs, unsigned int count, struct hrtimer_sleeper *to); diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index 86f67f652b95..6c8fb7300558 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -369,7 +369,7 @@ void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, } /** - * unqueue_multiple - Remove various futexes from their hash bucket + * futex_unqueue_multiple - Remove various futexes from their hash bucket * @v: The list of futexes to unqueue * @count: Number of futexes in the list * @@ -379,7 +379,7 @@ void futex_wait_queue(struct futex_hash_bucket *hb, struct futex_q *q, * - >=0 - Index of the last futex that was awoken; * - -1 - No futex was awoken */ -static int unqueue_multiple(struct futex_vector *v, int count) +int futex_unqueue_multiple(struct futex_vector *v, int count) { int ret = -1, i; @@ -407,7 +407,7 @@ static int unqueue_multiple(struct futex_vector *v, int count) * - 0 - Success * - <0 - -EFAULT, -EWOULDBLOCK or -EINVAL */ -static int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) +int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *woken) { struct futex_hash_bucket *hb; bool retry = false; @@ -469,7 +469,7 @@ static int futex_wait_multiple_setup(struct futex_vector *vs, int count, int *wo * was woken, we don't return error and return this index to * userspace */ - *woken = unqueue_multiple(vs, i); + *woken = futex_unqueue_multiple(vs, i); if (*woken >= 0) return 1; @@ -554,7 +554,7 @@ int futex_wait_multiple(struct futex_vector *vs, unsigned int count, __set_current_state(TASK_RUNNING); - ret = unqueue_multiple(vs, count); + ret = futex_unqueue_multiple(vs, count); if (ret >= 0) return ret; From patchwork Fri Jul 28 16:42:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 127783 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp589404vqg; Fri, 28 Jul 2023 10:32:25 -0700 (PDT) X-Google-Smtp-Source: APBJJlFq6zRCAR+f8rQ/WVs+6Fl5b9g5Lb1CvrU+042WTSHON6d5qGjJheVcQEaNCSv1r0+bJLOd X-Received: by 2002:a17:90b:147:b0:268:42a2:35db with SMTP id em7-20020a17090b014700b0026842a235dbmr2048146pjb.48.1690565544909; Fri, 28 Jul 2023 10:32:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690565544; cv=none; d=google.com; s=arc-20160816; b=ZZfbOQFuX7Fn7bqXvpEfM6eIH2FV80BbwlrPkbbf8W7l2Z7VYjfCHAtz2/4XeXCY8Z ce44FIBNl56Bvb7kxFkpblyvhkTmCvSLn0FuUyqHg1mGTseSqH2fxPvebixTgwPUAClF qemwosbmtUPW/smVobGehpfqg0ZF0p3uNfsf7FM4oEQ1dXEHi06m3gwMix6E/anRwP38 t1L2Y6Kp3qwr0X9vum/0V3as972Kbcxos1tBPSNDOQOeCjE0KJA14Ac3GJA0OyKlIJvU u9F8VXDau6nOSMVzNTMhafbGD7KGa6URwvGH8MUuWuPt5ULzilHSbhAzzUopCTHb7NYd /+jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=drbhhN+DgCzYIDWIkMO3R/dcC4lBZr1F8QCpgicu1Fc=; fh=DnWmIIJo3B+rGTNXkDMzawWuDVsJB9JjWDIPE9Mz7eQ=; b=BZGFxzuucJlmQwzugG0PyZENrNuaOkqOK7RSqq600AtdPHFlVbdGu8CgqU2uAe/G5s 7oDmksoSrQIgl4arf08vCkieBaU8YYGcjlfGRYAF2W+btBOSQbbi4U1LPMhk7THyD9xA ESYj32kiZ/ajJIX4Vory1KDsMZGZbvwhL5+BtsbgcUfs1jxzxmO3jErbblb+dHjx+Bxo E5yCQlb3idm8Mfefs1/E95ddKyg/1S5Z7mqsSqlUBASq8KHfnI7CuLujoT7Pm2daICO4 N34PJOdIM52N+NT15ChgYEgtaS894hGiFYX/+qILM3hDXLQTxrsgAKmEa+Kafq7PzG20 KwiA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=KFM1fbdw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kc12-20020a17090333cc00b001b39ba937d1si3199927plb.330.2023.07.28.10.32.10; Fri, 28 Jul 2023 10:32:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20221208.gappssmtp.com header.s=20221208 header.b=KFM1fbdw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234866AbjG1QoB (ORCPT + 99 others); Fri, 28 Jul 2023 12:44:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234035AbjG1Qm7 (ORCPT ); Fri, 28 Jul 2023 12:42:59 -0400 Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9B84422C for ; Fri, 28 Jul 2023 09:42:50 -0700 (PDT) Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-7835bbeb6a0so33280539f.0 for ; Fri, 28 Jul 2023 09:42:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20221208.gappssmtp.com; s=20221208; t=1690562570; x=1691167370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=drbhhN+DgCzYIDWIkMO3R/dcC4lBZr1F8QCpgicu1Fc=; b=KFM1fbdwvRrsljCjrkepuuqyPrAaKzObZiavJxeWD4pfvPJLr3n0YLL31QNQ3DBMve Du1HIhROVhrBmIabwyFRABSuGjA7AQJgCROXuZ/G7IrlyrXuuxtAymS1LRwBiqZbDQtJ di/x2uY+xdGTL+JaoyETBIHeRG4rAL3Hf03nHsjbDvESUBgopLYmY2upYfbAJu/vN0is jFPIf+ySmbAKPSazj14/5lJE3KmZt6EAc4Sr1kikqjwBCxTdohs3ndVUQxnNmjzxb+AC WykeQHIVt4o4J0Vr02vhs4xeuTfZLVAca9zufF3LtJuLpZoZzrFDd3ao8MVCbd8yHNIi S4mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690562570; x=1691167370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=drbhhN+DgCzYIDWIkMO3R/dcC4lBZr1F8QCpgicu1Fc=; b=T8TOi5BiZlOWhgg/lfoUZtH70jYDKRGYeJlkCBI6MxrliXZ8kVGvUHe8jc5adQUVfk azKZ0Xk7GiMYIDyRG5dhHGC+DZbzp6FuOYpf/5G2o6BYqapL0dWCRzD60NCs6UI68gr+ zBkzA7C0oIJ+YNRAuNcn0nLpyNwOO4HmYrqLBr2OPfr2jXLCjMLlqqET1TkknLNrzfVg RMMAADeC1jutvDtsKvvb/+6otZbBf0FNvArz6jFEoqC8T0ywSPiicXyJHBRAeoSoQiyl oxVvp0uOzEJfsrCgVcxvcQJM6PYBWvIY5xol4RcXgM7of6y+aW1h2HMVxht0TVlNppY+ GNIw== X-Gm-Message-State: ABy/qLYeUiSbHqYGH1cLpB2aYLpgw4Iq5lAXmbAZN8hjZkG6YE3plafE 2KaWajdZmcU7FLTOZ+YpQCE21Q== X-Received: by 2002:a92:dc51:0:b0:346:1919:7cb1 with SMTP id x17-20020a92dc51000000b0034619197cb1mr43797ilq.2.1690562570053; Fri, 28 Jul 2023 09:42:50 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id b2-20020a029a02000000b0042b37dda71asm1158808jal.136.2023.07.28.09.42.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Jul 2023 09:42:49 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org, linux-kernel@vger.kernel.org Cc: peterz@infradead.org, andres@anarazel.de, tglx@linutronix.de, Jens Axboe Subject: [PATCH 12/12] io_uring: add support for vectored futex waits Date: Fri, 28 Jul 2023 10:42:35 -0600 Message-Id: <20230728164235.1318118-13-axboe@kernel.dk> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230728164235.1318118-1-axboe@kernel.dk> References: <20230728164235.1318118-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772686456582973416 X-GMAIL-MSGID: 1772686456582973416 This adds support for IORING_OP_FUTEX_WAITV, which allows registering a notification for a number of futexes at once. If one of the futexes are woken, then the request will complete with the index of the futex that got woken as the result. This is identical to what the normal vectored futex waitv operation does. Use like IORING_OP_FUTEX_WAIT, except sqe->addr must now contain the a pointer to a struct futex_waitv array, and sqe->off must now contain the number of elements in that array. Waiting on N futexes could be done with IORING_OP_FUTEX_WAIT as well, but that punts a lot of the work to the application: 1) Application would need to submit N IORING_OP_FUTEX_WAIT requests, rather than just a single IORING_OP_FUTEX_WAITV. 2) When one futex is woken, application would need to cancel the remaining N-1 requests that didn't trigger. While this is of course doable, having a single vectored futex wait makes for much simpler application code. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 1 + io_uring/futex.c | 164 ++++++++++++++++++++++++++++++++-- io_uring/futex.h | 2 + io_uring/opdef.c | 11 +++ 4 files changed, 169 insertions(+), 9 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 3bd2d765f593..420f38675769 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -238,6 +238,7 @@ enum io_uring_op { IORING_OP_SENDMSG_ZC, IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAKE, + IORING_OP_FUTEX_WAITV, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/futex.c b/io_uring/futex.c index f58ab33bfb32..c7936a34319a 100644 --- a/io_uring/futex.c +++ b/io_uring/futex.c @@ -14,10 +14,15 @@ struct io_futex { struct file *file; - u32 __user *uaddr; + union { + u32 __user *uaddr; + struct futex_waitv __user *uwaitv; + }; unsigned long futex_val; unsigned long futex_mask; + unsigned long futexv_owned; u32 futex_flags; + unsigned int futex_nr; }; struct io_futex_data { @@ -44,6 +49,13 @@ void io_futex_cache_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->futex_cache, io_futex_cache_entry_free); } +static void __io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) +{ + req->async_data = NULL; + hlist_del_init(&req->hash_node); + io_req_task_complete(req, ts); +} + static void io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) { struct io_futex_data *ifd = req->async_data; @@ -52,22 +64,59 @@ static void io_futex_complete(struct io_kiocb *req, struct io_tw_state *ts) io_tw_lock(ctx, ts); if (!io_alloc_cache_put(&ctx->futex_cache, &ifd->cache)) kfree(ifd); - req->async_data = NULL; - hlist_del_init(&req->hash_node); - io_req_task_complete(req, ts); + __io_futex_complete(req, ts); } -static bool __io_futex_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) +static void io_futexv_complete(struct io_kiocb *req, struct io_tw_state *ts) { - struct io_futex_data *ifd = req->async_data; + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct futex_vector *futexv = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + int res = 0; - /* futex wake already done or in progress */ - if (!futex_unqueue(&ifd->q)) + io_tw_lock(ctx, ts); + + res = futex_unqueue_multiple(futexv, iof->futex_nr); + if (res != -1) + io_req_set_res(req, res, 0); + + kfree(req->async_data); + req->flags &= ~REQ_F_ASYNC_DATA; + __io_futex_complete(req, ts); +} + +static bool io_futexv_claimed(struct io_futex *iof) +{ + return test_bit(0, &iof->futexv_owned); +} + +static bool io_futexv_claim(struct io_futex *iof) +{ + if (test_bit(0, &iof->futexv_owned) || + test_and_set_bit(0, &iof->futexv_owned)) return false; + return true; +} + +static bool __io_futex_cancel(struct io_ring_ctx *ctx, struct io_kiocb *req) +{ + /* futex wake already done or in progress */ + if (req->opcode == IORING_OP_FUTEX_WAIT) { + struct io_futex_data *ifd = req->async_data; + + if (!futex_unqueue(&ifd->q)) + return false; + req->io_task_work.func = io_futex_complete; + } else { + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + + if (!io_futexv_claim(iof)) + return false; + req->io_task_work.func = io_futexv_complete; + } hlist_del_init(&req->hash_node); io_req_set_res(req, -ECANCELED, 0); - req->io_task_work.func = io_futex_complete; io_req_task_work_add(req); return true; } @@ -146,6 +195,54 @@ int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static void io_futex_wakev_fn(struct wake_q_head *wake_q, struct futex_q *q) +{ + struct io_kiocb *req = q->wake_data; + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + + if (!io_futexv_claim(iof)) + return; + if (unlikely(!__futex_wake_mark(q))) + return; + + io_req_set_res(req, 0, 0); + req->io_task_work.func = io_futexv_complete; + io_req_task_work_add(req); +} + +int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct futex_vector *futexv; + int ret; + + /* No flags or mask supported for waitv */ + if (unlikely(sqe->fd || sqe->buf_index || sqe->file_index || + sqe->addr2 || sqe->addr3)) + return -EINVAL; + + iof->uaddr = u64_to_user_ptr(READ_ONCE(sqe->addr)); + iof->futex_nr = READ_ONCE(sqe->len); + if (!iof->futex_nr || iof->futex_nr > FUTEX_WAITV_MAX) + return -EINVAL; + + futexv = kcalloc(iof->futex_nr, sizeof(*futexv), GFP_KERNEL); + if (!futexv) + return -ENOMEM; + + ret = futex_parse_waitv(futexv, iof->uwaitv, iof->futex_nr, + io_futex_wakev_fn, req); + if (ret) { + kfree(futexv); + return ret; + } + + iof->futexv_owned = 0; + req->flags |= REQ_F_ASYNC_DATA; + req->async_data = futexv; + return 0; +} + static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q) { struct io_futex_data *ifd = container_of(q, struct io_futex_data, q); @@ -170,6 +267,55 @@ static struct io_futex_data *io_alloc_ifd(struct io_ring_ctx *ctx) return kmalloc(sizeof(struct io_futex_data), GFP_NOWAIT); } +int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); + struct futex_vector *futexv = req->async_data; + struct io_ring_ctx *ctx = req->ctx; + int ret, woken = -1; + + io_ring_submit_lock(ctx, issue_flags); + + ret = futex_wait_multiple_setup(futexv, iof->futex_nr, &woken); + + /* + * The above call leaves us potentially non-running. This is fine + * for the sync syscall as it'll be blocking unless we already got + * one of the futexes woken, but it obviously won't work for an async + * invocation. Mark us runnable again. + */ + __set_current_state(TASK_RUNNING); + + /* + * We got woken while setting up, let that side do the completion + */ + if (io_futexv_claimed(iof)) { +skip: + io_ring_submit_unlock(ctx, issue_flags); + return IOU_ISSUE_SKIP_COMPLETE; + } + + /* + * 0 return means that we successfully setup the waiters, and that + * nobody triggered a wakeup while we were doing so. < 0 or 1 return + * is either an error or we got a wakeup while setting up. + */ + if (!ret) { + hlist_add_head(&req->hash_node, &ctx->futex_list); + goto skip; + } + + io_ring_submit_unlock(ctx, issue_flags); + if (ret < 0) + req_set_fail(req); + else if (woken != -1) + ret = woken; + io_req_set_res(req, ret, 0); + kfree(futexv); + req->flags &= ~REQ_F_ASYNC_DATA; + return IOU_OK; +} + int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); diff --git a/io_uring/futex.h b/io_uring/futex.h index ddc9e0d73c52..0847e9e8a127 100644 --- a/io_uring/futex.h +++ b/io_uring/futex.h @@ -3,7 +3,9 @@ #include "cancel.h" int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags); +int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags); int io_futex_wake(struct io_kiocb *req, unsigned int issue_flags); #if defined(CONFIG_FUTEX) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index c9f23c21a031..b9e1e12cac9c 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -443,6 +443,14 @@ const struct io_issue_def io_issue_defs[] = { .issue = io_futex_wake, #else .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_FUTEX_WAITV] = { +#if defined(CONFIG_FUTEX) + .prep = io_futexv_prep, + .issue = io_futexv_wait, +#else + .prep = io_eopnotsupp_prep, #endif }, }; @@ -670,6 +678,9 @@ const struct io_cold_def io_cold_defs[] = { [IORING_OP_FUTEX_WAKE] = { .name = "FUTEX_WAKE", }, + [IORING_OP_FUTEX_WAITV] = { + .name = "FUTEX_WAITV", + }, }; const char *io_uring_get_opcode(u8 opcode)