From patchwork Thu Dec 1 18:11:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28492 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp412386wrr; Thu, 1 Dec 2022 10:13:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf4LWYTXEP1Fci6jZurMeyFIWHejQzy3ziOrLeuvVZ1h4tS/YM2Vd5QtLOYpmRNtNMNpEt6l X-Received: by 2002:a17:906:9c87:b0:7b2:e875:626b with SMTP id fj7-20020a1709069c8700b007b2e875626bmr6136374ejc.373.1669918424058; Thu, 01 Dec 2022 10:13:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918424; cv=none; d=google.com; s=arc-20160816; b=QVARMvtgSrZWAEE6hJg+wjsEZk3Mfb2GlwEBBZnOYoLsNufGOw0b5QmYFi8SVCtERd PGa2jD1bBmmlkEsH6bpodxoqjlKyYGFOe3ouUTbYXlDbc5Z2YdxNg5/OjG/0f8H9njfA NgrOxT8kLGRwhzb/1onsDyfWGtYshYKb80pwwMmGHwkHABdwOMJ2uMhhU13GXCQc6A7M HQCOSf1HiKgWsHRH8Naxg7z8ZcFia8y7I44T+hdgOr4b4ZZuT+IkUhvHe/KWCVy9u4i8 53oUaehmVYa78+wUrkPfr1hj7K1/hxNHD1k7e4M6VliqU9sECAkV6B5SysJu07stoF0L ra2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8PHeEQvI03GO1u4arOs5eKGxrrWj+QcJXenB1+d+Ev4=; b=sJ7FYRuFh57nzi8NrbNLqzSOBNzoRxL6LtIquU0RngC3INbEmHe4bciOXEhEbot5AC ObZs70ctycoLdJDBSqzpvoTS6Y1ubhT6YFAlKXAFOoZwaFBCUSx5W91aSKYqoV9z+RdO 3eNyYvQ0uX6viSG6gM1fa8UyJFgQo11++SOaYS4rdfgQz9MeHEh+AJdXCPuwv2zC8HGW g5eXyUKzNCQ8yYE0p+VFwfIS5I5Il4I5RfOhO6emA1d9iu4W+0jipQ35K93YZqvzAx5t V31VdC0OiQB1wlTj5xyrvhivbmIm37o2tSHDWA0iAstO7JKrr1QUkOMfgoSZjjL/NEEc Ixbw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b="HZsYyn/z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m9-20020a056402430900b0046ab1d20ca2si4228705edc.439.2022.12.01.10.13.19; Thu, 01 Dec 2022 10:13:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b="HZsYyn/z"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230251AbiLASMK (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229839AbiLASMG (ORCPT ); Thu, 1 Dec 2022 13:12:06 -0500 Received: from mail-il1-x132.google.com (mail-il1-x132.google.com [IPv6:2607:f8b0:4864:20::132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D16AFB7DE8 for ; Thu, 1 Dec 2022 10:12:05 -0800 (PST) Received: by mail-il1-x132.google.com with SMTP id bp12so1089892ilb.9 for ; Thu, 01 Dec 2022 10:12:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8PHeEQvI03GO1u4arOs5eKGxrrWj+QcJXenB1+d+Ev4=; b=HZsYyn/z6EZxF36zedEHApAUBJ4P2dRhxWAAmEyxvmr310UQWzVRPVbApPIsSlk1o/ rgSJjhyFcbZgkEzBYSXVuoKf9N6ik+dqn61leEofV6Rr7hRtWk0lyGfnaKdO0PG7+u/+ 44cZp9cYXcWPQk/YuxfkuaNFMmi/6uKPYn8HnxVOej4A7MQVSb/ZXS0V5Rpe6SQ0IeAo NSmxA1q2sZBjPXe37/6uKtE1H4SbpwEB8Gu314VMk29RlzXWaHSGKH7Ri0Yr5PRMbyvl m8y70Uk6erMJKkVghZF++Qv7+nRhEa2ncIOxeLZqW+GFWUPYdDyxCEI0gYmrEfKlxpaN VJ6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8PHeEQvI03GO1u4arOs5eKGxrrWj+QcJXenB1+d+Ev4=; b=gdVXWOslFx3XPFWIyyDUr6LrraTAK5/0KEl2e54RlRRIeakValmWUFELU3wNsVNe2y WrAFJsp4p4/PbVmfbLbYli3xf6rc3LsVPhm/IrM/H6j3OSwSwuQDLLSVNlPQHZo/kTIL 4cS3sQR4SGErpU0p4o4vbhCRezo6v2ldE4gzLt0mXEalQ1LXFrwH88NzebsdHFS1OD3K VMtA8M7oCgio+U5vvY8czYsR1q2qQaFb0RbOvsmLC7wkXa/BvlJS032OSIAwDE0e4fVJ gBcgG6IgxIYbAvi6KIwWJlmeVKgr+QNUACL+Je1mTB/B08RQIVxttl4SEbxmaXTI4sGw recQ== X-Gm-Message-State: ANoB5plN6ERdTZje4Mndf1lFITlil1IJDSp0QcKUPqg2JWLkNuUByVDf 4nvDQxyLBkNAorOs+2+kTykWR6eMeZKaOGBh X-Received: by 2002:a05:6e02:1251:b0:303:1c15:2818 with SMTP id j17-20020a056e02125100b003031c152818mr8384519ilq.87.1669918324962; Thu, 01 Dec 2022 10:12:04 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:04 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 1/7] eventpoll: cleanup branches around sleeping for events Date: Thu, 1 Dec 2022 11:11:50 -0700 Message-Id: <20221201181156.848373-2-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036381016334607?= X-GMAIL-MSGID: =?utf-8?q?1751036381016334607?= Rather than have two separate branches here, collapse them into a single one instead. No functional changes here, just a cleanup in preparation for changes in this area. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 52954d4637b5..3061bdde6cba 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1869,14 +1869,15 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * important. */ eavail = ep_events_available(ep); - if (!eavail) + if (!eavail) { __add_wait_queue_exclusive(&ep->wq, &wait); - - write_unlock_irq(&ep->lock); - - if (!eavail) + write_unlock_irq(&ep->lock); timed_out = !schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS); + } else { + write_unlock_irq(&ep->lock); + } + __set_current_state(TASK_RUNNING); /* From patchwork Thu Dec 1 18:11:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28495 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp412927wrr; Thu, 1 Dec 2022 10:14:48 -0800 (PST) X-Google-Smtp-Source: AA0mqf4hAo0ZhRQVNZikCYIzr3bl203sfO8vqQYQxgXZQvAV0jj9SvkmCQn8LOnbNU84lJmz9GLh X-Received: by 2002:a05:6402:a41:b0:462:273b:6b5c with SMTP id bt1-20020a0564020a4100b00462273b6b5cmr59335924edb.57.1669918488076; Thu, 01 Dec 2022 10:14:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918488; cv=none; d=google.com; s=arc-20160816; b=QqlRZXeIhNrgtUT79C1OaEDaRBleEPNM2JvDayEoXrlmkwQ9bvKmegsubRTmSjrVcs F6vVcid/HkkKG+R5h3m08e2HqQMKP9mYKIYHf3mjz8B5IYt3HXXpq4DcoACXYPK5+Zug bySyLJPULCjRyQjQe2PaxeSEgWGDiUjcNSswtdgXkWTnjDnt7xZhiFHSNaPk4xFv7Woj TxFaYCFT30nHeUrtLLoHp1BU5p64Hg0cR2+kTJhRMHfZZ+i+uegW/fB2VV24WS+dyWsE 6hTtvJQx+mzRFJ0A2WOWRAzUP1a+jgUFAQFzZ7xhLn3OVpzSkjLEYAW9FR8SaXiNLCp3 576g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TCMSGu8wCAuxi+YEaeqt2Nm6ljarPNoUCebXSLfB1Fw=; b=kcEYp0hIRbEvwaogDshMNpCm5o9s7d97h7A+w8Tdg+kfGSQzu3mvSvnHbPv+yRjjHn lAN81pdbf6Ps+/FmjO28g20TVkcxBfB+22+oiXozh/Vgc530qLg/Piz4r3oG9/ORmWxa iqe1uaoBHQy7PcbIlNxlVU6q6QTZXBRvG4CeZNXJLUtQ6DQhsX7nEBd9qxULJ3Z625QE JDOmTAIyl44cBXNZP/QDai0DZwP5ccYe/HdxIB8W27TUIhyNEB+1ouAmA+OewbiFeJy2 39ZuQZKTaYCxMDSAz6saHxf5lJ998WDvfAtjPDqfOJR7cgs5pNLrx3ytteHf73sjXraI qSCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=7rToxcBA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w16-20020a05640234d000b0046b999e8862si4199473edc.20.2022.12.01.10.14.21; Thu, 01 Dec 2022 10:14:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=7rToxcBA; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230270AbiLASMN (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229608AbiLASMH (ORCPT ); Thu, 1 Dec 2022 13:12:07 -0500 Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B6674B7DD3 for ; Thu, 1 Dec 2022 10:12:06 -0800 (PST) Received: by mail-io1-xd32.google.com with SMTP id z131so1580267iof.3 for ; Thu, 01 Dec 2022 10:12:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TCMSGu8wCAuxi+YEaeqt2Nm6ljarPNoUCebXSLfB1Fw=; b=7rToxcBAjNQi+teOR/HtEPsnLOYMOnWUeha2jHX+ubOPphUUlFzGLJ4Fe1IYPdO1VW enEBIs7LitVgRCKTuHrjuThBnVfxtBFEMCju3P6P9WcCV1t9LWJhlo3mkRwr6SrAwO1D yoqZbuamMTV2WlHMUkepdyL5Xqor2YcYSaaTktTtZNAFOZhGhF4FUnPijBD3hwA7DvNH RHAlKTRVQqxVvJvcBxfqIqMWd6mNhSUYAzvuqN6ViWqrv4BAvQm18wBgyzIwfR7zpR7X 1mkSD3MYCxFWbk4qT1FMG9SddowhtGk3HGi6pPUSrixd5nXBqwQXeQ8hhBpKqqvpGASM IBMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TCMSGu8wCAuxi+YEaeqt2Nm6ljarPNoUCebXSLfB1Fw=; b=Cxw28E7pwoq4K4BrdIEYGP8RCkV2mxuTai8ZDoofl888XNGfS1hQeImir9B0cNJdu6 h+QH8Z/6AROCUpBO0mDMNb7T95bXZQG/Hvzn+OkorcRrLY1WUrAQguIMWyXMjeWbi4ra zB/N6EG45OIJTFjiukEfdziTek6pW3sVn1fwV7a4WIyZe0tAUcb24IAgXP3lGhHEeeJF W0qhCDMI47MQB32k5/gnGjf6AjiwWVZ/8crpYsCepp8eVC/cuUK2bkL+COF+mAS+CHk+ mJPs3ahhmhcbb2dc62bwuWCf6eJCBURJgrn29GFE2pSo/S2D1edaKAKHv1gltL0grySA tb4w== X-Gm-Message-State: ANoB5pnn7pkEG/qXzzkzh+ia3h+NB8z7RPjYFmwnDSRvXRhAErBg1t0T ymPR1mrqL0epAW1vGL2TT0xgyaE2aYpw8N+w X-Received: by 2002:a05:6638:15cc:b0:389:e983:dfd1 with SMTP id i12-20020a05663815cc00b00389e983dfd1mr8192666jat.306.1669918325874; Thu, 01 Dec 2022 10:12:05 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:05 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 2/7] eventpoll: don't pass in 'timed_out' to ep_busy_loop() Date: Thu, 1 Dec 2022 11:11:51 -0700 Message-Id: <20221201181156.848373-3-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036448129712785?= X-GMAIL-MSGID: =?utf-8?q?1751036448129712785?= It's known to be 'false' from the one call site we have, as we break out of the loop if it's not. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3061bdde6cba..64d7331353dd 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -396,12 +396,12 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) * * we must do our busy polling with irqs enabled */ -static bool ep_busy_loop(struct eventpoll *ep, int nonblock) +static bool ep_busy_loop(struct eventpoll *ep) { unsigned int napi_id = READ_ONCE(ep->napi_id); if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { - napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, + napi_busy_loop(napi_id, ep_busy_loop_end, ep, false, BUSY_POLL_BUDGET); if (ep_events_available(ep)) return true; @@ -453,7 +453,7 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) #else -static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) +static inline bool ep_busy_loop(struct eventpoll *ep) { return false; } @@ -1826,7 +1826,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) return 0; - eavail = ep_busy_loop(ep, timed_out); + eavail = ep_busy_loop(ep); if (eavail) continue; From patchwork Thu Dec 1 18:11:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28493 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp412746wrr; Thu, 1 Dec 2022 10:14:29 -0800 (PST) X-Google-Smtp-Source: AA0mqf4/AIB3SU0OwwWpa+Fbp/wodq8Lr7zU69nW7EH4V4vXVmWMHJae5lWxutA7zdD/z9Ut+L6u X-Received: by 2002:a17:906:7d50:b0:79e:4880:dd83 with SMTP id l16-20020a1709067d5000b0079e4880dd83mr47026215ejp.166.1669918469298; Thu, 01 Dec 2022 10:14:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918469; cv=none; d=google.com; s=arc-20160816; b=ZaiW6zCB81JbqZHJ5e+C13PleUpUw6uznCNah0x6N3GjKQOb7OXqSzmbb5UuuG4Nul ys+H8O0pAb9nb2ba5yItGBT79xmwq8QxgItWmmOULjXrmUef5O/13eGCiRAmuyth5FHM vCal1odH9r0/1N1ISMv5ispzp7WrP+d+7QRStKzFTa2lvLSNTIvbHZcZSlHEkWMJTwng vLWsV3DHL59iUWL6UBGaasb85IiHPjEea2eL9xXK0Qn7XfWGiU5vYurYTq/mXgZQjCzd ssB874CpfIkaoY3A8WqWyL3kBkE2u5sdajzl9Hq5v1JwIEc4aYbGA8ztj0WAPDErMD7z 3MQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=d8OpHlPRST08skeh7BHSZbFN09k6AB6QzxLidf3Pxy0=; b=Sux+kaSioVR/ii3Ake/9Krs/g4deasNj0ZWYkI2SmLtE9d9EPTutUwSiQltX/VeUpK ur4rhRApvgkGmhPuUdu6lFSOwFlUErl1l0pkB1QhiSBoGxWmkDeGxKEfe415RPUTVrxJ 5z6xxYJ56RFELuZUf/gFBYkJ9RqkFZOMVY5oVREeYv69Zh10RbCOGNde6flZZbJDQk2n oP7hVDuSpkae6Z5Kqc1YbtimoHi3JrIay9yUHoTDxlI5e1WrXuU9Lf5Jq1kcm/hZ9qz2 au1IQtpKxeFOsPbS3UXdKBR2IvNWRSLcKTXl5gOyY3oY4eVhRxiYUiWcLFhvGEcSMIBi 9Flw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=AS0uFZT7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sa35-20020a1709076d2300b0078d4c9d77adsi4548778ejc.94.2022.12.01.10.14.05; Thu, 01 Dec 2022 10:14:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=AS0uFZT7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230312AbiLASMR (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230164AbiLASMH (ORCPT ); Thu, 1 Dec 2022 13:12:07 -0500 Received: from mail-il1-x12a.google.com (mail-il1-x12a.google.com [IPv6:2607:f8b0:4864:20::12a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A14EB7DDD for ; Thu, 1 Dec 2022 10:12:07 -0800 (PST) Received: by mail-il1-x12a.google.com with SMTP id m15so1097270ilq.2 for ; Thu, 01 Dec 2022 10:12:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d8OpHlPRST08skeh7BHSZbFN09k6AB6QzxLidf3Pxy0=; b=AS0uFZT7AMQnrCsnu0myce/68PNqhpRsA5u5f4LRNoWMlDHGfdkpnX4Bi7cN5cdYGb xqdvKOmhCzI+5xIdNOa7IXXJKN76QUhQ5bEE0TGc9TT1CzwQWBwxaBo9dcoAm2EXczuO BWbTIe5bxzNBixBo+grxsNP/S6uE+oBHuP7FwcSGjYpqt9pNgmik/w/epK7az0/nTi6J SU1JdTULxEiQvzxCmGA4dAqUAdR7cFcERSrlB3wYXqjxQcTWauNZkbX3up+4lfaLMAno nD9XRVUYifvWGAzNsFQcvl/jDfoyDKtULpSDCOxy/rpRlexRJGGOuiApfmzFdFHr2iUL J55w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d8OpHlPRST08skeh7BHSZbFN09k6AB6QzxLidf3Pxy0=; b=pK6sOFTzgIU/vqAVuCSOipkY0uDCRd0YLzWusWPvM7sS0p8juvdMkKXF/ovetjuiBI gd+drR9dTr8VKwgKq1yTXp3XbUyqgobls6GC20RvST3PUFEwuflnd3K+VzzYfhQgi1hP vB/nhYn4DFlqU+RnhGfXWYn4FLoKZvsebMa41jcuigNQ4KjiWpmcdDXRdfZ9C7CZRiau BtIsAs/cMFHhWHPhfl1zF5Hs5g2vKO+B6kwIdS0oo0dn0gf0xjND2gNcrJNTuwhqlquO miXZvNWyEHZ2M8viatTe8tpfm/FyhMY4D9/CS+tPc5GCHGXVv8YbQKz1aOm29lmtirhV Whtw== X-Gm-Message-State: ANoB5pmHo03BWj47nD3D8nPW1whf6PBhS7c8xlI9JDY0Rb37kO2AjQjs p/iOz8Zg9XBCzlubQql8Tz08gBrCoFnuluCK X-Received: by 2002:a92:d689:0:b0:303:2806:1ca0 with SMTP id p9-20020a92d689000000b0030328061ca0mr5247364iln.247.1669918326690; Thu, 01 Dec 2022 10:12:06 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:06 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 3/7] eventpoll: split out wait handling Date: Thu, 1 Dec 2022 11:11:52 -0700 Message-Id: <20221201181156.848373-4-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036428715187710?= X-GMAIL-MSGID: =?utf-8?q?1751036428715187710?= In preparation for making changes to how wakeups and sleeps are done, move the timeout scheduling into a helper and manage it rather than rely on schedule_hrtimeout_range(). Signed-off-by: Jens Axboe --- fs/eventpoll.c | 68 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 55 insertions(+), 13 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 64d7331353dd..888f565d0c5f 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1762,6 +1762,47 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, return ret; } +struct epoll_wq { + wait_queue_entry_t wait; + struct hrtimer timer; + bool timed_out; +}; + +static enum hrtimer_restart ep_timer(struct hrtimer *timer) +{ + struct epoll_wq *ewq = container_of(timer, struct epoll_wq, timer); + struct task_struct *task = ewq->wait.private; + + ewq->timed_out = true; + wake_up_process(task); + return HRTIMER_NORESTART; +} + +static void ep_schedule(struct eventpoll *ep, struct epoll_wq *ewq, ktime_t *to, + u64 slack) +{ + if (ewq->timed_out) + return; + if (to && *to == 0) { + ewq->timed_out = true; + return; + } + if (!to) { + schedule(); + return; + } + + hrtimer_init_on_stack(&ewq->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS); + ewq->timer.function = ep_timer; + hrtimer_set_expires_range_ns(&ewq->timer, *to, slack); + hrtimer_start_expires(&ewq->timer, HRTIMER_MODE_ABS); + + schedule(); + + hrtimer_cancel(&ewq->timer); + destroy_hrtimer_on_stack(&ewq->timer); +} + /** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer. @@ -1782,13 +1823,15 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, struct timespec64 *timeout) { - int res, eavail, timed_out = 0; + int res, eavail; u64 slack = 0; - wait_queue_entry_t wait; ktime_t expires, *to = NULL; + struct epoll_wq ewq; lockdep_assert_irqs_enabled(); + ewq.timed_out = false; + if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); to = &expires; @@ -1798,7 +1841,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Avoid the unnecessary trip to the wait queue loop, if the * caller specified a non blocking operation. */ - timed_out = 1; + ewq.timed_out = true; } /* @@ -1823,7 +1866,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, return res; } - if (timed_out) + if (ewq.timed_out) return 0; eavail = ep_busy_loop(ep); @@ -1850,8 +1893,8 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * performance issue if a process is killed, causing all of its * threads to wake up without being removed normally. */ - init_wait(&wait); - wait.func = ep_autoremove_wake_function; + init_wait(&ewq.wait); + ewq.wait.func = ep_autoremove_wake_function; write_lock_irq(&ep->lock); /* @@ -1870,10 +1913,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, */ eavail = ep_events_available(ep); if (!eavail) { - __add_wait_queue_exclusive(&ep->wq, &wait); + __add_wait_queue_exclusive(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); - timed_out = !schedule_hrtimeout_range(to, slack, - HRTIMER_MODE_ABS); + ep_schedule(ep, &ewq, to, slack); } else { write_unlock_irq(&ep->lock); } @@ -1887,7 +1929,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, */ eavail = 1; - if (!list_empty_careful(&wait.entry)) { + if (!list_empty_careful(&ewq.wait.entry)) { write_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, @@ -1896,9 +1938,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Thus, when wait.entry is empty, it needs to harvest * events. */ - if (timed_out) - eavail = list_empty(&wait.entry); - __remove_wait_queue(&ep->wq, &wait); + if (ewq.timed_out) + eavail = list_empty(&ewq.wait.entry); + __remove_wait_queue(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); } } From patchwork Thu Dec 1 18:11:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28494 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp412895wrr; Thu, 1 Dec 2022 10:14:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf5RyPc63/0h9JzqYTZp63bFioFCffnmVT5V5s0Scr51QXmEUue/9NQjwcMI8nClZitlCH6D X-Received: by 2002:a17:906:64e:b0:7b7:c1b9:8cb4 with SMTP id t14-20020a170906064e00b007b7c1b98cb4mr41678588ejb.96.1669918484098; Thu, 01 Dec 2022 10:14:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918484; cv=none; d=google.com; s=arc-20160816; b=IAVhN300Zbs65a8yOIrrZaj2812GQmuYxjyhDbuX9KLanNzzsdJChaIpokXSKSAcMD OLQSrdqMjz23FvcMoH+93xUhNHNvl+gttEVitiYKZkjVP6rj93IlN6RhlqngV3ooeQsO 9ESSywPt/a3blctc2OlWvI3iOwC2FTe7MtmXUh1dGRBzvf6DS6Jvq56RXSqO1Shwe5g/ twG9tG97zYLFEircQbSrYPWEQjqR90VDbsPC0Bhb1u0GEeJCOHKyyWSZNajA9IbzOMXA Be4pgul5izHgh2KQ4ePUAeS7cYPoNvXPUfY8ZJwG0I0DePRkzz+qVq0fZv33H0HnvZ/c z4Sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+7XieXpcJrzhapQHtWCNoW0rQy5yiKhkq34SDOyw0xw=; b=ghCz12K3kmB4rpwYyLZTKYYFUoV5ezfWTD+8/4RUPMIZQfB5ZxlOYswiEevDPzg4Yv 8Ws53mPOt2Ysrs9/zBaqvVynszVR5hL5s6A2awSiStmKAO0G9AGdYxqqO2e9ZyOLZwWg wvMLAOwzjkn+xDJdHLY/45cr0VR36ZTIYhSuC/VF3Q7J1AQ1/Z6rFVcB+A3+ZrLIRDKl f3TL6SaFgJnWlwcNdlivN2CdxeTlP4uEw9ZiEb5lrieJTMGwjf3rZ6pIgs00FeKamvm6 u5ZsbXr/QcKhTRTIZ/HHOON6XBZu/YidTcD+IgY1wE2mkUxKTzTVbr3AhO7MbQLVfeZR FmhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=rCV2Kahu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o13-20020aa7c50d000000b0046abe65b8f7si4037337edq.159.2022.12.01.10.14.20; Thu, 01 Dec 2022 10:14:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=rCV2Kahu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230185AbiLASMX (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230218AbiLASMI (ORCPT ); Thu, 1 Dec 2022 13:12:08 -0500 Received: from mail-io1-xd2e.google.com (mail-io1-xd2e.google.com [IPv6:2607:f8b0:4864:20::d2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BA7AA1C0A for ; Thu, 1 Dec 2022 10:12:08 -0800 (PST) Received: by mail-io1-xd2e.google.com with SMTP id n188so1570042iof.8 for ; Thu, 01 Dec 2022 10:12:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+7XieXpcJrzhapQHtWCNoW0rQy5yiKhkq34SDOyw0xw=; b=rCV2KahuRSA2UomgHjcLL2MPSHWQzvEOZHnqXl9CNih1pE+gZRpjzdYg3AGoNxGtH/ lrwbTserdKCot3tq/5ePqPYHCEDByLsr5EJeyAa3WUpPRYldyDwDb72H4U+e7EULEDfX YOz044AliNTRJSeVBbissUP76TH1dDpAxrwe5pgd94G4wvCJ4IeVnDfJhMx47sKWeWPs 0C7kWE1xTeT+Dx73CkQ/oLdCov1SSDwhGCyL0CKWRohvyTJvqo4qo5nur82aLKeNrnxf rQRbX7ukSMrqtcrZIvRHBe9oddDEKsDHVoB5vCoS6H8vIAXT8EF6vPM5WVzceR1ifqHH kwow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+7XieXpcJrzhapQHtWCNoW0rQy5yiKhkq34SDOyw0xw=; b=ECi0HhtZe8LEBgsi6nEXZ2dzxsh3uzYYE7KRVome7IhNOqEutPwIdNHRjHihNtaILC GukkJjarid8/Sq/bRhfOxqlVMYLZWzTQ39wAMTrnrQp26nnAmG6YQvbhAIRZSd5bAh3e s+OA1K8DH/z2lG2jdpLQXz4gdhAtWAPiviJGSBH4Sv1zrlPSUtaueM9vkZsw4aCdCLWQ +iYFIvSIRtquRc80ZwWpH6uxMZ/JUFkm4DUJ8Wgs6XQtISGIPhpWwGStOuD/EZ4WjIg+ VhTRlCJKWQnkDkpFgMGvIk3lbtzZLIucBQ3FZthHauWelHI8pPbAv9QmHAyZq+LHl2U+ +aVA== X-Gm-Message-State: ANoB5plKH2CiAggMn5sW2lsOfyuajFY+c5i1S66NkZZDPWbxcvbFLNqI JeBlwhz+YwE6EoDYBJk7c9kbfcO96Knf9rLM X-Received: by 2002:a02:a710:0:b0:389:d089:4233 with SMTP id k16-20020a02a710000000b00389d0894233mr12448957jam.18.1669918327697; Thu, 01 Dec 2022 10:12:07 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:07 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 4/7] eventpoll: move expires to epoll_wq Date: Thu, 1 Dec 2022 11:11:53 -0700 Message-Id: <20221201181156.848373-5-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036444366481746?= X-GMAIL-MSGID: =?utf-8?q?1751036444366481746?= This makes the expiration available to the wakeup handler. No functional changes expected in this patch, purely in preparation for being able to use the timeout on the wakeup side. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 888f565d0c5f..0994f2eb6adc 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1765,6 +1765,7 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, struct epoll_wq { wait_queue_entry_t wait; struct hrtimer timer; + ktime_t timeout_ts; bool timed_out; }; @@ -1825,7 +1826,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, { int res, eavail; u64 slack = 0; - ktime_t expires, *to = NULL; + ktime_t *to = NULL; struct epoll_wq ewq; lockdep_assert_irqs_enabled(); @@ -1834,7 +1835,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); - to = &expires; + to = &ewq.timeout_ts; *to = timespec64_to_ktime(*timeout); } else if (timeout) { /* From patchwork Thu Dec 1 18:11:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28496 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp412970wrr; Thu, 1 Dec 2022 10:14:55 -0800 (PST) X-Google-Smtp-Source: AA0mqf77KCIA0SbPrF+GisrKSoijz8xHvXQE3sZTRTYLQe1b35vSg77C0pWjlOR/pfKXnSaXy+xv X-Received: by 2002:a17:906:d297:b0:7b8:4c22:2d6c with SMTP id ay23-20020a170906d29700b007b84c222d6cmr41757003ejb.144.1669918495397; Thu, 01 Dec 2022 10:14:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918495; cv=none; d=google.com; s=arc-20160816; b=IX/ROYnNM1EvXb3Qg5mlL51kU1YITqlZCawUPWwrlBLrhyJEzC4Z5u5W63dirU10Yb H9Mda787lej7JEb9VZcEnq3n3fdBv/6yskn9qVnlikYpU2ezf2KERixz297PLODWGrVC E+5ZUi9tbBzVjfSE0G7gpbpqwdbCELSI31hIA1MJnS5yIuRp54lRpyRZYiRNC1G3+TO+ SlT1ZY5JkpOSzWqvlmShM9Wjebi1KCrX0T/mpo7dzoh+tZgw4yK5TBuxtBZyGdQgeq0W wSI8mVF9BQ3wsjGEtcNWFDfL/3qBpEJU5HfqPTkFI9XYwDAZWYeQIR2fqWo2gcFpwa85 JG1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T2Hzb+xxyKnPzMsKtR+WvmF1ywGYO4tzf59V+gc/0/s=; b=Klc6LQy2aW+qtu2i/iQXROyZCZ9T8+EcnGGUBapcSAVm5VM3onezDdMMT6sqSHsUDm YYyV+CDIfeXbF6AXMtBJ2KyG9uSC00H5DODTwO7DCHCrMSvpfRKKE7GeCL3WLpS3Sw3h Mw/xJWMcHlOaB1mxngeNqUVbo1vGtnXDg4seCYHbJ8GqN/mDuHHTA5gV63E46vMD58Me aYZ+2Xw0OP9l0RFT5kEuiWZbnIzza23Jke+oTFfjFXaOcIsB/h0Is9D9BbzfxPzDG5kr TnHTNn27Us1qnw2jWAnh6F5qyCwRslUX6GHU53H8vl7+Q1398S2HroUbhLJ+SgL0wlTr 3okg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=2atU+qY8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hh17-20020a170906a95100b007abafd4d7d0si3334671ejb.702.2022.12.01.10.14.31; Thu, 01 Dec 2022 10:14:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=2atU+qY8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230440AbiLASMa (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36778 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230359AbiLASMS (ORCPT ); Thu, 1 Dec 2022 13:12:18 -0500 Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4B4FB8455 for ; Thu, 1 Dec 2022 10:12:09 -0800 (PST) Received: by mail-io1-xd2c.google.com with SMTP id e189so1586847iof.1 for ; Thu, 01 Dec 2022 10:12:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T2Hzb+xxyKnPzMsKtR+WvmF1ywGYO4tzf59V+gc/0/s=; b=2atU+qY8N6U1TWuEjoXo0Ek+wKx6gWrLYp8D9cZz1SEd++H+eWVuXdr3WUmYAaBF3h CXDqX2fX2YFDXbnYSeIWfgeuJyAoP/MnjAsqnXzRedinp+W7XDc7ppleItMH3Ati2r1r pWU+OXSKGhu/TbrMFfI7gUobNXqYO3tuC+SBghlInhOM9GAdacLA5zPErWxgwlToLDPk VLBDRdh/ETAMA76uckSDzQRnLzivJQnR0ZzmDxBpX9FrTbWFBuexQuphd37ILJwWHd0E /cfJYCD7LwtgUHaWTI2vY2Waup5fInS5DVQuUsNCxIiaNr7qjw7Sm35SjXYyN9jMkbTq pfSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T2Hzb+xxyKnPzMsKtR+WvmF1ywGYO4tzf59V+gc/0/s=; b=olwCyHB2iE2I/lFNwSoWDgEQmxl1GXmyog7O4fQMkG2pLYWJ/O4snC+Rp5TczHbXsJ lFmGCv+0T+tMDxoCVqLZ7O5wKhThgHcEUZcr/vJrxjfqg1SjYmNwacKFxz54V08bZUkB Q8H4P8zHATkFJ/NBm+ib4KAuJKW9BkoZ26WTN2+aaTZoqv6IM9PolwZ+7Ft4i71nEk77 A23DAcSJENn3VI6azBaiSTlGxPnTcwg/jdLRri2TJEU75OHYelkg+Ct9KjcqH62ySarL jTmkmm2Hn1w+HUtsSuEB/bsJum8M53xhcHaL02uliOPek/vT96GnBFyIfejUj6ieO8w3 p9vg== X-Gm-Message-State: ANoB5pn+cpGTZbdsuQdXL6AC9ypvPJS5SJh+uLBjC2fADYgJSj2Omw8A Y7DkrOnjZFrmti8SKksWPn844GbmS62ffDzl X-Received: by 2002:a02:16c8:0:b0:38a:c4d:931f with SMTP id a191-20020a0216c8000000b0038a0c4d931fmr3207615jaa.176.1669918328499; Thu, 01 Dec 2022 10:12:08 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:08 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 5/7] eventpoll: move file checking earlier for epoll_ctl() Date: Thu, 1 Dec 2022 11:11:54 -0700 Message-Id: <20221201181156.848373-6-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036456269849479?= X-GMAIL-MSGID: =?utf-8?q?1751036456269849479?= This just cleans up the checking a bit, in preparation for a change that will need access to 'ep' earlier. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 0994f2eb6adc..962d897bbfc6 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -2111,6 +2111,20 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, if (!f.file) goto error_return; + /* + * We have to check that the file structure underneath the file + * descriptor the user passed to us _is_ an eventpoll file. + */ + error = -EINVAL; + if (!is_file_epoll(f.file)) + goto error_fput; + + /* + * At this point it is safe to assume that the "private_data" contains + * our own data structure. + */ + ep = f.file->private_data; + /* Get the "struct file *" for the target file */ tf = fdget(fd); if (!tf.file) @@ -2126,12 +2140,10 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, ep_take_care_of_epollwakeup(epds); /* - * We have to check that the file structure underneath the file descriptor - * the user passed to us _is_ an eventpoll file. And also we do not permit - * adding an epoll file descriptor inside itself. + * We do not permit adding an epoll file descriptor inside itself. */ error = -EINVAL; - if (f.file == tf.file || !is_file_epoll(f.file)) + if (f.file == tf.file) goto error_tgt_fput; /* @@ -2147,12 +2159,6 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, goto error_tgt_fput; } - /* - * At this point it is safe to assume that the "private_data" contains - * our own data structure. - */ - ep = f.file->private_data; - /* * When we insert an epoll file descriptor inside another epoll file * descriptor, there is the chance of creating closed loops, which are From patchwork Thu Dec 1 18:11:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28498 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp414352wrr; Thu, 1 Dec 2022 10:17:45 -0800 (PST) X-Google-Smtp-Source: AA0mqf7w1njBshKMWcOqN6ynMkETAxwdTZmfui0T2OIddw4qc83qVJeUGxP9YswJq9++SzXWlx0A X-Received: by 2002:a17:906:369b:b0:78d:34a:f466 with SMTP id a27-20020a170906369b00b0078d034af466mr57135814ejc.162.1669918665573; Thu, 01 Dec 2022 10:17:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918665; cv=none; d=google.com; s=arc-20160816; b=iI/wXSq9G5JXNUoSnK5U7EYrfbs0M/YVfpBmyFNHSw++vu4/7L5T9PYwy6czXeEGzZ psYrUAFjpX30Kgf25PkLWpmM1GuMhINS2tjvWzwNYVY260aEUw/OoGA8Iy/u11Fe3DMM tcdQ0DvtO1hEh3JlXN22vNuQNlWGTzX0oXJKAn2graebT0wIvaFkS4E9ZPzQNnaRBHQq jMDtItFSjTHybMtFYvT+cSEXY2IXTyPQFKEJmjiVzK1UhXT/BppccmHJwJduk7CT5vDx H2bVIf8rPjqXSljJS9kqBpHJxK4GCkFVqFQ1+cb28Jn8DHx42FONtnF03GI7ooIVuo+y 0JJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mZ9tl6KkNetvRAWXe2iAAf/ML4fBDffEt0E3CXrPzCE=; b=NJGBf0eympqWmXysk5KcuxSOWDgTIehceVZ4c8uOv/Kckth0mENc+Y2SHyN+DaXw0b Z/ptwcr5+vKyHqGrBoKBrX/nf4mBAKIR+KH1vJXEtoW4zBlQ/2tE0f32eEjmkCLM8ZOB 2J77TpJxi/8pE+2r1Ty8pkKxAls9+jcDPnPfKmd4whtw2uBiVf8yIfTNmh8c/lAYskAp 5GGlCnDfxFv3gVUhXJkRINfD7cweft/MK9NJimtSt0uGguaBCoZJlJ85xJ2oDLym7ZTP OUrPnqJFV4Nqfv4Zd2aOrXRiDfdYSmhCh875gtYyrqUd05kOhKDbwGX2DAXtDje3pXQ6 U6Kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=l4wGJmKV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id br13-20020a170906d14d00b007c0a4551010si3220044ejb.613.2022.12.01.10.17.17; Thu, 01 Dec 2022 10:17:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=l4wGJmKV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231158AbiLASMd (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230197AbiLASMV (ORCPT ); Thu, 1 Dec 2022 13:12:21 -0500 Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21C80B845D for ; Thu, 1 Dec 2022 10:12:10 -0800 (PST) Received: by mail-io1-xd32.google.com with SMTP id i80so1606003ioa.0 for ; Thu, 01 Dec 2022 10:12:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mZ9tl6KkNetvRAWXe2iAAf/ML4fBDffEt0E3CXrPzCE=; b=l4wGJmKVt1fPfhJvQBLb5fQkAKw+lUzYeSTILSbU6Uq+yHzEtXv1G/VdU4rETW2k0a WgEKMyF1NjMLgqGbaejKRlOXnP1hSlHSJYkJdgZAmu+SPy+TmPHkrSW13WnaxD5fgD8+ IhvSF9aaSloA3mi1nq9od2wRv3ISoh12Uxm8LSdCFB7H3nEo8Iwfxn4v5r4LfT0JE7yD fnOP0lo0vweKMI4qLFDYINpw9TyXqgvMlnIhKZAEKAmKmUY8fq4/H6wW88GH4paIE4Xr UTHG/9PwV1oEImy4nEfT+GcIgYvHe9ohvof3nu5pVjQ0sd3+wMNkUItC4k8sRXIrWnBp n/Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mZ9tl6KkNetvRAWXe2iAAf/ML4fBDffEt0E3CXrPzCE=; b=OhDoTtCEk2enzFGJn+i8gZrkXZ87NfdC6d4eTLcOGSeFU+0BgFoywLVI971LrNHN/F ME6+x1+YoDikahn+BprfMo2p4ML0i41TawqtlY9FwYk6BRmyxSxhUZBr3K0bjidJHQy1 cS1vaFPIqEXuT7r2FO61CH9+8/RsDCJjxMyVh9/2blv/HUiyMu9Dm2e9+z4v0q4apAlP 7awNbEL4qKmFvzKr9QSalXrLMYm/YV61UVlp2+CX5SjrxoBSP+2gJvpAiIfd7oIZtNbz w01ZKmlOkzGK7urCCcE7WSFCDVMPTebRa6kAfm78D4gDi/Sir9KsFgITwwd5p7dUVPos gQRw== X-Gm-Message-State: ANoB5plkXyF6vdH9O48Z1vHaQCWZ6EB6h+m3evJuEHo2WCnOR0uQ/ZqX zFKw5yFNYrW5uitwiK/+ILw9qwadSX2RHHfj X-Received: by 2002:a6b:fb13:0:b0:6de:383e:4146 with SMTP id h19-20020a6bfb13000000b006de383e4146mr24784036iog.48.1669918329401; Thu, 01 Dec 2022 10:12:09 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:08 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 6/7] eventpoll: add support for min-wait Date: Thu, 1 Dec 2022 11:11:55 -0700 Message-Id: <20221201181156.848373-7-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036634320522374?= X-GMAIL-MSGID: =?utf-8?q?1751036634320522374?= This adds the necessary infrastructure to support a minimum wait for reaping events, API for setting or applying a minimum wait will come in the following patches. For medium workload efficiencies, some production workloads inject artificial timers or sleeps before calling epoll_wait() to get better batching and higher efficiencies. While this does help, it's not as efficient as it could be. By adding support for epoll_wait() for this directly, we can avoids extra context switches and scheduler and timer overhead. As an example, running an AB test on an identical workload at about ~370K reqs/second, without this change and with the sleep hack mentioned above (using 200 usec as the timeout), we're doing 310K-340K non-voluntary context switches per second. Idle CPU on the host is 27-34%. With the the sleep hack removed and epoll set to the same 200 usec value, we're handling the exact same load but at 292K-315k non-voluntary context switches and idle CPU of 33-41%, a substantial win. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 84 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 71 insertions(+), 13 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 962d897bbfc6..daa9885d9c2b 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -117,6 +117,9 @@ struct eppoll_entry { /* The "base" pointer is set to the container "struct epitem" */ struct epitem *base; + /* min wait time if (min_wait_ts) & 1 != 0 */ + ktime_t min_wait_ts; + /* * Wait queue item that will be linked to the target file wait * queue head. @@ -217,6 +220,9 @@ struct eventpoll { u64 gen; struct hlist_head refs; + /* min wait for epoll_wait() */ + unsigned int min_wait_ts; + #ifdef CONFIG_NET_RX_BUSY_POLL /* used to track busy poll napi_id */ unsigned int napi_id; @@ -1747,6 +1753,32 @@ static struct timespec64 *ep_timeout_to_timespec(struct timespec64 *to, long ms) return to; } +struct epoll_wq { + wait_queue_entry_t wait; + struct hrtimer timer; + ktime_t timeout_ts; + ktime_t min_wait_ts; + struct eventpoll *ep; + bool timed_out; + int maxevents; + int wakeups; +}; + +static bool ep_should_min_wait(struct epoll_wq *ewq) +{ + if (ewq->min_wait_ts & 1) { + /* just an approximation */ + if (++ewq->wakeups >= ewq->maxevents) + goto stop_wait; + if (ktime_before(ktime_get_ns(), ewq->min_wait_ts)) + return true; + } + +stop_wait: + ewq->min_wait_ts &= ~(u64) 1; + return false; +} + /* * autoremove_wake_function, but remove even on failure to wake up, because we * know that default_wake_function/ttwu will only fail if the thread is already @@ -1756,27 +1788,37 @@ static struct timespec64 *ep_timeout_to_timespec(struct timespec64 *to, long ms) static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned int mode, int sync, void *key) { - int ret = default_wake_function(wq_entry, mode, sync, key); + struct epoll_wq *ewq = container_of(wq_entry, struct epoll_wq, wait); + int ret; + /* + * If min wait time hasn't been satisfied yet, keep waiting + */ + if (ep_should_min_wait(ewq)) + return 0; + + ret = default_wake_function(wq_entry, mode, sync, key); list_del_init(&wq_entry->entry); return ret; } -struct epoll_wq { - wait_queue_entry_t wait; - struct hrtimer timer; - ktime_t timeout_ts; - bool timed_out; -}; - static enum hrtimer_restart ep_timer(struct hrtimer *timer) { struct epoll_wq *ewq = container_of(timer, struct epoll_wq, timer); struct task_struct *task = ewq->wait.private; + const bool is_min_wait = ewq->min_wait_ts & 1; + + if (!is_min_wait || ep_events_available(ewq->ep)) { + if (!is_min_wait) + ewq->timed_out = true; + ewq->min_wait_ts &= ~(u64) 1; + wake_up_process(task); + return HRTIMER_NORESTART; + } - ewq->timed_out = true; - wake_up_process(task); - return HRTIMER_NORESTART; + ewq->min_wait_ts &= ~(u64) 1; + hrtimer_set_expires_range_ns(&ewq->timer, ewq->timeout_ts, 0); + return HRTIMER_RESTART; } static void ep_schedule(struct eventpoll *ep, struct epoll_wq *ewq, ktime_t *to, @@ -1831,12 +1873,16 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, lockdep_assert_irqs_enabled(); + ewq.min_wait_ts = 0; + ewq.ep = ep; + ewq.maxevents = maxevents; ewq.timed_out = false; + ewq.wakeups = 0; if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); + ewq.timeout_ts = timespec64_to_ktime(*timeout); to = &ewq.timeout_ts; - *to = timespec64_to_ktime(*timeout); } else if (timeout) { /* * Avoid the unnecessary trip to the wait queue loop, if the @@ -1845,6 +1891,18 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, ewq.timed_out = true; } + /* + * If min_wait is set for this epoll instance, note the min_wait + * time. Ensure the lowest bit is set in ewq.min_wait_ts, that's + * the state bit for whether or not min_wait is enabled. + */ + if (!ewq.timed_out && ep->min_wait_ts) { + ewq.min_wait_ts = ktime_add_us(ktime_get_ns(), + ep->min_wait_ts); + ewq.min_wait_ts |= (u64) 1; + to = &ewq.min_wait_ts; + } + /* * This call is racy: We may or may not see events that are being added * to the ready list under the lock (e.g., in IRQ callbacks). For cases @@ -1913,7 +1971,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * important. */ eavail = ep_events_available(ep); - if (!eavail) { + if (!eavail || ewq.min_wait_ts & 1) { __add_wait_queue_exclusive(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); ep_schedule(ep, &ewq, to, slack); From patchwork Thu Dec 1 18:11:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 28497 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp413350wrr; Thu, 1 Dec 2022 10:15:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf4qV4z5614ydUVtWJGD1Vm99pNexeYbcVYP+jYTthGbQMVr6ahnE3tJYWABU2r9+xMFaXod X-Received: by 2002:a17:90b:1095:b0:219:237c:7986 with SMTP id gj21-20020a17090b109500b00219237c7986mr26491493pjb.220.1669918543841; Thu, 01 Dec 2022 10:15:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669918543; cv=none; d=google.com; s=arc-20160816; b=seCD6jX2lnP6iR+bTO57uAmxyS7e1mZeYVSkiwNFkCWUfFRLG3UM+EsO8NvXiFFqsc hMV0QoT53ZmXLb23n3/6VwfzY5SJqVbxf3LwIY0kHL03Qj+d3s7+bikwTGF9nWQhwesj 3DZDX/AuDjmYSmU91cQ2cDedtyzmmuoU+K8k2pG3GmDSUM2CR9FUh7tHCrlCsqGqLr/G 2WNvXSou+wZ4M4ut0sRZUFDWLyhlv2enDtltLGvhQM5AGh7N0VWqT/H9b3UpceI8sTUT eIDoNQPWgT1NslOBfx9zlAsVAEEARWsWHyVJfG2PUXkNEYvoCv7A/sT4kWyiiHx6N1Q2 62Tw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=hAGk4MFjlfW7Mob6ujVIOlyczZbXb2RyPAqfUdaDlEo=; b=yhel/FWNWZQo90P05oUPaijZxUVS3HdPJ0BU6ehslN8XxtqT8XNkLkMxHuZgNMCXhA s0tkRaGEt0QQxIofxVKI/DjdoqIEvaNnkZ4Uw3QuOljmt4Y0rFn78ERS/F08B0Hc1Wpd pDI3U65lHUm9DfJVvTjonaX2Okk+lZNRDFk3e7vHk0KFA+DZ5I97iE38tGQmxgmc2BqL /n/0pVtqvGJGhid699jBH0V+E1BC0bIGv0RGoaMZxfS12mltF3h93uGdYKgkU/JagjP5 ceA3OyFJRYsejycb5vcNwxUUz2hWzujHAfFxOuKFqkRjvdU55KoDk1dqDwa2+lKzR47H 8DnQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=Gnqziq8x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k29-20020a634b5d000000b004786a8301c3si4169675pgl.586.2022.12.01.10.15.28; Thu, 01 Dec 2022 10:15:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=Gnqziq8x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231179AbiLASMi (ORCPT + 99 others); Thu, 1 Dec 2022 13:12:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230456AbiLASMX (ORCPT ); Thu, 1 Dec 2022 13:12:23 -0500 Received: from mail-io1-xd2b.google.com (mail-io1-xd2b.google.com [IPv6:2607:f8b0:4864:20::d2b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7364B8474 for ; Thu, 1 Dec 2022 10:12:11 -0800 (PST) Received: by mail-io1-xd2b.google.com with SMTP id h184so1440208iof.10 for ; Thu, 01 Dec 2022 10:12:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hAGk4MFjlfW7Mob6ujVIOlyczZbXb2RyPAqfUdaDlEo=; b=Gnqziq8xRmWSBCtI5LbkgYdL2sMBi4ijnnNDCjyb1I6NyYRb4wWQRfi1KeWDcV6woX 2hBG2JA6tPkR84DL4X+rhLHSfSacd65MbGAg32TAPk9YlGxw9kcZgSYXFGNGCC0L5YgY +AuvkWHv/JFbbonRZzYiDMVHxoX9MAkrOh3UCZuIcqxQAktZruc/xq+PU1feO+g8bBS7 DfS7IYpIU+ZQSkmZeoRLSQwzB/++gpslbtWACrtpE7SHmjRbQrhtK/+tRdg7UXMzqpQw Uh8ymHGPUCn+JbALqvUi8JNEliEEGmRAUlCbzIKJhVyPNtyMey7JusEhVCtcfubPIMbD /pfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hAGk4MFjlfW7Mob6ujVIOlyczZbXb2RyPAqfUdaDlEo=; b=zgj1wCh4aHoa5oIhpeom0aGcnIx9NEyemYRp/oxLBKwCcBqozrywT4XZai4zGkqA+j 3GH1XpmLX8lG0p6poZd19Say2qvNYe3MYJbjn08c+YeWd7oJKW4NLOWufbHPcHbXbvrv uQU2WulNXdV/qpqeB33bf2eFMQ1bxBhF2qOPrWStBkcq9daXZYifT0VWvrOYF0zdZS15 +dZztPjsG0oZujtEQUW8PrMcYqSdxTEi9nK1BWSDe+pl2EeTfm3tBF/Rier8UlhFN7ps w8koqCvoE6QXdO/Cn7o1WSDNOP9UOZnSveR+MU/roMmlP2TRJgwPKRjFJivWOknHkLMK eJ0w== X-Gm-Message-State: ANoB5pmllp+IFcMq2c+M9oIijzJrqUb9rlaOvyLIKZoQh8vBIBfngn9A oTIjxAz6cdQ3MsAgf0py+ozSn9jjbfjAQ6QP X-Received: by 2002:a05:6638:4709:b0:389:e195:e8fb with SMTP id cs9-20020a056638470900b00389e195e8fbmr10218197jab.254.1669918330191; Thu, 01 Dec 2022 10:12:10 -0800 (PST) Received: from m1max.localdomain ([207.135.234.126]) by smtp.gmail.com with ESMTPSA id y21-20020a027315000000b00374fe4f0bc3sm1842028jab.158.2022.12.01.10.12.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Dec 2022 10:12:09 -0800 (PST) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: soheil@google.com, willemdebruijn.kernel@gmail.com, stefanha@redhat.com, Jens Axboe Subject: [PATCH 7/7] eventpoll: add method for configuring minimum wait on epoll context Date: Thu, 1 Dec 2022 11:11:56 -0700 Message-Id: <20221201181156.848373-8-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221201181156.848373-1-axboe@kernel.dk> References: <20221201181156.848373-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1751036506882992544?= X-GMAIL-MSGID: =?utf-8?q?1751036506882992544?= Add support for EPOLL_CTL_MIN_WAIT, which can be used to define a minimum reap time for an epoll context. Basic test case: struct d { int p1, p2; }; static void *fn(void *data) { struct d *d = data; char b = 0x89; /* Generate 2 events 20 msec apart */ usleep(10000); write(d->p1, &b, sizeof(b)); usleep(10000); write(d->p2, &b, sizeof(b)); return NULL; } int main(int argc, char *argv[]) { struct epoll_event ev, events[2]; pthread_t thread; int p1[2], p2[2]; struct d d; int efd, ret; efd = epoll_create1(0); if (efd < 0) { perror("epoll_create"); return 1; } if (pipe(p1) < 0) { perror("pipe"); return 1; } if (pipe(p2) < 0) { perror("pipe"); return 1; } ev.events = EPOLLIN; ev.data.fd = p1[0]; if (epoll_ctl(efd, EPOLL_CTL_ADD, p1[0], &ev) < 0) { perror("epoll add"); return 1; } ev.events = EPOLLIN; ev.data.fd = p2[0]; if (epoll_ctl(efd, EPOLL_CTL_ADD, p2[0], &ev) < 0) { perror("epoll add"); return 1; } /* always wait 200 msec for events */ ev.data.u64 = 200000; if (epoll_ctl(efd, EPOLL_CTL_MIN_WAIT, -1, &ev) < 0) { perror("epoll add set timeout"); return 1; } d.p1 = p1[1]; d.p2 = p2[1]; pthread_create(&thread, NULL, fn, &d); /* expect to get 2 events here rather than just 1 */ ret = epoll_wait(efd, events, 2, -1); printf("epoll_wait=%d\n", ret); return 0; } If EPOLL_CTL_MIN_WAIT is used with a timeout of 0, it is a no-op, and acts the same as if it wasn't called to begin with. Only a non-zero usec delay value will result in a wait time being applied for reaping events. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 13 ++++++++++++- include/linux/eventpoll.h | 2 +- include/uapi/linux/eventpoll.h | 1 + 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index daa9885d9c2b..ec7ffce8265a 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -2183,6 +2183,17 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, */ ep = f.file->private_data; + /* + * Handle EPOLL_CTL_MIN_WAIT upfront as we don't need to care about + * the fd being passed in. + */ + if (op == EPOLL_CTL_MIN_WAIT) { + /* return old value */ + error = ep->min_wait_ts; + ep->min_wait_ts = epds->data; + goto error_fput; + } + /* Get the "struct file *" for the target file */ tf = fdget(fd); if (!tf.file) @@ -2315,7 +2326,7 @@ SYSCALL_DEFINE4(epoll_ctl, int, epfd, int, op, int, fd, { struct epoll_event epds; - if (ep_op_has_event(op) && + if ((ep_op_has_event(op) || op == EPOLL_CTL_MIN_WAIT) && copy_from_user(&epds, event, sizeof(struct epoll_event))) return -EFAULT; diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h index 3337745d81bd..cbef635cb7e4 100644 --- a/include/linux/eventpoll.h +++ b/include/linux/eventpoll.h @@ -59,7 +59,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, /* Tells if the epoll_ctl(2) operation needs an event copy from userspace */ static inline int ep_op_has_event(int op) { - return op != EPOLL_CTL_DEL; + return op != EPOLL_CTL_DEL && op != EPOLL_CTL_MIN_WAIT; } #else diff --git a/include/uapi/linux/eventpoll.h b/include/uapi/linux/eventpoll.h index 8a3432d0f0dc..81ecb1ca36e0 100644 --- a/include/uapi/linux/eventpoll.h +++ b/include/uapi/linux/eventpoll.h @@ -26,6 +26,7 @@ #define EPOLL_CTL_ADD 1 #define EPOLL_CTL_DEL 2 #define EPOLL_CTL_MOD 3 +#define EPOLL_CTL_MIN_WAIT 4 /* Epoll event masks */ #define EPOLLIN (__force __poll_t)0x00000001