From patchwork Sun Oct 30 22:01:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1981798wru; Sun, 30 Oct 2022 15:03:54 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6+q31L5m9beDIO6WgR0U+EB9NpnjI+3hnve467rRnvjhFp7hIeJU4xVhi+UKljpw33AeNl X-Received: by 2002:a63:8949:0:b0:46f:57b1:35bd with SMTP id v70-20020a638949000000b0046f57b135bdmr9755030pgd.623.1667167433669; Sun, 30 Oct 2022 15:03:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667167433; cv=none; d=google.com; s=arc-20160816; b=YXfyHmrzNP/Xnr9TVylKjIx2a2+vF3WrDU/dBXVoLCaHXFU6Me5a6dc1NGO5lM8ocZ FIAJQVdsIhxndDifPsk0r/kCvB2aKVM5Yr1JAZB/yENzdvaWH062zniSwENjSvvmz0GN gv9AqvxuwOvvFJT8RIIo5B7Hxna22IabSwI19zPzPUpWA6nHogo1qYbLKSTL05QVwIDH FBTNJ7Z+OG+C0C3p06Pk6sdi69rcI7pS8rXH3VteH30wJ4qG9ht+jQkrDYuKJsdvYu3b yD5xsXmy5Xl9e/ZpYCmufTQHjCA2GX/e8zy8iuJuXlUpbEuRSGgm5uMLV1IDXHnLc+sQ RncQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8PHeEQvI03GO1u4arOs5eKGxrrWj+QcJXenB1+d+Ev4=; b=WX343VwYGP0WAosH8AjOt0dL7XFZ+G75mN7c8++xP67422HiEc5fXeNydP3Z+TKIsU cvsA1nfII//LSwM+ECfbaH/NuQDY3j4PmG/aAlnA+KS2+WrnOYXOdbKiXoUr57c84CzK mn5ooVweWBvsETDfNFNiYZdXvrjPDyxgnv6gWrQ/dfuVhelZbiN1xNf2lpRjhcFE/WWW NV4aW7o0SfhIiXn9Rv9t6zflq361363FBsnYbhSVHsGRIIW2P0aeIX7o0C8KJgv/6o3d 0KAexrH5vAZYvDuvSh0Vm1/Ti1Rq4nEa0YDAtE/Sd9YMmNrYXKOX8T/JiScxc1Q7x+Vi XHGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=6l9nZSzx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eb19-20020a056a004c9300b00565c028c4edsi5805038pfb.355.2022.10.30.15.03.41; Sun, 30 Oct 2022 15:03:53 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=6l9nZSzx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229944AbiJ3WCS (ORCPT + 99 others); Sun, 30 Oct 2022 18:02:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229930AbiJ3WCM (ORCPT ); Sun, 30 Oct 2022 18:02:12 -0400 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95E32A1BE for ; Sun, 30 Oct 2022 15:02:09 -0700 (PDT) Received: by mail-pj1-x1033.google.com with SMTP id o7so5703363pjj.1 for ; Sun, 30 Oct 2022 15:02:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8PHeEQvI03GO1u4arOs5eKGxrrWj+QcJXenB1+d+Ev4=; b=6l9nZSzxsOlgev3XI2BPsdlCYcxkwe86MfpEVfJczMSX1+0I/rH8AlHqSqBvvyUFD+ FLOo+9r+o+aWu5kEugJOUFPwjCw4I54idiEjMpMSPIltB8jPPCqhqN8P1bXH5sssnvQA qkfQNjn4W48osJfV2juo6XvIlw9Z/Hure5Ix8EsJEn5JkL+e3nOK6wqxUNn8vX9vqmPp rG+PyTTCQLqQ6k6pwkXoxeA+qz4Tgabrzzbj+3ooWXmaRO+62+Qi0xNSgA3F6qi5GTnj gMK/ksFvl1WR/UPEciiD58drbBPYSQwTv7nUZ+wLrJehGnY27OoUj1IHxXCBypqKpjKr qU9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8PHeEQvI03GO1u4arOs5eKGxrrWj+QcJXenB1+d+Ev4=; b=kBXWnhdTONVxRWkrvozqzBWM15CSlYWET7nerFyxdW1toYQ/A9pWjIg4s4JFtBHY8s rjcOA95PwZBSRNzJAE5C54UruTvPkfyYmJ8VTlNyUDGCAo0r7DrQFzVGHi8bJuBVLV/4 z+pslT5N1pgvY+a6FG/VVQsiYG0qOo73/NbJtxAd5f3a81kLLXBndgGUXrUo7BHFW+Z8 he2j7p4Q20MH3NzZMRTxLs/tPTUrSizqA1aIVKa7WAGZ9wuI95YA5CNrfFVepycTvtVB OoWZ1HFWa50qZRddLiQHov153es6y+ZHrGe7UzTgkoI3nMCMJIB5eGrthbtrw+2LPhbb xGug== X-Gm-Message-State: ACrzQf09nVe6bOjVX7WY1hIKbu4pQIN/+mD17TaF1FVhYcPjniaX3Bax nbqMk/gf6Q/HU2/yLstJIqaBto8zpgoxs12F X-Received: by 2002:a17:903:2452:b0:186:99e0:672d with SMTP id l18-20020a170903245200b0018699e0672dmr11199251pls.95.1667167328768; Sun, 30 Oct 2022 15:02:08 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id y3-20020aa79e03000000b0056d73ef41fdsm562852pfq.75.2022.10.30.15.02.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 15:02:08 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/6] eventpoll: cleanup branches around sleeping for events Date: Sun, 30 Oct 2022 16:01:58 -0600 Message-Id: <20221030220203.31210-2-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221030220203.31210-1-axboe@kernel.dk> References: <20221030220203.31210-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748151758485333356?= X-GMAIL-MSGID: =?utf-8?q?1748151758485333356?= Rather than have two separate branches here, collapse them into a single one instead. No functional changes here, just a cleanup in preparation for changes in this area. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 52954d4637b5..3061bdde6cba 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1869,14 +1869,15 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * important. */ eavail = ep_events_available(ep); - if (!eavail) + if (!eavail) { __add_wait_queue_exclusive(&ep->wq, &wait); - - write_unlock_irq(&ep->lock); - - if (!eavail) + write_unlock_irq(&ep->lock); timed_out = !schedule_hrtimeout_range(to, slack, HRTIMER_MODE_ABS); + } else { + write_unlock_irq(&ep->lock); + } + __set_current_state(TASK_RUNNING); /* From patchwork Sun Oct 30 22:01:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1981836wru; Sun, 30 Oct 2022 15:03:59 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6STB0WvwAg2vYNgWSDgbN3UaTQqSEplvcgXY53kiumqZzUQ4aiJpav7FyKg7Z8ly7ZsI9U X-Received: by 2002:a63:8643:0:b0:462:9b02:a0c1 with SMTP id x64-20020a638643000000b004629b02a0c1mr9909905pgd.536.1667167439562; Sun, 30 Oct 2022 15:03:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667167439; cv=none; d=google.com; s=arc-20160816; b=cYMMCkarSpo4YagrfH0L3s9hbNOewLSD3QH3g2ipAuHO2ghQspI8p8h64zn1lAjXPI sMMre5nUU0zyhUt325+qCpOt3RcYxcfxPWHt1b4V5+pR5bepSfLZxUAQg4inshmtItTI x+TxEs1zcwIKBWuMRhfu+ZrktmOHzwTo4wnP+vwf8fwvqq6xf6XGibVUKQN3u4euKqz3 KGNWvhrqBMyTX/OnOiTkYGo5+7M6waCw59sGXFUK0MBbQVs5QZyJDLe7354JbqpPPb4V WZAYbbryOx1K9gUg0bjG3DRkED08s1lpHzPqcBB5Xf0gp9hMxa2qoNuiTjWgt+LE2p1n dEyA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TCMSGu8wCAuxi+YEaeqt2Nm6ljarPNoUCebXSLfB1Fw=; b=uvcKrFWF7cAzGogJHiHGqevnoLoUIMLApkPmaebhNKvMaIB+gtFxXEjw87Zio1X7Jk ub+1OGFQKSn1CyaV5hfI6mZKNd/3xvYxIUkPY93uDhgfASWgia6xYdgcZ91rvCFPKXog 6553fJH9416A/F4piqxUL9kQqrsHsYnHZ0i8+6RhKWj2X1bt+audi5EJ+d1rty+GAPYi sikuSR07f1+khe1Q5UeRW0MdDhHnBk1GUshffhYc4qiz3FbbHASkXlnbb/czGFgK0SHa kE8K7NEAEpkBFI5w4GGwL84l8CiYOyffqDGU2FYOtgt1XR7p9XrZaEeXeb92PKtsc0HR tNJA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=H1OQbWeL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v70-20020a638949000000b0046ec3bd90bbsi6077622pgd.60.2022.10.30.15.03.41; Sun, 30 Oct 2022 15:03:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=H1OQbWeL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229996AbiJ3WCe (ORCPT + 99 others); Sun, 30 Oct 2022 18:02:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229954AbiJ3WCQ (ORCPT ); Sun, 30 Oct 2022 18:02:16 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30E1FAE60 for ; Sun, 30 Oct 2022 15:02:10 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id f23so9228740plr.6 for ; Sun, 30 Oct 2022 15:02:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TCMSGu8wCAuxi+YEaeqt2Nm6ljarPNoUCebXSLfB1Fw=; b=H1OQbWeLIKTClEI0Y18BCREEaUPyB4u9aov6gnRCwP3FMlWoDhoOyrQj9dTUw8f7Rl Lnc+rX7GLGtNIG7+mOPq5mJ218zFKXipmPROig7C1ec18KZ542PUt0JccxGsjh5HO1a0 /z8kUmiSoWowwx6pl7CmnUz6U/WQmRT/NWSJGgn5Kjr93Rcz9iCWvN5ozoSLbcjCLLaW vQSLjW+N399VOHIFg9wUxuq3+NKkDgXzX84jiBotca1ufvXrHs73a7xFaaZLJbSrPgpF kI9+im4ilsx5ns6Kwr6KHXhyinzcJ9zfi1ofygZp48zSk618zFDCdZssNjQg3mnekR5P YICw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TCMSGu8wCAuxi+YEaeqt2Nm6ljarPNoUCebXSLfB1Fw=; b=ZA0ePgo1TNywxC70BsmKi1e1pdahfCqQ1oTLebS2BtlaegboAMN0ZA2BxVxBz7yMeD oOYq9N2BXme3XJ+EEAIO+7KTxFAXyP0JVn2/vcaCKQtyRofikhbv4jGBY2sb8jS+KLaa Tn8h8yDN0XHDqlQQ2x7caAjdpEtcJGwk+ufJRLuRe9t3DxG2zAwI70JisY/c/AEB2ZP3 zjPpAsQkiThCNzKQDS1XexiShSjObJWp3cYE1ZsPEqbbNrH0D+X6oCVLzoBgBLxsgMdU 6ahRKVMucvxgaSGEQ+rA+a7VQAYOEXi8ZB4zfwFWXw0b1i149a38inqLRR1ckgAN4XN3 Y46g== X-Gm-Message-State: ACrzQf15v70fv0u+TjepNY7hi3ob/AyX57YZz81o8vJsEqyeaasAQz1q A+fATDw+zf9B9J4idCsuqjxi9eswrb6NfCAs X-Received: by 2002:a17:90a:ca87:b0:212:d2bd:82f5 with SMTP id y7-20020a17090aca8700b00212d2bd82f5mr11677337pjt.203.1667167329708; Sun, 30 Oct 2022 15:02:09 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id y3-20020aa79e03000000b0056d73ef41fdsm562852pfq.75.2022.10.30.15.02.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 15:02:09 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/6] eventpoll: don't pass in 'timed_out' to ep_busy_loop() Date: Sun, 30 Oct 2022 16:01:59 -0600 Message-Id: <20221030220203.31210-3-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221030220203.31210-1-axboe@kernel.dk> References: <20221030220203.31210-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748151765014090295?= X-GMAIL-MSGID: =?utf-8?q?1748151765014090295?= It's known to be 'false' from the one call site we have, as we break out of the loop if it's not. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3061bdde6cba..64d7331353dd 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -396,12 +396,12 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) * * we must do our busy polling with irqs enabled */ -static bool ep_busy_loop(struct eventpoll *ep, int nonblock) +static bool ep_busy_loop(struct eventpoll *ep) { unsigned int napi_id = READ_ONCE(ep->napi_id); if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { - napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, + napi_busy_loop(napi_id, ep_busy_loop_end, ep, false, BUSY_POLL_BUDGET); if (ep_events_available(ep)) return true; @@ -453,7 +453,7 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) #else -static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) +static inline bool ep_busy_loop(struct eventpoll *ep) { return false; } @@ -1826,7 +1826,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timed_out) return 0; - eavail = ep_busy_loop(ep, timed_out); + eavail = ep_busy_loop(ep); if (eavail) continue; From patchwork Sun Oct 30 22:02:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13072 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1981966wru; Sun, 30 Oct 2022 15:04:24 -0700 (PDT) X-Google-Smtp-Source: AMsMyM52K0Qet+UTLh/hx7M26ymyOFBlkDG9EO4FpSiBLjFQfNq0yUg0alwk8T0t2yMD4QIpNwn6 X-Received: by 2002:a17:902:d509:b0:187:fd9:d8e9 with SMTP id b9-20020a170902d50900b001870fd9d8e9mr7286561plg.140.1667167464364; Sun, 30 Oct 2022 15:04:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667167464; cv=none; d=google.com; s=arc-20160816; b=uOfbGAfhpaGB5hpF19fcauJLyOuwtsqNvRzDoWAZrE2cCxltQl/bgQSyPlOBNndwWl IAnwm2PneEdjWx9w9RzscEK2ZHbIffVyohQnUpZFvL23Gmkw9P4aeFscfJAo36jsDRei sZ9jaaqyH6enPtHu8XXXZ3OkuWWMzcny2jq0d3Yo2Q/l2DbYtvHBIJKzOZaqUyIh41Ex rL1ZQp6RUZ8WlA2odQGe+Txph+7Xw+6u6zQNe3lvLFYvw54SnKWLI1iKgOXNbf9iw/o1 J3/nrfcMykbeNtiQw5i97xiGdpAv98XubusT4SF68g9DarqRIB6wjcX7RsRwRa8Yv/wR puyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=d8OpHlPRST08skeh7BHSZbFN09k6AB6QzxLidf3Pxy0=; b=kt9W/nuI50FCje2wD29jH3JhieIdELL52xoMYyr3yyP/4ENiiV4Zn278lZG/+wvAk7 1DAuTiFURPs+bR22CYQPq0HYDTMToXVu7IZdN4JkjkL5AsqZHxDwicIe7oKq/1nbV+qV XLS9C94W8dOua78vh6shLxKVVtqM1MIo0aslJGWew4O2u2TS7lF3P8Wr19Le4AQNxGd7 DV2fnwo02SRrB1iZUCLimJUeHWAEIen/fW+yvxxHuEwEaMHwXFdXwTr4NUHZ0GRRw+J/ hYb5HXF9qUT6+XpuwMWUBcj2Hr3Dxw3I5zJTGrJQhXEidU52Eq/+Y+oYQuZUS6ezyDgc pikQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=FtnVuw2d; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k1-20020a63ff01000000b00456b507fb2asi7027713pgi.446.2022.10.30.15.04.08; Sun, 30 Oct 2022 15:04:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=FtnVuw2d; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229902AbiJ3WCh (ORCPT + 99 others); Sun, 30 Oct 2022 18:02:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229956AbiJ3WCQ (ORCPT ); Sun, 30 Oct 2022 18:02:16 -0400 Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31954BCB0 for ; Sun, 30 Oct 2022 15:02:11 -0700 (PDT) Received: by mail-pg1-x52c.google.com with SMTP id f9so9225023pgj.2 for ; Sun, 30 Oct 2022 15:02:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d8OpHlPRST08skeh7BHSZbFN09k6AB6QzxLidf3Pxy0=; b=FtnVuw2djVhLJ39B4Iu0YH3QmQxMkIGxXHKnGZA7K1Sw4fYMnyIG96gEfOp4ciSi3F cp2w+v7oMMq0Yi2VBZSN6Hrdemo+vuHjooi0IikAo7UQPpKQ88+6759GdWUK+E0kufEv JmuhNI176JjjlMg/MU4+jDvC333l9Ozzl1KVuDENgYG8GAaobnWkN8D4i3P+gSnLe3wp DwP0mgFKh8mD0/6QaVxqR1lyCrEHmDnbYsdnvwA22dLE7MQLkOCq8F03DuPRZPcY7+uP bMXRGPp77FPPOBjK+xLWlYkO6HIx11jiV+KgVJpNw+o0kooBQ3k8mQHdrQSzFG9blCoz kwRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d8OpHlPRST08skeh7BHSZbFN09k6AB6QzxLidf3Pxy0=; b=6OvVb2cJuRK83iKxoZPplRDtMxjQGH5OYBG4CuiQ3fjxgABJmC2LVt87ZW2Uo3t9gD gS5SkhVMHzU/sh7BIm2Ejr6uw+GNdBV+ciiA8Mjq8puB1tbFPwOVxG95An0COEoIWi0b aSEpVyK2blu0io1EpqiP5VuawS8bDyYMXfnRQwoPH2Of6toZV82zKZ5ovjxQ/brgI8aB 6DGbueZp0z5brDZi+KXVsmcK4OezrUqdavmXOk4kMz6nhurvSN7+jZHln07jNmq87fYc gvtpMNLuwrLJ61c1p1MTE4f4QXRvqHFsAYHurnmCGh+iA/SEMKQ8LQnfl1tZwHQK8Zw5 guMw== X-Gm-Message-State: ACrzQf2VAtNIwhNHpGFfQFLKoLf0V1eVjrR6BoUb4if3vG8z2e+Q1T8G 5UOI6+nqRjFL0rz+CWeh2sFVwlBRUa6Swer2 X-Received: by 2002:a05:6a00:1348:b0:56b:f5c0:1d9d with SMTP id k8-20020a056a00134800b0056bf5c01d9dmr10963195pfu.45.1667167330589; Sun, 30 Oct 2022 15:02:10 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id y3-20020aa79e03000000b0056d73ef41fdsm562852pfq.75.2022.10.30.15.02.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 15:02:10 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/6] eventpoll: split out wait handling Date: Sun, 30 Oct 2022 16:02:00 -0600 Message-Id: <20221030220203.31210-4-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221030220203.31210-1-axboe@kernel.dk> References: <20221030220203.31210-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748151791171661078?= X-GMAIL-MSGID: =?utf-8?q?1748151791171661078?= In preparation for making changes to how wakeups and sleeps are done, move the timeout scheduling into a helper and manage it rather than rely on schedule_hrtimeout_range(). Signed-off-by: Jens Axboe --- fs/eventpoll.c | 68 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 55 insertions(+), 13 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 64d7331353dd..888f565d0c5f 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1762,6 +1762,47 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, return ret; } +struct epoll_wq { + wait_queue_entry_t wait; + struct hrtimer timer; + bool timed_out; +}; + +static enum hrtimer_restart ep_timer(struct hrtimer *timer) +{ + struct epoll_wq *ewq = container_of(timer, struct epoll_wq, timer); + struct task_struct *task = ewq->wait.private; + + ewq->timed_out = true; + wake_up_process(task); + return HRTIMER_NORESTART; +} + +static void ep_schedule(struct eventpoll *ep, struct epoll_wq *ewq, ktime_t *to, + u64 slack) +{ + if (ewq->timed_out) + return; + if (to && *to == 0) { + ewq->timed_out = true; + return; + } + if (!to) { + schedule(); + return; + } + + hrtimer_init_on_stack(&ewq->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS); + ewq->timer.function = ep_timer; + hrtimer_set_expires_range_ns(&ewq->timer, *to, slack); + hrtimer_start_expires(&ewq->timer, HRTIMER_MODE_ABS); + + schedule(); + + hrtimer_cancel(&ewq->timer); + destroy_hrtimer_on_stack(&ewq->timer); +} + /** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer. @@ -1782,13 +1823,15 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, struct timespec64 *timeout) { - int res, eavail, timed_out = 0; + int res, eavail; u64 slack = 0; - wait_queue_entry_t wait; ktime_t expires, *to = NULL; + struct epoll_wq ewq; lockdep_assert_irqs_enabled(); + ewq.timed_out = false; + if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); to = &expires; @@ -1798,7 +1841,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Avoid the unnecessary trip to the wait queue loop, if the * caller specified a non blocking operation. */ - timed_out = 1; + ewq.timed_out = true; } /* @@ -1823,7 +1866,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, return res; } - if (timed_out) + if (ewq.timed_out) return 0; eavail = ep_busy_loop(ep); @@ -1850,8 +1893,8 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * performance issue if a process is killed, causing all of its * threads to wake up without being removed normally. */ - init_wait(&wait); - wait.func = ep_autoremove_wake_function; + init_wait(&ewq.wait); + ewq.wait.func = ep_autoremove_wake_function; write_lock_irq(&ep->lock); /* @@ -1870,10 +1913,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, */ eavail = ep_events_available(ep); if (!eavail) { - __add_wait_queue_exclusive(&ep->wq, &wait); + __add_wait_queue_exclusive(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); - timed_out = !schedule_hrtimeout_range(to, slack, - HRTIMER_MODE_ABS); + ep_schedule(ep, &ewq, to, slack); } else { write_unlock_irq(&ep->lock); } @@ -1887,7 +1929,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, */ eavail = 1; - if (!list_empty_careful(&wait.entry)) { + if (!list_empty_careful(&ewq.wait.entry)) { write_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, @@ -1896,9 +1938,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Thus, when wait.entry is empty, it needs to harvest * events. */ - if (timed_out) - eavail = list_empty(&wait.entry); - __remove_wait_queue(&ep->wq, &wait); + if (ewq.timed_out) + eavail = list_empty(&ewq.wait.entry); + __remove_wait_queue(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); } } From patchwork Sun Oct 30 22:02:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13073 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1982225wru; Sun, 30 Oct 2022 15:05:10 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6LUDtgaBBsmPeMIQ+wHHTDqINqqIJIctJIaZ8c/T4hc80hRpXxXjbnz6TGz8TszzWx8eXd X-Received: by 2002:a17:90a:df04:b0:213:1cf1:435d with SMTP id gp4-20020a17090adf0400b002131cf1435dmr11808081pjb.210.1667167509991; Sun, 30 Oct 2022 15:05:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667167509; cv=none; d=google.com; s=arc-20160816; b=sBWBnVerL/uT3vPbem+7kZadW3rIeMHu6xY60kyklrIXo48zRu9/FQXXYkYI3BwmVs amp+DMSybr9vdG4ANav1T9d/THMAcJm1dmgDS0GREf0Xksa6QaVr9eJiIse35CZ+6yZH 6hCgY2qUm3dT3OujyjOAwgqzK8zk/66eR7TGmb48qKGlu+H+oypjI47uplncuhRxoGhC eoCrpvR8VetLaQmjlIa3HHGgjrYechNCKsvw9sSy+MT5dQyWZOJ7A3u9C45viygMt6Vu BFWS8xtChUC50d0UuAs6PsN+WeiE1/d20VPtjpC2fLks9BGzrJTCdmlOhPziA89ZIMRt OsxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+7XieXpcJrzhapQHtWCNoW0rQy5yiKhkq34SDOyw0xw=; b=kLXzBgELPzYIkMnfG06LJtZsI7FVJzJ/27zA4O/luKaalZ8/noNbRoscJRKCxRRr8m JakLF4PpUGD6uzPDBTr4WKmSC+ZLaL5QMreEhFipEu8eLARzKoS88ybqs+qiMxKvxN3M yB/+E3ViqPf/D82MivkybuuhFApDBLSOe4LDG+RTZNSDyE6m4Aj0wLYv6YUXk03SGCnG F5mYlldYDX8Ls7W/8rwU4L3Hf8/S2adiwHAPZ1jJHY/vQXl6InVpRpmkDjYrMDYW1VW3 zJffskdqIJz4UOpAfToVeTtumXdSJcc2KnZfisTTpqfvzhsQjX1y01ooxu9946Vmg5/S DI9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=Z2rp9mxa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id oa6-20020a17090b1bc600b002136a8424e9si7328781pjb.1.2022.10.30.15.04.57; Sun, 30 Oct 2022 15:05:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=Z2rp9mxa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229895AbiJ3WCl (ORCPT + 99 others); Sun, 30 Oct 2022 18:02:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229967AbiJ3WCQ (ORCPT ); Sun, 30 Oct 2022 18:02:16 -0400 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98297BC31 for ; Sun, 30 Oct 2022 15:02:12 -0700 (PDT) Received: by mail-pj1-x1031.google.com with SMTP id l6so8933325pjj.0 for ; Sun, 30 Oct 2022 15:02:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+7XieXpcJrzhapQHtWCNoW0rQy5yiKhkq34SDOyw0xw=; b=Z2rp9mxaTzm5Kh46RSwGk7G6rZY/KQhbeDA2IAjzJGp/KbgOvSg7hhvjOqyULqd566 //TWc8mDkHbQ+DYXM0UrXa1OFHB+kcFlfhSr5EpBPAZ4BenGOAV8UQytYF1B2Zu15GZW 5wSN+m6knpruLvqwjn/CbyS0LcSLGDOxBx2XJBPBYD36NcCOwgnf8PE0K83/qoHEWITB CoB+1QFBWq2K9ejFNlUmTFH4z4jR/GN1EKDtyv0QkZWRvmvLrhnWvunYux57608rO2K1 NyfP1zPcMuWKUItcpD6jYYjGRh9hLNQQpDbfHBw2vNSNUUlLYIpcgwApfo0fwBwOZ87a d44A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+7XieXpcJrzhapQHtWCNoW0rQy5yiKhkq34SDOyw0xw=; b=T8Rad4xsqogYtRXMSTdVoXUXX8a4Qmpn5tBZIdfq8Zoc2IYE/L1491eMLtwNP++PsF UEBMZ2lpShdekq8joZCjxPdWQpfyKC8TdsjZkVOMrXcctGkF8F9gwjd2PHQnl+4hqZ9C mMVQ/xlcA9YJQFKQTEzPBTPGUtLZbjYCQ36LI1e1sdnxsE18wvkE+6Ya/LvuDy92j2mG ttKOdKvZ3Vmg3DqRSjAFUz2rbKNPMWYalrlOrrvrq3OcadQFV8q+Z8VmsvQiIiF8N1ZP rqsd4OR9HBExQ+92pIqUTDUbeTpPLdwBXKGZHiQEqVKnS+nGWV1ezv2TpIBbIJvSLFP8 tEOQ== X-Gm-Message-State: ACrzQf0xM0fapMIKn/zXGMKaSxUUhKHZUlNiF1mM9BWE11LFy8oY3W9G cFoKh/bYE5puc0ZuHrsEmi1PoQGFjN15GAqT X-Received: by 2002:a17:903:258b:b0:186:8bb2:de32 with SMTP id jb11-20020a170903258b00b001868bb2de32mr11380804plb.63.1667167331690; Sun, 30 Oct 2022 15:02:11 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id y3-20020aa79e03000000b0056d73ef41fdsm562852pfq.75.2022.10.30.15.02.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 15:02:11 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/6] eventpoll: move expires to epoll_wq Date: Sun, 30 Oct 2022 16:02:01 -0600 Message-Id: <20221030220203.31210-5-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221030220203.31210-1-axboe@kernel.dk> References: <20221030220203.31210-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748151838960857397?= X-GMAIL-MSGID: =?utf-8?q?1748151838960857397?= This makes the expiration available to the wakeup handler. No functional changes expected in this patch, purely in preparation for being able to use the timeout on the wakeup side. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 888f565d0c5f..0994f2eb6adc 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1765,6 +1765,7 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, struct epoll_wq { wait_queue_entry_t wait; struct hrtimer timer; + ktime_t timeout_ts; bool timed_out; }; @@ -1825,7 +1826,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, { int res, eavail; u64 slack = 0; - ktime_t expires, *to = NULL; + ktime_t *to = NULL; struct epoll_wq ewq; lockdep_assert_irqs_enabled(); @@ -1834,7 +1835,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); - to = &expires; + to = &ewq.timeout_ts; *to = timespec64_to_ktime(*timeout); } else if (timeout) { /* From patchwork Sun Oct 30 22:02:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13070 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1981928wru; Sun, 30 Oct 2022 15:04:14 -0700 (PDT) X-Google-Smtp-Source: AMsMyM64mavsJm9FfNVC9iKjcdG+8pKyMwsLQrZIwNeUL7/H7JiepTPviHGLrZsIaKL0/DyIu6bW X-Received: by 2002:a63:551b:0:b0:44b:7dcf:92ee with SMTP id j27-20020a63551b000000b0044b7dcf92eemr10070035pgb.610.1667167454584; Sun, 30 Oct 2022 15:04:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667167454; cv=none; d=google.com; s=arc-20160816; b=bxrVuEeal528F0c4xe0GSEIxuCjAcoS9hldckNzr0pIO9fu/RJf6UQ8wuTF5MauEGv 4JOEkwFWTjeCroU6780AdzskMnQVEJS9duATG2pi3bv0+Gum1ahJEPmTtAxgQvreMqmN NIctIjd1B4RO83NlBepYsDKP4pa1UtULWVwE1FcVtEzhb7oKM4elCAVcQkq6l2hjmZSc uIut6K5yqwr74t6ThrZRAgDWDsuNe1/7O9++oXvWcWR5GdmvwcNtnhGuKhm/O7/FnHh6 zqJwEkU2HmOZdEwx5VKCI5Du07/wmYerW7+1ExJd6KUcXl7lySaG33rdCO/r5OmhEH8+ VPtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T2Hzb+xxyKnPzMsKtR+WvmF1ywGYO4tzf59V+gc/0/s=; b=PN0DgAJeQf5LdNA/9NwzVfOtYf4vL9U0gAV8+9kkNMPEWEo1F2/w3n0jELW9hjSyit oIIN9dZCaI1tp2y/eReJSz/dPsjOMnLkWo5QXNVI+ldWbj+8tj05REgau2ZZKeKdfbe4 YwuGGbzDuQQyH4jHiKxnJoafNTlayh4UdNHuoPK4cul09zWQ3EMSM/4ZplQk0dMcLYh/ p5JZ17LHHaLspzZfXz4scfOeW9SpsGmVs4Z/fMpnHssiFRxp43VpkZoYk9a190YcLvnu yCjvptQg2zT9T8C/k0cXxHeidGVHAg35p27O6Kg1o5NBmD3Zb+CCUbOE85ZRneQp3ZGq ztAg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=4SbYyN02; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m8-20020a170902bb8800b00183243c7a2dsi6035226pls.406.2022.10.30.15.03.58; Sun, 30 Oct 2022 15:04:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=4SbYyN02; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229980AbiJ3WCo (ORCPT + 99 others); Sun, 30 Oct 2022 18:02:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229932AbiJ3WCU (ORCPT ); Sun, 30 Oct 2022 18:02:20 -0400 Received: from mail-pg1-x532.google.com (mail-pg1-x532.google.com [IPv6:2607:f8b0:4864:20::532]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5448DB1DE for ; Sun, 30 Oct 2022 15:02:13 -0700 (PDT) Received: by mail-pg1-x532.google.com with SMTP id f193so9252165pgc.0 for ; Sun, 30 Oct 2022 15:02:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T2Hzb+xxyKnPzMsKtR+WvmF1ywGYO4tzf59V+gc/0/s=; b=4SbYyN02zU3+59sahg+YLvVvWkSd64+1XsaqdsPV7wLhPizJiL1ZOSfFWLEXqALNYa 6W6QeSVCcOzhNwakNJ4zPTTE1Kep4cZIA88KTO2RA+MLTQn/knPn/uclsfs5FWpp2r6X MBwpK2jSIc9AuGhnLLcYN95HkF+kptFd0w5wvA/YtE9N5+9YKHf7bcTuPECr48LAKx0C 7ea3GPXX5Pek4eIhJCfGxGLaMzcEtWvp/NBeUf5cfL4o2WKF375Mdt8o3+ozsisYNfV4 LlCBO/tvvMnw7IofiBWylGe7Pp676nxsruegSlfD1fNjwlCGunhC8UcSHHnfzrI6bgv5 Dicw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T2Hzb+xxyKnPzMsKtR+WvmF1ywGYO4tzf59V+gc/0/s=; b=0HMQqce5lNVSMDniMIjgmltkvIFhDsj7pHLN+NcFoR4FWrQ5VTH2DyEFe1o/Q5b3ur pZdHBsumE0IbQfmQWNBYH9AnAuDGYRaLlHh/4Hb3gbbS5uBpJ14Fg8zLuaxjMLWl6Je5 NA6PDtcB617csaVSN6WwB35svFmADxYGnPh0CRXRlQnME7efSOIDqAHLYXkAx6akJPdS 54+sNseVi53T+ngeGnFsYsHwuwR+6UKGxTn22VuggYt6L+dZ5ikWcJwV9hKR5fHFz5rT XIXPYoNpPZs1kmfYtnem8lFmoLe+oup45rCCqj84HO9oT6i7uH1jQEuRCYLeJM0vjNuM yDxQ== X-Gm-Message-State: ACrzQf0rgctVSvTJkJLL+GxN4B/FykwJ7piRX5SQ/q78hF/sxClJjNgX B6TbrUOmihhQsNp9A62UH5u4xLaEiWQRqwJN X-Received: by 2002:a63:f103:0:b0:439:398f:80f8 with SMTP id f3-20020a63f103000000b00439398f80f8mr9698925pgi.494.1667167332660; Sun, 30 Oct 2022 15:02:12 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id y3-20020aa79e03000000b0056d73ef41fdsm562852pfq.75.2022.10.30.15.02.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 15:02:12 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/6] eventpoll: move file checking earlier for epoll_ctl() Date: Sun, 30 Oct 2022 16:02:02 -0600 Message-Id: <20221030220203.31210-6-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221030220203.31210-1-axboe@kernel.dk> References: <20221030220203.31210-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748151780493033950?= X-GMAIL-MSGID: =?utf-8?q?1748151780493033950?= This just cleans up the checking a bit, in preparation for a change that will need access to 'ep' earlier. Signed-off-by: Jens Axboe --- fs/eventpoll.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 0994f2eb6adc..962d897bbfc6 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -2111,6 +2111,20 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, if (!f.file) goto error_return; + /* + * We have to check that the file structure underneath the file + * descriptor the user passed to us _is_ an eventpoll file. + */ + error = -EINVAL; + if (!is_file_epoll(f.file)) + goto error_fput; + + /* + * At this point it is safe to assume that the "private_data" contains + * our own data structure. + */ + ep = f.file->private_data; + /* Get the "struct file *" for the target file */ tf = fdget(fd); if (!tf.file) @@ -2126,12 +2140,10 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, ep_take_care_of_epollwakeup(epds); /* - * We have to check that the file structure underneath the file descriptor - * the user passed to us _is_ an eventpoll file. And also we do not permit - * adding an epoll file descriptor inside itself. + * We do not permit adding an epoll file descriptor inside itself. */ error = -EINVAL; - if (f.file == tf.file || !is_file_epoll(f.file)) + if (f.file == tf.file) goto error_tgt_fput; /* @@ -2147,12 +2159,6 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, goto error_tgt_fput; } - /* - * At this point it is safe to assume that the "private_data" contains - * our own data structure. - */ - ep = f.file->private_data; - /* * When we insert an epoll file descriptor inside another epoll file * descriptor, there is the chance of creating closed loops, which are From patchwork Sun Oct 30 22:02:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1981933wru; Sun, 30 Oct 2022 15:04:15 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6NRmPqSQFuQ3lxpbmTh8UgOKLf8bb3CydlZeRQygbO3b+5NUBiIsQah7ddWqD0TzcX/vYU X-Received: by 2002:a17:902:e810:b0:186:e9ff:4ec2 with SMTP id u16-20020a170902e81000b00186e9ff4ec2mr10767585plg.26.1667167455580; Sun, 30 Oct 2022 15:04:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667167455; cv=none; d=google.com; s=arc-20160816; b=zuencOPJJBejmJp4h7VZ15z82khmh0B+2y2AT2rFuhveeYl1dj843IrAcDA/ML5Hq/ HG3jxVInHgyMcpGAENOXRf7PRMWAivjE282xni6iTl7uxX9ndRINpHI5MWM3gug8Uyt2 z+fWhT8Gx1Et4FXA/wNVSB3Sv3ljS3p5JwAMe3oHdRS+/al0U5Yz+jjlmH8k6rq8aczT IPY6VnFqeM+MLR2JhyeFzWkX45a0SbhvWVzeCXS6R7Mx4UEpbQRpz+VsQEGs1xaaXac5 tEKAayVpy2N4rq9wPKwUNuiSt3SRdAXbz68B70VW/vkBnUPEJzy/9dknc2Rw7gbNl3hp ELNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pwT9F3VGgPNDdfJa3yPd0dcN35etavVFYMg4RY5YL0o=; b=AvTgdV2bMRkUPUYOknniiSRvCxnFOUP3szL5Im5t4Kz9FX21KAUAXIKkEuTgMoirAZ 8enApvkXC4F6zodv3vnWDAd7EJsboWvOwZnmTktXfxDUvZfc2cWbOXWYhczjJ9++00gi O6ntcvCVDgwb3sAxPdpdzqHmWlyln98srowxwOScaJakYBPKOlAaqZdCpyqeY6hwfFRU 0NwNzhUOnirluJDTt/xbuPv9ESyfLJ0PH3Z7agLe7jXqvt4pNhD7ZwDeQ1ITZA36kVJi BDq4C/tR8YfmBtBKiYiUI6FPOvEax7SCdwFIjGHcSSzMhxp/5rKYtl4YqLVU6OU1afau YOTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=VIkEe8kf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u32-20020a634560000000b0045b0a113f6asi6308583pgk.846.2022.10.30.15.04.02; Sun, 30 Oct 2022 15:04:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=VIkEe8kf; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229948AbiJ3WCs (ORCPT + 99 others); Sun, 30 Oct 2022 18:02:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229714AbiJ3WCc (ORCPT ); Sun, 30 Oct 2022 18:02:32 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99D75A1BE for ; Sun, 30 Oct 2022 15:02:14 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id b11so8905989pjp.2 for ; Sun, 30 Oct 2022 15:02:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pwT9F3VGgPNDdfJa3yPd0dcN35etavVFYMg4RY5YL0o=; b=VIkEe8kfBczuf+b12NGhnxMQuafEN/XF99q0aDh4J9MnhwgaMH7hDBHb9s3jvYnyTg ImBJ8HIU6TWGEhH9qVBxjZDvHd8JpOSmpvoktzoCxu2ynpY/0oPVA0ymJRJnbFRRw6KB aMxQWidSacHx7htlYYplGXwVmuHkU7evUOXkUvg0EBLxwtSEviMeCodR2MG/Z1Tkhvs+ 6tEMfVStRqMe3+TVpGewnXQoFSZH9DEzI8Mg3D5IbWx4GX0ZSxrF9W31BhGX3xP82rPK K1NDqcdmIGhhiCrYTSCEXi6SBeV8LkBSZXCMvpBVgddljCOA/ZuTi0g9v4Z+9HqwnvLB p0zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pwT9F3VGgPNDdfJa3yPd0dcN35etavVFYMg4RY5YL0o=; b=eH1U/KEr0AhhwRBLtLxMf/xyHAiBx30BEk+IccqqRwQKdrepUDQFnmrguLpbK/+Uyi Eqd4+LBM9GtqP1XlPazCh9wgBLEyslx2aXHWg/OeZItjVeeNCDKixN0oO3gffbb7tpy8 NHpIcK4n2zrz4CfI09iXbx3zhSg+z9UXtFfNqj1+A2huTkMk/cbQhcCZHDoVItsARrpe DePe5ZjZa45SUM6Td2ly6AI6ChRFiQciNx/fFy6rfzN7smkkjP5OTeYjWCmmJeo6BpBx LeWrcPmhjtaCnGBTzcv9t25HnGXVIgY/sT++PWTp0Rw0r8fLnIlog+2Z97AqECJ2Iw9e 6amw== X-Gm-Message-State: ACrzQf1aqJ9jC55D9RU9T2L4UDWSwjwFZo2uAr2Dy28rv09WX1nH9CRF cm+mbKxmnbv/3mW2nYmVQUVvAyjPsiQgOEaz X-Received: by 2002:a17:902:b614:b0:186:940d:7e98 with SMTP id b20-20020a170902b61400b00186940d7e98mr11208913pls.80.1667167333625; Sun, 30 Oct 2022 15:02:13 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id y3-20020aa79e03000000b0056d73ef41fdsm562852pfq.75.2022.10.30.15.02.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 30 Oct 2022 15:02:13 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/6] eventpoll: add support for min-wait Date: Sun, 30 Oct 2022 16:02:03 -0600 Message-Id: <20221030220203.31210-7-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221030220203.31210-1-axboe@kernel.dk> References: <20221030220203.31210-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748151781961831161?= X-GMAIL-MSGID: =?utf-8?q?1748151781961831161?= Rather than just have a timeout value for waiting on events, add EPOLL_CTL_MIN_WAIT to allow setting a minimum time that epoll_wait() should always wait for events to arrive. For medium workload efficiencies, some production workloads inject artificial timers or sleeps before calling epoll_wait() to get better batching and higher efficiencies. While this does help, it's not as efficient as it could be. By adding support for epoll_wait() for this directly, we can avoids extra context switches and scheduler and timer overhead. As an example, running an AB test on an identical workload at about ~370K reqs/second, without this change and with the sleep hack mentioned above (using 200 usec as the timeout), we're doing 310K-340K non-voluntary context switches per second. Idle CPU on the host is 27-34%. With the the sleep hack removed and epoll set to the same 200 usec value, we're handling the exact same load but at 292K-315k non-voluntary context switches and idle CPU of 33-41%, a substantial win. Basic test case: struct d { int p1, p2; }; static void *fn(void *data) { struct d *d = data; char b = 0x89; /* Generate 2 events 20 msec apart */ usleep(10000); write(d->p1, &b, sizeof(b)); usleep(10000); write(d->p2, &b, sizeof(b)); return NULL; } int main(int argc, char *argv[]) { struct epoll_event ev, events[2]; pthread_t thread; int p1[2], p2[2]; struct d d; int efd, ret; efd = epoll_create1(0); if (efd < 0) { perror("epoll_create"); return 1; } if (pipe(p1) < 0) { perror("pipe"); return 1; } if (pipe(p2) < 0) { perror("pipe"); return 1; } ev.events = EPOLLIN; ev.data.fd = p1[0]; if (epoll_ctl(efd, EPOLL_CTL_ADD, p1[0], &ev) < 0) { perror("epoll add"); return 1; } ev.events = EPOLLIN; ev.data.fd = p2[0]; if (epoll_ctl(efd, EPOLL_CTL_ADD, p2[0], &ev) < 0) { perror("epoll add"); return 1; } /* always wait 200 msec for events */ ev.data.u64 = 200000; if (epoll_ctl(efd, EPOLL_CTL_MIN_WAIT, -1, &ev) < 0) { perror("epoll add set timeout"); return 1; } d.p1 = p1[1]; d.p2 = p2[1]; pthread_create(&thread, NULL, fn, &d); /* expect to get 2 events here rather than just 1 */ ret = epoll_wait(efd, events, 2, -1); printf("epoll_wait=%d\n", ret); return 0; } Signed-off-by: Jens Axboe --- fs/eventpoll.c | 97 +++++++++++++++++++++++++++++----- include/linux/eventpoll.h | 2 +- include/uapi/linux/eventpoll.h | 1 + 3 files changed, 85 insertions(+), 15 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 962d897bbfc6..9e00f8780ec5 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -117,6 +117,9 @@ struct eppoll_entry { /* The "base" pointer is set to the container "struct epitem" */ struct epitem *base; + /* min wait time if (min_wait_ts) & 1 != 0 */ + ktime_t min_wait_ts; + /* * Wait queue item that will be linked to the target file wait * queue head. @@ -217,6 +220,9 @@ struct eventpoll { u64 gen; struct hlist_head refs; + /* min wait for epoll_wait() */ + unsigned int min_wait_ts; + #ifdef CONFIG_NET_RX_BUSY_POLL /* used to track busy poll napi_id */ unsigned int napi_id; @@ -1747,6 +1753,32 @@ static struct timespec64 *ep_timeout_to_timespec(struct timespec64 *to, long ms) return to; } +struct epoll_wq { + wait_queue_entry_t wait; + struct hrtimer timer; + ktime_t timeout_ts; + ktime_t min_wait_ts; + struct eventpoll *ep; + bool timed_out; + int maxevents; + int wakeups; +}; + +static bool ep_should_min_wait(struct epoll_wq *ewq) +{ + if (ewq->min_wait_ts & 1) { + /* just an approximation */ + if (++ewq->wakeups >= ewq->maxevents) + goto stop_wait; + if (ktime_before(ktime_get_ns(), ewq->min_wait_ts)) + return true; + } + +stop_wait: + ewq->min_wait_ts &= ~(u64) 1; + return false; +} + /* * autoremove_wake_function, but remove even on failure to wake up, because we * know that default_wake_function/ttwu will only fail if the thread is already @@ -1756,27 +1788,37 @@ static struct timespec64 *ep_timeout_to_timespec(struct timespec64 *to, long ms) static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, unsigned int mode, int sync, void *key) { - int ret = default_wake_function(wq_entry, mode, sync, key); + struct epoll_wq *ewq = container_of(wq_entry, struct epoll_wq, wait); + int ret; + + /* + * If min wait time hasn't been satisfied yet, keep waiting + */ + if (ep_should_min_wait(ewq)) + return 0; + ret = default_wake_function(wq_entry, mode, sync, key); list_del_init(&wq_entry->entry); return ret; } -struct epoll_wq { - wait_queue_entry_t wait; - struct hrtimer timer; - ktime_t timeout_ts; - bool timed_out; -}; - static enum hrtimer_restart ep_timer(struct hrtimer *timer) { struct epoll_wq *ewq = container_of(timer, struct epoll_wq, timer); struct task_struct *task = ewq->wait.private; + const bool is_min_wait = ewq->min_wait_ts & 1; + + if (!is_min_wait || ep_events_available(ewq->ep)) { + if (!is_min_wait) + ewq->timed_out = true; + ewq->min_wait_ts &= ~(u64) 1; + wake_up_process(task); + return HRTIMER_NORESTART; + } - ewq->timed_out = true; - wake_up_process(task); - return HRTIMER_NORESTART; + ewq->min_wait_ts &= ~(u64) 1; + hrtimer_set_expires_range_ns(&ewq->timer, ewq->timeout_ts, 0); + return HRTIMER_RESTART; } static void ep_schedule(struct eventpoll *ep, struct epoll_wq *ewq, ktime_t *to, @@ -1831,12 +1873,16 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, lockdep_assert_irqs_enabled(); + ewq.min_wait_ts = 0; + ewq.ep = ep; + ewq.maxevents = maxevents; ewq.timed_out = false; + ewq.wakeups = 0; if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); + ewq.timeout_ts = timespec64_to_ktime(*timeout); to = &ewq.timeout_ts; - *to = timespec64_to_ktime(*timeout); } else if (timeout) { /* * Avoid the unnecessary trip to the wait queue loop, if the @@ -1845,6 +1891,18 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, ewq.timed_out = true; } + /* + * If min_wait is set for this epoll instance, note the min_wait + * time. Ensure the lowest bit is set in ewq.min_wait_ts, that's + * the state bit for whether or not min_wait is enabled. + */ + if (ep->min_wait_ts) { + ewq.min_wait_ts = ktime_add_us(ktime_get_ns(), + ep->min_wait_ts); + ewq.min_wait_ts |= (u64) 1; + to = &ewq.min_wait_ts; + } + /* * This call is racy: We may or may not see events that are being added * to the ready list under the lock (e.g., in IRQ callbacks). For cases @@ -1913,7 +1971,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * important. */ eavail = ep_events_available(ep); - if (!eavail) { + if (!eavail || ewq.min_wait_ts & 1) { __add_wait_queue_exclusive(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); ep_schedule(ep, &ewq, to, slack); @@ -2125,6 +2183,17 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, */ ep = f.file->private_data; + /* + * Handle EPOLL_CTL_MIN_WAIT upfront as we don't need to care about + * the fd being passed in. + */ + if (op == EPOLL_CTL_MIN_WAIT) { + /* return old value */ + error = ep->min_wait_ts; + ep->min_wait_ts = epds->data; + goto error_fput; + } + /* Get the "struct file *" for the target file */ tf = fdget(fd); if (!tf.file) @@ -2257,7 +2326,7 @@ SYSCALL_DEFINE4(epoll_ctl, int, epfd, int, op, int, fd, { struct epoll_event epds; - if (ep_op_has_event(op) && + if ((ep_op_has_event(op) || op == EPOLL_CTL_MIN_WAIT) && copy_from_user(&epds, event, sizeof(struct epoll_event))) return -EFAULT; diff --git a/include/linux/eventpoll.h b/include/linux/eventpoll.h index 3337745d81bd..cbef635cb7e4 100644 --- a/include/linux/eventpoll.h +++ b/include/linux/eventpoll.h @@ -59,7 +59,7 @@ int do_epoll_ctl(int epfd, int op, int fd, struct epoll_event *epds, /* Tells if the epoll_ctl(2) operation needs an event copy from userspace */ static inline int ep_op_has_event(int op) { - return op != EPOLL_CTL_DEL; + return op != EPOLL_CTL_DEL && op != EPOLL_CTL_MIN_WAIT; } #else diff --git a/include/uapi/linux/eventpoll.h b/include/uapi/linux/eventpoll.h index 8a3432d0f0dc..81ecb1ca36e0 100644 --- a/include/uapi/linux/eventpoll.h +++ b/include/uapi/linux/eventpoll.h @@ -26,6 +26,7 @@ #define EPOLL_CTL_ADD 1 #define EPOLL_CTL_DEL 2 #define EPOLL_CTL_MOD 3 +#define EPOLL_CTL_MIN_WAIT 4 /* Epoll event masks */ #define EPOLLIN (__force __poll_t)0x00000001