From patchwork Fri Oct 28 21:43:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 12574 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1059137wru; Fri, 28 Oct 2022 14:48:56 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4N8LZTSGmI3OSh3j1oHQsKqpe+9wPljkEFBsCn5HUAzu3nhzhM3j06PRDSO/6mwUcjpSFT X-Received: by 2002:a17:906:3607:b0:7ad:a798:cdc0 with SMTP id q7-20020a170906360700b007ada798cdc0mr1220021ejb.357.1666993736214; Fri, 28 Oct 2022 14:48:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666993736; cv=none; d=google.com; s=arc-20160816; b=iTClhYoyWQNZyFLOoEsLc7N3d0zEshFNloTjVX+49GuZ2pbzVVndY09Jd4+Cc+Lbrl o0QWnnZEzOBh9TJPADuFba7ovGGxu755hstvkqfiZRzRpLwEybFqxMRTEJa4pXD9GCL1 qEj7mU7ZQNnx0VG/Q8P6cxtvPVHUSTFyo3eqJA2DcruS6yjA1JKKh3UsLoqFXRQF6EnZ DlBxeED8Bx9DCcYr6ZfOl8SItryZRHlA6dhmIXPXE0u/Qm4ydWdlRcy3WBMs4RtIak0+ 6/nSaKdaaa/7sxAdhh8eFMltlnJ6Jt8KtDKL45263BL/ixtBPvhpqvGrQFQl0uqRKCw5 iqtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=tiguh3u9eggshsoZh3GPe1CsJMkdP/JI4kBBBED1cmY=; b=UtPCYu0FRXCEzBHXeiaxaMsZOekWTYeyIR5e9wxPthttz6kTF5oRvKgUD50X1BGAHO AUG+dC+NuTDSvAqtztPRd6yFPsifM/vTaweY07FsAId/wHfbOD9pmFqcx9it0EH/P309 6/gZVXA+l2+f2+8csQfjaZgYhlam+e/PtYLGD6f9HIYN6BfsnZiSUGZsg1uhI69vL6KB /GDRj0ytUn7SvKOYlDkT3XIXCL8UzEieLN/T78h7o+KYyBMedHIKrJ0uOgn0iFGrO5ht d1QJetBhtEPsjnYffdTRf4SbLm4o/qjxbETdp+3Z0g/8MJ/YdHZ7Y8EXrVKhaZrjkX1j DBkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=hx1oQADE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n14-20020a05640205ce00b00461fa05b004si6448849edx.105.2022.10.28.14.48.33; Fri, 28 Oct 2022 14:48:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=hx1oQADE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229862AbiJ1Vnq (ORCPT + 99 others); Fri, 28 Oct 2022 17:43:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230015AbiJ1Vnd (ORCPT ); Fri, 28 Oct 2022 17:43:33 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E9FA24BA88 for ; Fri, 28 Oct 2022 14:43:32 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id b11so5739766pjp.2 for ; Fri, 28 Oct 2022 14:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tiguh3u9eggshsoZh3GPe1CsJMkdP/JI4kBBBED1cmY=; b=hx1oQADEdyNFaX9tPTLZ7yu68JZNzS/ZRcV4/4ftjbbkrAqlVDlR59NVfME3EASL7q SmUZXZegPFn0vYGmx5BxprclqeaTTW4QnM2arglNSSFwKQrl1iodtZu8BXYqFEqnVuFi ljvLjzPtuZoTpUh1IE0dt0gDTqUuRuJZSvKmAldJaFcuvir511BxZioWBr8snZtpOmNU kGRxJs0HfTGxIVuRLAEQsgxVWerlzfBzFwlw1DdShu+GHUP2kGzSd8BQMNa4mjKlAhrd 4Xx36O4qXEWvcpXh4xVohKWyW6vQoZAyKJl8g8ZBtW4724mea8IKCS9fUeXxyXQ3S0AK SwRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tiguh3u9eggshsoZh3GPe1CsJMkdP/JI4kBBBED1cmY=; b=PWhpEskgsKfnHwgo6xpnEJTKnEdENFlYC5LH546PrL9Ga9DRxhsKFyxpHRfWDgej8e e9thchDMOZEaSFkUX2/GVtr7mVbxLBvyyrTs24R6YnujK0NfNjs5hB8Z6cnSBYX540u/ C6tborJgwwW8rFtrj4JoreSGyd7I4Ir0XipxG5ic+c8RYj0Gf9On6eqserXO5c53vs6k ek5yA33Jo7V5Qa1SHUIqXHdVamD8GlxR4f5/9A4G8cHvQN+xEb5kOXKojSq1DcXz5e6g 5unZ7ncANUQZe3euhPamlyMDhHRfci0bXpDr3zuU/oWJ5pMgtdTKe34f68KKpltkGoVb VxCA== X-Gm-Message-State: ACrzQf0i2ZFuwjrkqmZQ0EHfkZThDFlkAeK1sUm06M9DeOggyoBnl0N4 2o6FJnnTUWELXz1WtIkDlTPb5NkM5MJ78dZ0 X-Received: by 2002:a17:90a:4ece:b0:213:1130:ca9c with SMTP id v14-20020a17090a4ece00b002131130ca9cmr17984235pjl.17.1666993411637; Fri, 28 Oct 2022 14:43:31 -0700 (PDT) Received: from localhost.localdomain ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id u6-20020a17090a1d4600b002130c269b6fsm2993855pju.1.2022.10.28.14.43.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Oct 2022 14:43:31 -0700 (PDT) From: Jens Axboe To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/5] eventpoll: split out wait handling Date: Fri, 28 Oct 2022 15:43:22 -0600 Message-Id: <20221028214325.13496-3-axboe@kernel.dk> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221028214325.13496-1-axboe@kernel.dk> References: <20221028214325.13496-1-axboe@kernel.dk> MIME-Version: 1.0 X-Spam-Status: No, score=1.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,RCVD_IN_SBL_CSS,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Level: * X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747969624139229520?= X-GMAIL-MSGID: =?utf-8?q?1747969624139229520?= In preparation for making changes to how wakeups and sleeps are done, move the timeout scheduling into a helper and manage it rather than rely on schedule_hrtimeout_range(). Signed-off-by: Jens Axboe --- fs/eventpoll.c | 70 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 56 insertions(+), 14 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3061bdde6cba..f53bb4ec9e91 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1762,6 +1762,47 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, return ret; } +struct epoll_wq { + wait_queue_entry_t wait; + struct hrtimer timer; + bool timed_out; +}; + +static enum hrtimer_restart ep_timer(struct hrtimer *timer) +{ + struct epoll_wq *ewq = container_of(timer, struct epoll_wq, timer); + struct task_struct *task = ewq->wait.private; + + ewq->timed_out = true; + wake_up_process(task); + return HRTIMER_NORESTART; +} + +static void ep_schedule(struct eventpoll *ep, struct epoll_wq *ewq, ktime_t *to, + u64 slack) +{ + if (ewq->timed_out) + return; + if (to && *to == 0) { + ewq->timed_out = true; + return; + } + if (!to) { + schedule(); + return; + } + + hrtimer_init_on_stack(&ewq->timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS); + ewq->timer.function = ep_timer; + hrtimer_set_expires_range_ns(&ewq->timer, *to, slack); + hrtimer_start_expires(&ewq->timer, HRTIMER_MODE_ABS); + + schedule(); + + hrtimer_cancel(&ewq->timer); + destroy_hrtimer_on_stack(&ewq->timer); +} + /** * ep_poll - Retrieves ready events, and delivers them to the caller-supplied * event buffer. @@ -1782,13 +1823,15 @@ static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, int maxevents, struct timespec64 *timeout) { - int res, eavail, timed_out = 0; + int res, eavail; u64 slack = 0; - wait_queue_entry_t wait; ktime_t expires, *to = NULL; + struct epoll_wq ewq; lockdep_assert_irqs_enabled(); + ewq.timed_out = false; + if (timeout && (timeout->tv_sec | timeout->tv_nsec)) { slack = select_estimate_accuracy(timeout); to = &expires; @@ -1798,7 +1841,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Avoid the unnecessary trip to the wait queue loop, if the * caller specified a non blocking operation. */ - timed_out = 1; + ewq.timed_out = 1; } /* @@ -1823,10 +1866,10 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, return res; } - if (timed_out) + if (ewq.timed_out) return 0; - eavail = ep_busy_loop(ep, timed_out); + eavail = ep_busy_loop(ep, ewq.timed_out); if (eavail) continue; @@ -1850,8 +1893,8 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * performance issue if a process is killed, causing all of its * threads to wake up without being removed normally. */ - init_wait(&wait); - wait.func = ep_autoremove_wake_function; + init_wait(&ewq.wait); + ewq.wait.func = ep_autoremove_wake_function; write_lock_irq(&ep->lock); /* @@ -1870,10 +1913,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, */ eavail = ep_events_available(ep); if (!eavail) { - __add_wait_queue_exclusive(&ep->wq, &wait); + __add_wait_queue_exclusive(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); - timed_out = !schedule_hrtimeout_range(to, slack, - HRTIMER_MODE_ABS); + ep_schedule(ep, &ewq, to, slack); } else { write_unlock_irq(&ep->lock); } @@ -1887,7 +1929,7 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, */ eavail = 1; - if (!list_empty_careful(&wait.entry)) { + if (!list_empty_careful(&ewq.wait.entry)) { write_lock_irq(&ep->lock); /* * If the thread timed out and is not on the wait queue, @@ -1896,9 +1938,9 @@ static int ep_poll(struct eventpoll *ep, struct epoll_event __user *events, * Thus, when wait.entry is empty, it needs to harvest * events. */ - if (timed_out) - eavail = list_empty(&wait.entry); - __remove_wait_queue(&ep->wq, &wait); + if (ewq.timed_out) + eavail = list_empty(&ewq.wait.entry); + __remove_wait_queue(&ep->wq, &ewq.wait); write_unlock_irq(&ep->lock); } }