From patchwork Wed Jan 31 01:47:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 194426 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp1614310dyb; Tue, 30 Jan 2024 17:48:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IGMqWvPfeM46eEypZ5xUPzffrrT5uVlsm1CU5Rji9lhrOatbAFgc+4riq3hrmrB1bZwQBdL X-Received: by 2002:aa7:d417:0:b0:55c:7b3b:5990 with SMTP id z23-20020aa7d417000000b0055c7b3b5990mr73514edq.7.1706665728430; Tue, 30 Jan 2024 17:48:48 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706665728; cv=pass; d=google.com; s=arc-20160816; b=Tl5uPrDSZdas/v8xnvyA0BuDx8V/ohCTbg4iIYGI8tNl8KhUDSHXVravaXkSRlaStU 43TgIsEqNUbEdhF8Drdpd6nR8I59WgGBOcUVtg/Dwt/kOjlZrqrqjEyypGiV2+GkxRnr 81PpHBaZBhcLYkVjNEOhCTp4VvVAtZNcjiaR5q3HA2yK5QxBiQcXloViuS6MjipInJHE yd7HZ7x5ZYrWvZgZiq0EPbZhWSE3LfbjaKAV3fm8nDCWkDSxRuUs68HvfwVggX2ZVPp9 MR33xYilsWgLMRE1ZOh6RPZCeZ6TubfIawJQQkho4DzIbHKftlEh99y4f7QI4vYXHrAp W3tw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=XPyBFpA03VgPUcSSFnhqRnjQDUKtK8Zok+qzBXzC1y0=; fh=mM+KmrFA2dHdj26Z699u05B0Fz2Aiu+tcxvtwccQwCg=; b=envw36Q5nl5lkdyyWdodNInkWVSIzctD0y3znpM/ED5fcq2nR+rxGQtyTN1X3aJV/V 3mqiPp22Apa8iNL1mbK8u3NPPPT8hXnyw44S33u9qoumtnJR6Pe0IhuVf8W23H8XrXNt CD9o9EUdAS0cNXbg1BEh4q7qRE+GNg6U8Pp4yOx6PopbC4CEc/XhA1qUBgGsnHBDtHiI UKq2bHSLinR/fnoqEnsBGtIzlBJac0TlBw7KDqK0ZMO2OpcmR3ndDVhIxC2kEl7DHNoj 1Z5NtlO2XRG+n1JsvDYhIQkQeUE7nPXws6DhAdXnWIyfHcZd2u8JuLrsQT43NKY1hLSy ypXA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=Ct4VVviF; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-45605-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-45605-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCVpb0wgiWRULQYSXbI+eO0vI30kaZqCDkVAaVBq7LyXX23NU4eHLVLtHRPjgGmz8lDt+3NcSaCvI8U5mqOcVsy6O+zXpQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id ij12-20020a056402158c00b0055cd3f3153csi5044355edb.16.2024.01.30.17.48.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 17:48:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-45605-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=Ct4VVviF; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-45605-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-45605-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id DC0671F25F62 for ; Wed, 31 Jan 2024 01:48:47 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 277C1EAD7; Wed, 31 Jan 2024 01:47:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="Ct4VVviF" Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F4345223 for ; Wed, 31 Jan 2024 01:47:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706665668; cv=none; b=ab9ZpmzTAuus4B56nVRYxSIa4g4ivIEq+zyY1lQ0OBpe+PGl5kd0bndvcFDBGMBWQeauxCsajijXuqs3B07h4K+ADpyjGkubW0FwDpR4BMXYAOZ/LxL2OCeBKyw9cRi/E8LpUtgqdHNlWZv55b1YMwj4ALHhcVeAY9MCg7sNcws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706665668; c=relaxed/simple; bh=iOH3rDvMU6l8EibI+PkwMyfqV5hY0EoK3y01y1v3/4g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mYNsMxyMRcMXmwUxHpzg23osWdr6YqKDH4rptGsfiH3IoMW1ht7mYb8CKqXRFfYxTOy4RYly1NVl4q/1PXkp8ow0LCyajJE8Q5zbOTW8KgsqFPxT0koJx+tZKh/9L7O74I/s1hk9WdNZAC9Kkq3WzSAT1jh1/tzkxWBHbq9DqXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=Ct4VVviF; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5dbcfa0eb5dso864398a12.3 for ; Tue, 30 Jan 2024 17:47:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1706665666; x=1707270466; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XPyBFpA03VgPUcSSFnhqRnjQDUKtK8Zok+qzBXzC1y0=; b=Ct4VVviF+wjqtFEpy8XDL9kaii13JkxQL/TJTDbUmzvPA32VNjCrqlJMKNAXZneaJF 92XRAfVg7kZE44t8ZxMYJj7S2lBm0pqXl3vSMDEHxjdgl5F5RvBJLDlXFKE5s3FZIbFb 5TvPhIEBEjW0dVPmStvud9ktM5dZh4+P8M0ZA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706665666; x=1707270466; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XPyBFpA03VgPUcSSFnhqRnjQDUKtK8Zok+qzBXzC1y0=; b=i3wEhaCj47cTqT4//yTYbttMELc6KIuYLC6yje5bgz4ZbA5wHNyTnMWg1enI/BYVcA SyZEEo1ya1ltiI9mq/vsvUr0qd+214JFjF4ttSwiexC343s9vbJ3CmEP0pqVFrAwIjCZ uD7IPlh3PZf4EdxgKKGiryjMXuUBFFwcPdlRZDiws9YdBTPOnUh+jrl13ZztmtMENsV+ +MzVAwZ2TnOWbUElwv1MzBALs/cNuN2i/3upIoZHfD5dvpyCBvciF1fFk8Ty4CAP4svd TU05z6T+hLbNvFX5hgoq+7YfpM9pLw4Yugu9TkI/pqTCzRxX4QbHsScS1rD2AH78j2w1 PXvg== X-Gm-Message-State: AOJu0YxhW7XbPNUq1/mMkHlWGYDdeaczJgHoKkzucL5ONukr0EZam+Jq V3//LhImCkpNriYPo4RuYYjuE+4qJyPz/+x6iUph2OnMaKcRBsFE3lZ+Dd54C0fxM66BLdTYuSA 6shlShrkhkI5DXjhgjiaeBGeLhcKWQ1AJvwGYc3CXiGJ4xpCfT6TJXeFDXpb/zSnNAFMdJ9RDZE tWdfobZO9/w6rOSXxsu5PIOSovSJ8tZDs7wcWOFCPRtsoXZg== X-Received: by 2002:a05:6a20:7002:b0:19c:8fa7:66da with SMTP id h2-20020a056a20700200b0019c8fa766damr207347pza.1.1706665665985; Tue, 30 Jan 2024 17:47:45 -0800 (PST) Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id li6-20020a170903294600b001d90fe6da6esm1837846plb.305.2024.01.30.17.47.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 17:47:45 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v4 1/3] eventpoll: support busy poll per epoll instance Date: Wed, 31 Jan 2024 01:47:31 +0000 Message-Id: <20240131014738.469858-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240131014738.469858-1-jdamato@fastly.com> References: <20240131014738.469858-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789568722474489808 X-GMAIL-MSGID: 1789568722474489808 Allow busy polling on a per-epoll context basis. The per-epoll context usec timeout value is preferred, but the pre-existing system wide sysctl value is still supported if it specified. Note that this change uses an xor: either per epoll instance busy polling is enabled on the epoll instance or system wide epoll is enabled. Enabling both is disallowed. Signed-off-by: Joe Damato --- fs/eventpoll.c | 49 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3534d36a1474..ce75189d46df 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -227,6 +227,8 @@ struct eventpoll { #ifdef CONFIG_NET_RX_BUSY_POLL /* used to track busy poll napi_id */ unsigned int napi_id; + /* busy poll timeout */ + u64 busy_poll_usecs; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -386,12 +388,44 @@ static inline int ep_events_available(struct eventpoll *ep) READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR; } +/** + * busy_loop_ep_timeout - check if busy poll has timed out. The timeout value + * from the epoll instance ep is preferred, but if it is not set fallback to + * the system-wide global via busy_loop_timeout. + * + * @start_time: The start time used to compute the remaining time until timeout. + * @ep: Pointer to the eventpoll context. + * + * Return: true if the timeout has expired, false otherwise. + */ +static inline bool busy_loop_ep_timeout(unsigned long start_time, struct eventpoll *ep) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + unsigned long bp_usec = READ_ONCE(ep->busy_poll_usecs); + + if (bp_usec) { + unsigned long end_time = start_time + bp_usec; + unsigned long now = busy_loop_current_time(); + + return time_after(now, end_time); + } else { + return busy_loop_timeout(start_time); + } +#endif + return true; +} + #ifdef CONFIG_NET_RX_BUSY_POLL +static bool ep_busy_loop_on(struct eventpoll *ep) +{ + return !!ep->busy_poll_usecs || net_busy_loop_on(); +} + static bool ep_busy_loop_end(void *p, unsigned long start_time) { struct eventpoll *ep = p; - return ep_events_available(ep) || busy_loop_timeout(start_time); + return ep_events_available(ep) || busy_loop_ep_timeout(start_time, ep); } /* @@ -404,7 +438,7 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); - if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { + if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, BUSY_POLL_BUDGET); if (ep_events_available(ep)) @@ -430,7 +464,8 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) struct socket *sock; struct sock *sk; - if (!net_busy_loop_on()) + ep = epi->ep; + if (!ep_busy_loop_on(ep)) return; sock = sock_from_file(epi->ffd.file); @@ -442,7 +477,6 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) return; napi_id = READ_ONCE(sk->sk_napi_id); - ep = epi->ep; /* Non-NAPI IDs can be rejected * or @@ -466,6 +500,10 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) { } +static inline bool ep_busy_loop_on(struct eventpoll *ep) +{ + return false; +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -2058,6 +2096,9 @@ static int do_epoll_create(int flags) error = PTR_ERR(file); goto out_free_fd; } +#ifdef CONFIG_NET_RX_BUSY_POLL + ep->busy_poll_usecs = 0; +#endif ep->file = file; fd_install(fd, file); return fd; From patchwork Wed Jan 31 01:47:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 194427 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp1614356dyb; Tue, 30 Jan 2024 17:48:55 -0800 (PST) X-Google-Smtp-Source: AGHT+IEZValUMsRoJe3Kq9t5zgZ/PJxmt2MEXayVjuSwQmDuju2q/ntGHfEZtzl7V+PnBxWT6jOJ X-Received: by 2002:a25:2e4a:0:b0:dc2:4397:6add with SMTP id b10-20020a252e4a000000b00dc243976addmr352789ybn.32.1706665735450; Tue, 30 Jan 2024 17:48:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706665735; cv=pass; d=google.com; s=arc-20160816; b=b/rUFGcbyotY9k1zGvOZavUD7B5NEUEm8n0ixpouA4vJZbopMKU3pa/0gB0Qz2hLec iq92K4zNc4GsBNQMV9Bbj1mq6BAiFybIA1X40USK74gEFNrLc8xUC7ZsYS/rvC8AcbXy XteRJNshHT736yup4sL4ITO9k+548Y3VtDO2a3HPjXELMZsnDr0Lw/lyqY1zKsOadUeP TSa2kJ98Wdct4oWaoRhoM7kZMTNvxw5666q088e9odJ/pprnxsIkA3QVXvLCqXOrNVZv Bz3ACUFi7+5r9j4/YA572MgCdhXAQwjbTrBcFc+O2dTTAD7VT7Pl5Un+Qx1wU1vEX2Nc 8w3Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; fh=eJkfYnCQ7EwPhmPtEEu4DmqhkpaNK7gCtJmyRqXtXLA=; b=E1uXx+3JlmjP/4Z9jfQUE5d5yCXU2eI7BHIb1C5eHWN5M6NDG6FsIWCxN96793sE/3 KBD9yx8jZjLIhT+P+1g5LaVeGSTYV1xajF3a5hs8dp7IPf5AWt9nYguP9GYdRM3JUkqu MMA5KCNX2U07TETADQNYbBhHmPF+eUi63s5kbiIxyWHBos+2zNBsMi39ahlerHbJlkiH KTpPVLm+cizAxx4Yj+MwRJj6g1cm+zbbesyWVj4g1UXN9+KZFJwwXgkwJ4gbdaUTRsMs fX2v7eC16dFDIObg6wUE+xcstY/Q4ePUeB2JH1kApGt5K7uUBT0Jk5su3qYFqttVeeMY 8O+g==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b="qtm/73JW"; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-45606-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-45606-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCXfDApgPVFArgu1ueq8hY/wx9VCFjy6S1670jB3oiK6p9Ctxago0An4ZpO+Njhv00B0XDXgYbN117dKOJQFyyQAF1uOSA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id iw8-20020a0562140f2800b006818f0b3569si11413171qvb.182.2024.01.30.17.48.55 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 17:48:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-45606-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b="qtm/73JW"; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-45606-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-45606-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3D2D51C25210 for ; Wed, 31 Jan 2024 01:48:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 87ECBFBED; Wed, 31 Jan 2024 01:47:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="qtm/73JW" Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3EF025C99 for ; Wed, 31 Jan 2024 01:47:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706665670; cv=none; b=QAc/vxrUl2QWwORqG6CjMWZA1vNcK4jwMII4QeOwpo4OYEVQNKg/D7hctLTq2XTn7x5v1hirnbZPcfC4ivHr+6stJu8WSiCmPmHpO2hcFfCkNPVjVRrUrMDqYsfPoMkQe2FWTIDqyajgDFnjZcZrIO+1qAT+4mKdSUgyYqdyuY8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706665670; c=relaxed/simple; bh=+NvwukKrlDHz6eylGdtOKZKD8hFQLLmD1KnjinUcTaA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aEMleWKQXjAvc1sC5QwdO80/9n0cA7OIEu74ISJHOnMJj9sbAKCrZkL4HKFXNsHbwMvHy9xwPlOg+AOVZTdgP3+IDjkN3OCdnCMB5hGa5Y316HlJ907yYrj34ce0ux3aQkmVqJ3g+Hw12ZpTfvTXZTaBtq3pdV7+aZaELJIJzmQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=qtm/73JW; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1d73066880eso37005675ad.3 for ; Tue, 30 Jan 2024 17:47:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1706665668; x=1707270468; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; b=qtm/73JWx7S9OK0UV6TPJcE/rlOXvOveLlaD3eRlhDwbPAAuBPb/5a97Gluz6XwVSP A5rMWNs+JQiBtvpHVsyEXZnwnC/xP5BRqk+nJgzj2Ovgbj17EyjfHkueZzgmz+0gZb1D CLJYNVRys7ObuXJyvYu/F0W16yb0lgDTLw5kc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706665668; x=1707270468; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; b=HQzbQzZLylWNvbMykk6M9xTcSuHbzewGN6aSWMxEPBL9cNckjLiU+EAhk2JVDbt0ro /JafoNewVEI1wswX2zQUxLfyrkwFTOGvAPISchwLQArzrzq1zQW0bHXSpARHI9AqwX4v zx2FID54ydTmEae6SVChmwOt0W2cN6isI8c/DuTz0QFyludSQFxmhHENfhi2fKuvRDZM xMPy3dVg401C5Tx3V4c1Dsjbbk0nYzQq4kXoSFsRC5ylpwne/xHJ7rHOE0HoK+Dj2OaX mJu2vhuLo2ZqRGmvd9+em3lDd/cZuZfWbRDIBIbx6IQ0AkAJ6IJdbMwGDYLJbBTb+ssY Bgeg== X-Gm-Message-State: AOJu0YxCmLCvLjowMtveICljIbLZXV1KRef7IdbPPbPA5kRwHN2yhD01 JSTKKqTquypwCKCJDHYgHUstefMkq/t7VrNoeBO0vu5Fa+IY8DFYeTBd4AMN6xo9rXOLy7mGz5x BEnc7UbIDT8m/1ZGabzcLwZgTudSmWH+Cv+HV/OPQwgUUzKkk0zUWew8r3tp2V+lmC4kTh4zQKZ J+rTb6FyjIwUCSSgadWxe5ob9bv2CND0SHTXiKwLqRAYBawQ== X-Received: by 2002:a17:902:bcc5:b0:1d6:fe15:b588 with SMTP id o5-20020a170902bcc500b001d6fe15b588mr324376pls.64.1706665668134; Tue, 30 Jan 2024 17:47:48 -0800 (PST) Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id li6-20020a170903294600b001d90fe6da6esm1837846plb.305.2024.01.30.17.47.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 17:47:47 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v4 2/3] eventpoll: Add per-epoll busy poll packet budget Date: Wed, 31 Jan 2024 01:47:32 +0000 Message-Id: <20240131014738.469858-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240131014738.469858-1-jdamato@fastly.com> References: <20240131014738.469858-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789568730125802093 X-GMAIL-MSGID: 1789568730125802093 When using epoll-based busy poll, the packet budget is hardcoded to BUSY_POLL_BUDGET (8). Users may desire larger busy poll budgets, which can potentially increase throughput when busy polling under high network load. Other busy poll methods allow setting the busy poll budget via SO_BUSY_POLL_BUDGET, but epoll-based busy polling uses a hardcoded value. Fix this edge case by adding support for a per-epoll context busy poll packet budget. If not specified, the default value (BUSY_POLL_BUDGET) is used. Signed-off-by: Joe Damato --- fs/eventpoll.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ce75189d46df..3985434df527 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -229,6 +229,8 @@ struct eventpoll { unsigned int napi_id; /* busy poll timeout */ u64 busy_poll_usecs; + /* busy poll packet budget */ + u16 busy_poll_budget; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -437,10 +439,14 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); + u16 budget = READ_ONCE(ep->busy_poll_budget); + + if (!budget) + budget = BUSY_POLL_BUDGET; if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, - BUSY_POLL_BUDGET); + budget); if (ep_events_available(ep)) return true; /* @@ -2098,6 +2104,7 @@ static int do_epoll_create(int flags) } #ifdef CONFIG_NET_RX_BUSY_POLL ep->busy_poll_usecs = 0; + ep->busy_poll_budget = 0; #endif ep->file = file; fd_install(fd, file); From patchwork Wed Jan 31 01:47:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 194428 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp1614494dyb; Tue, 30 Jan 2024 17:49:20 -0800 (PST) X-Google-Smtp-Source: AGHT+IFdTzN6AJIsXGMv72p+LyUUjx5IM8400v7od/M9zHu8DZKFljek5faoz3J7BBg70Jfs4sd9 X-Received: by 2002:a17:906:fb0a:b0:a35:3c1f:b29f with SMTP id lz10-20020a170906fb0a00b00a353c1fb29fmr216220ejb.5.1706665760213; Tue, 30 Jan 2024 17:49:20 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706665760; cv=pass; d=google.com; s=arc-20160816; b=ecI0Kp00/3hVKgu7lBkDLsRFD7f7DyvlmZ8PQjsFLUgTOPL/UPmdkRXBHdkcytDHck GK0q6sQ4MUvY7IcdjQmUNS2Umsnzb0E46tCDCt0QyfAAeFNi9PBqpyKBahJvQ6woe4Co T0996Mok9vyEtpcF0rUHO1pRI1KnhMhaFWk+0cQLL1MP+Dm/aH/2esiuEH+MeBIzo7uV voXEkVZMdNdR9123Sg89DYixJkd4Ep6ycjeAu78HB/gUY7rmQmCZRsAQEbTdm+WLClsJ P7V9czJ9rzMad3zlOiY+ed50fImhj97KRUTmNv1x5KA/fB+5Ekn4tapQ8BB/QCeBMFz2 GRMA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=8eiQqj8AyPJBMkwxcjL/fD8XyFJwiWNQcNlhT/VtKOo=; fh=P86qBeV2CrgcCdf3Yn6dmoYcXBGrSao9oFFyeZ8ilbc=; b=Jf3OymavHEu7r/BqMEMc2m891b7WL964bsBAqNJz6E5GkdoCxVzDyf6wlWm/lZgLbS i3uoEodxLa6Q82QQgbVV4x7kvIkhVcfzQRuUwrW1f/NKJ6BR7+AtfTDcdSus6oYTbY2R zzbOFKjp5YQL/Op6kZJLRqjrdI4Owhw2u9HGkWJuI1pj1DKJBgvitjLng690FG1XIx4i O8Cbv44x4FhEU5xMYMbyRSjSjQoaIb8Uaw/CcVLvRJ7l1V5JCYJy/jH6JrD4mN1RT768 tvZhVLvw6CAyLcKX/cFzK7y2Gm+/4hiIrvUv/1ip/ois8aRIV9D/mo5mK+QV9CG7oQ/j lWIg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=LBX4ZcEd; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-45607-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-45607-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCUTOOoKXn+ehkpfcIj3TlPqrtSvh13yWcmXdMwUqjhgzfXdU2FRl7xBa/CBNeyoL+6FZPPBJznEnj6aACpYVRZUIghLmA== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id r3-20020a1709067fc300b00a35a09962fesi2817357ejs.128.2024.01.30.17.49.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 17:49:20 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-45607-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=LBX4ZcEd; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-45607-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-45607-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id A398B1F25644 for ; Wed, 31 Jan 2024 01:49:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D17D7134DD; Wed, 31 Jan 2024 01:48:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="LBX4ZcEd" Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E98C24C98 for ; Wed, 31 Jan 2024 01:47:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706665677; cv=none; b=U9p6OAgVcZFLyLlfs8h0+IGzWj7995ba87jQ01tDUj3bibVRhYet8jHVhr+X6lr+B5sDSqeXbq0iwdHNRLurn2UTmC//m88MivXgMGLMRPIzirUq9KamYTysuNEEM6kyNPL1k8NTS/XkC7GdEspauqnHvWlaGJzWdC6anRhC/zQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706665677; c=relaxed/simple; bh=PCgM0jeKVsYznDgZ+Is72gv1JvGoJVesIskQPr8m5bI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TtreKHqn6TSbGdx24JT9z0p2lra0xSNE8wkzu9AK4NQcRlUPM+rQwfDDC1Y5YuDjvkCrE/JxmD78U1FqHLk5qkEWS/x6WFjuABT7diR9z55X+JLMCw9NSnHyWbozu/jEEapx8V5dYxBMkYFE57oDYOBa0DSi8lvNme0/iubHb/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=LBX4ZcEd; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d70b0e521eso33030325ad.1 for ; Tue, 30 Jan 2024 17:47:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1706665674; x=1707270474; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8eiQqj8AyPJBMkwxcjL/fD8XyFJwiWNQcNlhT/VtKOo=; b=LBX4ZcEdP1MS1rkBBEBMw7hzj41jHuuW4HbTFgIDzvWrgKxKBrZDr01AelE7XpitZa 80JTYLeH2b5GdWDyCpAYDCZa8iMe2elI0oEKXKsz0qVTZ9B0eNJo4F+kzqIqZbWZ/SZv MWDEDbGL16imyHxrzn8oWhRp5znSl3R1MSjzU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706665674; x=1707270474; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8eiQqj8AyPJBMkwxcjL/fD8XyFJwiWNQcNlhT/VtKOo=; b=byy2DEFslMgYRWYhbLJWS3s1Me0J4gib05qkeyBbiYo9i8Pc21kPJ+0dPxiEm3gFqF 70N1EqUfabrP+mlO3MKH103cXrK1npD3jJ1UENkX9ia31HihIkaaqjcLFgAgPUj6S0F8 NrypYNMcIqLausxJ5GP9RzrV6/F5SuLblfDE5AOFtpvn1xWg1nROlaef0FWea8cCoUqK nGeaiA7wn1CjLP6CZxzfcVW556UJYm2LPcK1sWfkusdKeQ9mOd5QXwXP1Wx/BB/j4A5z jSC+M8+cqZvwfnaco6Q8u13+x3KOcs/dmvUosNd5oqxUZQZelGO0hR/ClrQ2EtRLu1o5 /XWQ== X-Gm-Message-State: AOJu0YxWBqlONApY+GBYih5unenhRqX6YZkTRo8gbt3/pOUBsxaUUarW HLomL05007vGkkTUuZyDoP4dP2pzw2nxcDPCW4GvwUYVniBEN2Ye3dkVBApr077uhIGfA+PAp+C 6stKlTGy2q6KmLEJVwJwgGlAZK9IgpYRNOycRH672YXx71NhzXZAuiOwKfIPL8mV98NX8HaUWf8 eVXaHHiNzUzSksHcGCBMW9633ZeGdSugPMCWcTlaDIKy7Hiw== X-Received: by 2002:a17:902:d38c:b0:1d8:ee30:aaac with SMTP id e12-20020a170902d38c00b001d8ee30aaacmr309763pld.69.1706665674460; Tue, 30 Jan 2024 17:47:54 -0800 (PST) Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id li6-20020a170903294600b001d90fe6da6esm1837846plb.305.2024.01.30.17.47.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 17:47:53 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, Joe Damato , Jonathan Corbet , Alexander Viro , Jan Kara , Michael Ellerman , Greg Kroah-Hartman , Nathan Lynch , Thomas Zimmermann , Maik Broemme , Steve French , Julien Panis , Jiri Slaby , Thomas Huth , Andrew Waterman , Albert Ou , Palmer Dabbelt , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v4 3/3] eventpoll: Add epoll ioctl for epoll_params Date: Wed, 31 Jan 2024 01:47:33 +0000 Message-Id: <20240131014738.469858-4-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240131014738.469858-1-jdamato@fastly.com> References: <20240131014738.469858-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789568756152136518 X-GMAIL-MSGID: 1789568756152136518 Add an ioctl for getting and setting epoll_params. User programs can use this ioctl to get and set the busy poll usec time or packet budget params for a specific epoll context. Parameters are limited: - busy_poll_usecs is limited to <= u32_max - busy_poll_budget is limited to <= NAPI_POLL_WEIGHT by unprivileged users (!capable(CAP_NET_ADMIN)). Signed-off-by: Joe Damato --- .../userspace-api/ioctl/ioctl-number.rst | 1 + fs/eventpoll.c | 65 +++++++++++++++++++ include/uapi/linux/eventpoll.h | 12 ++++ 3 files changed, 78 insertions(+) diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 457e16f06e04..b33918232f78 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -309,6 +309,7 @@ Code Seq# Include File Comments 0x89 0B-DF linux/sockios.h 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range 0x89 F0-FF linux/sockios.h SIOCDEVPRIVATE range +0x8A 00-1F linux/eventpoll.h 0x8B all linux/wireless.h 0x8C 00-3F WiNRADiO driver diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3985434df527..afdb91c6faa8 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -37,6 +37,7 @@ #include #include #include +#include #include /* @@ -495,6 +496,42 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) ep->napi_id = napi_id; } +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct eventpoll *ep; + struct epoll_params epoll_params; + void __user *uarg = (void __user *) arg; + + ep = file->private_data; + + switch (cmd) { + case EPIOCSPARAMS: + if (copy_from_user(&epoll_params, uarg, sizeof(epoll_params))) + return -EFAULT; + + if (epoll_params.busy_poll_usecs > U32_MAX) + return -EINVAL; + + if (epoll_params.busy_poll_budget > NAPI_POLL_WEIGHT && + !capable(CAP_NET_ADMIN)) + return -EPERM; + + ep->busy_poll_usecs = epoll_params.busy_poll_usecs; + ep->busy_poll_budget = epoll_params.busy_poll_budget; + return 0; + case EPIOCGPARAMS: + memset(&epoll_params, 0, sizeof(epoll_params)); + epoll_params.busy_poll_usecs = ep->busy_poll_usecs; + epoll_params.busy_poll_budget = ep->busy_poll_budget; + if (copy_to_user(uarg, &epoll_params, sizeof(epoll_params))) + return -EFAULT; + return 0; + default: + return -ENOIOCTLCMD; + } +} + #else static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) @@ -510,6 +547,12 @@ static inline bool ep_busy_loop_on(struct eventpoll *ep) { return false; } + +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return -EOPNOTSUPP; +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -869,6 +912,26 @@ static void ep_clear_and_put(struct eventpoll *ep) ep_free(ep); } +static long ep_eventpoll_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + int ret; + + if (!is_file_epoll(file)) + return -EINVAL; + + switch (cmd) { + case EPIOCSPARAMS: + case EPIOCGPARAMS: + ret = ep_eventpoll_bp_ioctl(file, cmd, arg); + break; + default: + ret = -EINVAL; + break; + } + + return ret; +} + static int ep_eventpoll_release(struct inode *inode, struct file *file) { struct eventpoll *ep = file->private_data; @@ -975,6 +1038,8 @@ static const struct file_operations eventpoll_fops = { .release = ep_eventpoll_release, .poll = ep_eventpoll_poll, .llseek = noop_llseek, + .unlocked_ioctl = ep_eventpoll_ioctl, + .compat_ioctl = compat_ptr_ioctl, }; /* diff --git a/include/uapi/linux/eventpoll.h b/include/uapi/linux/eventpoll.h index cfbcc4cc49ac..98e5ea525dd0 100644 --- a/include/uapi/linux/eventpoll.h +++ b/include/uapi/linux/eventpoll.h @@ -85,4 +85,16 @@ struct epoll_event { __u64 data; } EPOLL_PACKED; +struct epoll_params { + __aligned_u64 busy_poll_usecs; + __u16 busy_poll_budget; + + /* pad the struct to a multiple of 64bits for alignment on all arches */ + __u8 __pad[6]; +}; + +#define EPOLL_IOC_TYPE 0x8A +#define EPIOCSPARAMS _IOW(EPOLL_IOC_TYPE, 0x01, struct epoll_params) +#define EPIOCGPARAMS _IOR(EPOLL_IOC_TYPE, 0x02, struct epoll_params) + #endif /* _UAPI_LINUX_EVENTPOLL_H */