From patchwork Mon Feb 5 21:04:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 197070 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1179957dyb; Mon, 5 Feb 2024 14:16:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IFI/rlsU5pJcanAXgK5SIs8cN3j0Tsc/Nr1NKv9YdP57+qOqnv2+RPFPPkieGvFQzvLsmvf X-Received: by 2002:a05:6a20:394b:b0:19e:5e0c:3bb8 with SMTP id r11-20020a056a20394b00b0019e5e0c3bb8mr1590900pzg.7.1707171403978; Mon, 05 Feb 2024 14:16:43 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707171403; cv=pass; d=google.com; s=arc-20160816; b=fY3ZR5AtZgRzLYpKhwWCHtC28+gMPzqDybiyhVmuKqSGUTsVnUYptXYSem27L4jn2w SvAChBPjsm0p90qR2zMGrgWTb3OHWqGJhMpCf2R9qXuRWM+hcVQZq0WODuLfgCfMrFRo G/b0YNnsf08/HplqVZ4L33Aq8tCkmduqPIR8vESSwPFj9+b0aXcBTdNAd1W3/a6RVr4X O2ofg8YzCESGACuyesCNLD9RgT03MMKKlp61cp9Ry19ClhdukGAUVsKyTHb5HUccD0OE 6hCNKCWujAHjTAYVuGIsn7gphzBZSE38uWVnbK7bywvJTUi9tsppNB06AimP5F39LBvC rBuQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=//WWP3LIv7F69f9OvZE6kAJ/ncD/SOZi/tM1IcQuzWk=; fh=skA7SLVRv358uqRyuCHjBwVBIPhTdENtzypQpjF+Njc=; b=LjKW0FZZMdaRQp5M0IZbtwWXHk66fxzLN+/wl2OXwh6jZjpefABHyyvc4IrQ7i07JS vZzq/GNO+xVimEGIeUx6FaagGhjT2m1Xlg6nk9fAKxQHQYs363Gw7r0gZ2Bt6p/BSyuX nDjHxvuAXpg8xXctpUQC0QM3R9smfPnVTcb/pOFWXYebs695XdmkqgsvViJRF3PHImQx SxNT+ZvwW6yNe3+W3DAc10rMpbUo85IckIkyrWINiSAVMpFkHM9nq7fs5l7Y2TVwR1cd 3yQVfw7omXTtjymdDFQwjClRZDIwkOKeT9lNijy2R5V4hRWGERI5tGWTqD1VwuTE93xV jOig==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=Z9p1Typm; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53938-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53938-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCUyDZGd23EILsqsY4/rtw7HoaLys2agDU4y+96N4WK7f/aM7Jfq78OkblRhw7erNWXn1yqfO+95WVYfzCICeEioC/N3rA== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id j22-20020a632316000000b005be3683ec6asi529642pgj.184.2024.02.05.14.16.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 14:16:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-53938-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=Z9p1Typm; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53938-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53938-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id AF0F8B27105 for ; Mon, 5 Feb 2024 21:57:58 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8569813D4E6; Mon, 5 Feb 2024 21:05:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="Z9p1Typm" Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 99E1213B79A for ; Mon, 5 Feb 2024 21:05:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167105; cv=none; b=fEFdeyzY+35z7PIN5tHsElgJZpcEMMPlJdR7iYc/jSuEIYcTiV2eS4lECj6ZOO68EvCiwI5ruZGiztDO6f0ePHkQ5j2yup5S3QrC5jrszogEmvaqfK/90dyrFoqviVqjv/srbsP9sFop3sGLeOVm8x4u13/ZgJ4qDj8T1qLO9JY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167105; c=relaxed/simple; bh=gQDKvGU/ghxRMfcflvdwZ024OwvFboTrSVuhgGT5PSY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Pga4PgqKfbfcjuFNGAs0dIDFGqTiV0WyW0WKQ+Iei8No2rz6gsXsW8uIjeyr97/WdL7V47sG6jhVWnee2KHN3lM3+tcLkJwgV50RVRZOP34lJt8hTyAHcQMPWNAEwnluI3lWFkqIKWPmGxWDqte/r1sR4kQgUo+VGBlQ+ZwE+9A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=Z9p1Typm; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6e055baec89so162568b3a.1 for ; Mon, 05 Feb 2024 13:05:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707167102; x=1707771902; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=//WWP3LIv7F69f9OvZE6kAJ/ncD/SOZi/tM1IcQuzWk=; b=Z9p1TypmZBncIk3hMD3ssbrfbGNvV//XFL3azhO5lzVQ3E3tzoSmMHONBmlisJZYiG ZMZiiGkgGSxZJjSft0H4/VdeUKNTyPmplMCthd9vrMGOs2hNYy8x5/AiCHv0h+h3s5jp 7v9zjyEi+3j6DrJxYtAlokCirPCDmcZDz/T2o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707167102; x=1707771902; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=//WWP3LIv7F69f9OvZE6kAJ/ncD/SOZi/tM1IcQuzWk=; b=VfaqEuTtrURbFUFe+EkeLiC93KqatTc1OR1Z0kUOqqx1Hje5SKSyOgZIpIYNiNcywb ktwe3elzpz8bqjbv22c5aJBuj2aSzVxZOGAdSHc6T+VFlBoclYEeXLIInH9XzAqkDS46 xnuAzOkPygm9tCEoIyg9gxK9pCi/bEHQTDzD0FcPMjwu79FZ/259C2b1T3nLTLFOORDI V1JnFzgixQyvInKhaomwx4EAn7ffhumdFb2AxAf2/e4d5L2CrWzbK5r/xVT3nxNSKO+j PMCCooY5tcCbpzREJ6y0w6ujRV0NpzJTMcs71Ks0uD/0K8FVFULR9V8ME6GndlRzreXx RkYQ== X-Gm-Message-State: AOJu0YxyjcVhpSoudthnjHFhNPP5FnXeiNrBhYprQBVqPlz/BU1H8icx wwQqBjBQERmLghuDObuCX2n6XjHUGmQ6yOQLGvBBQY01V3M+r7iSzGNCLnzLeq5XFxfUuUXEVkU OA14LhaEva6mDQvfKQvqB/H/UL8aRXpCCIYmY+DB7dkGEnWDAQhEQMoY3Qt5q4p+mJbX8eWNj2P kL2WabVBXvsQU7P6zCLZs3apBzxprnTf2H1/poZ7QLoGDovw== X-Received: by 2002:aa7:88d0:0:b0:6e0:405f:ef8d with SMTP id k16-20020aa788d0000000b006e0405fef8dmr5450050pff.8.1707167101863; Mon, 05 Feb 2024 13:05:01 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCV0V7zW2Chf3BKlfYDuPfCvy127acHgLLl/mxXnRfIXqkHQCrQaQCPHdzazH8xfDSiXyKXxDAueYNvzI5Np8XwDPN9VFhyCsI/seiejAqnkBwWYlAIraIZTqvzT1jWdBJjgIYJg+S8qdc+J0padB7TRXlFeO4wU8oN/787VAdH9YX3HjMvC1iTU15fBaj57sOpTXPiLe0gaLgRk+Oyn0hCBygc8JUZPOQCQLCa58l9VRsWqlBWhy0ldc0/xQdKWLedO4A3GZgOAWY8FTZGYhutzcb++BPzwHk8tt509jQHGEA0E4smfPQBmWPK3b6zYUSjNCDkb7s/N0d/A5cTdU/6+/Zc84r4SFUUF7ey9P4dHQbE8oSxdo9/6grXe0IblW1hW28CRUI+KROaAQx2fKD0/BRNzFiShZRS6At7UG1y7RzCxXhEB00PPg4SkrBTf14nbukfvwi8OcCoP6cWk8gPxqPBdafgnzncvXql9bw48+qEi4/SC96wpgrucv6d6tr/9+lvtgLRAEVljlg7OP3N8l8kkGjbHSXTQQM2aSmzw8DrTZUMewHKF+EWZtdugy/VYyeJj7kJXH13ydUAmVmKgir3VjWEg4zW7FrKpSmujyLaJs5Iv9cLIZ4r7ccI= Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id p9-20020aa79e89000000b006e03efbcb3esm315750pfq.73.2024.02.05.13.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 13:05:01 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v6 1/4] eventpoll: support busy poll per epoll instance Date: Mon, 5 Feb 2024 21:04:46 +0000 Message-Id: <20240205210453.11301-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240205210453.11301-1-jdamato@fastly.com> References: <20240205210453.11301-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790098962281889241 X-GMAIL-MSGID: 1790098962281889241 Allow busy polling on a per-epoll context basis. The per-epoll context usec timeout value is preferred, but the pre-existing system wide sysctl value is still supported if it specified. Signed-off-by: Joe Damato --- fs/eventpoll.c | 49 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3534d36a1474..ce75189d46df 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -227,6 +227,8 @@ struct eventpoll { #ifdef CONFIG_NET_RX_BUSY_POLL /* used to track busy poll napi_id */ unsigned int napi_id; + /* busy poll timeout */ + u64 busy_poll_usecs; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -386,12 +388,44 @@ static inline int ep_events_available(struct eventpoll *ep) READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR; } +/** + * busy_loop_ep_timeout - check if busy poll has timed out. The timeout value + * from the epoll instance ep is preferred, but if it is not set fallback to + * the system-wide global via busy_loop_timeout. + * + * @start_time: The start time used to compute the remaining time until timeout. + * @ep: Pointer to the eventpoll context. + * + * Return: true if the timeout has expired, false otherwise. + */ +static inline bool busy_loop_ep_timeout(unsigned long start_time, struct eventpoll *ep) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + unsigned long bp_usec = READ_ONCE(ep->busy_poll_usecs); + + if (bp_usec) { + unsigned long end_time = start_time + bp_usec; + unsigned long now = busy_loop_current_time(); + + return time_after(now, end_time); + } else { + return busy_loop_timeout(start_time); + } +#endif + return true; +} + #ifdef CONFIG_NET_RX_BUSY_POLL +static bool ep_busy_loop_on(struct eventpoll *ep) +{ + return !!ep->busy_poll_usecs || net_busy_loop_on(); +} + static bool ep_busy_loop_end(void *p, unsigned long start_time) { struct eventpoll *ep = p; - return ep_events_available(ep) || busy_loop_timeout(start_time); + return ep_events_available(ep) || busy_loop_ep_timeout(start_time, ep); } /* @@ -404,7 +438,7 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); - if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { + if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, BUSY_POLL_BUDGET); if (ep_events_available(ep)) @@ -430,7 +464,8 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) struct socket *sock; struct sock *sk; - if (!net_busy_loop_on()) + ep = epi->ep; + if (!ep_busy_loop_on(ep)) return; sock = sock_from_file(epi->ffd.file); @@ -442,7 +477,6 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) return; napi_id = READ_ONCE(sk->sk_napi_id); - ep = epi->ep; /* Non-NAPI IDs can be rejected * or @@ -466,6 +500,10 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) { } +static inline bool ep_busy_loop_on(struct eventpoll *ep) +{ + return false; +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -2058,6 +2096,9 @@ static int do_epoll_create(int flags) error = PTR_ERR(file); goto out_free_fd; } +#ifdef CONFIG_NET_RX_BUSY_POLL + ep->busy_poll_usecs = 0; +#endif ep->file = file; fd_install(fd, file); return fd; From patchwork Mon Feb 5 21:04:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 197049 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1166361dyb; Mon, 5 Feb 2024 13:58:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IGYjS9RIfy/miGh9+hm5LyWtCchbMLe6bBPGaaGDwkDOzQ0FnS521oI6YeeQfipeX6rf4p8 X-Received: by 2002:a05:6808:318e:b0:3bd:cacd:332d with SMTP id cd14-20020a056808318e00b003bdcacd332dmr1638067oib.40.1707170302496; Mon, 05 Feb 2024 13:58:22 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707170302; cv=pass; d=google.com; s=arc-20160816; b=WaaiAdVCVC0YJhQ8zm9yGhWYJDq+PQ5HSRh9S1lMwmJ8RhOC9Kods8NLlMMIOJfguO eBijedP6UYiIBsBZjURr0nxeOgZVrtt2VYqDO2LOyawc7djcFzC1qYcs5+Tvv7+wD1KX uDas/+FgmHgQXrK6taNYp2fb8VZ0nFfG4CEdav6NwGgGlrO7dyD576MmaKrJy/5mnM9+ UFAAXon4kKVqQyhRb7bqsklPHahI182/x/NNmCHWInGz9W7zR5zr7vycRyNC1vi7bfXY o3Nxv3LcaVz9nAE8huSNpGQQLxsfh2KJZmo71+kAl8mCEVlA/lFbuzoxRs4noBQ974ro 5rXw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; fh=OePLFNFthtpgmToJAc4kqZFswzIinpZD7Lui3rwlTvA=; b=xyuIY+P8gok5Hj8J7xw/xjXVcwy5trsUxn6/XveVGNwqNO/NrOFvVlg3JXCkQjqAUs up9aVXmuow0TqCKMi+1TdtccmkL7qCRCUUTJlg/XNCmGenugvzNdZLA3nU338XxlKOXD lhnFz36NhiKu7AcFfqo+tBGWqb1QDW9Gd+VeJGfKijXcfofq3ORY8eetLws5z6MSmBW5 9Bh0bg5fJK5awmW0Xg4x+CwVGk0C3BBQ+Jwn7y9B8CnzNyOldsX8OkPn4/kZQk9KbsfN Cs7UD+KfYP7pfVyB5dRUhdGLCL69s4lT1pH4zkk1g0bUSUJ6TqAGQ5Foj2j2/30+mTFB WqVg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=fKEmRb0U; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53939-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53939-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCXEf1MmckjSQPxsOiYZTDwqpsh2c+a9e1YjtdFiw6rYle30+uM8QoGS/dE4tdKEr9qN0i8HFeAHHFTKPWi0obMuIDfQCw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id i11-20020a05620a144b00b007856007f7basi779361qkl.349.2024.02.05.13.58.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 13:58:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-53939-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=fKEmRb0U; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53939-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53939-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 2752A1C28209 for ; Mon, 5 Feb 2024 21:58:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 433B313DBB7; Mon, 5 Feb 2024 21:05:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="fKEmRb0U" Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ADBC913BE83 for ; Mon, 5 Feb 2024 21:05:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167107; cv=none; b=HdLHe3o0HS2dsxpUOe19DNMpLAFcMRX+5BWDjdHlnMB1XA5Nqcbb3X6s8BVTShQ2NDgwvGlqMZIAhWjvh1sg1arXNG6cciSSQtKMA2zWffRgERC9Xb7WSirl6Z+o0Fva72OPH+CFdLDKGm3VQj9zvtjpANIopcv0K/FkAvjK4a0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167107; c=relaxed/simple; bh=+NvwukKrlDHz6eylGdtOKZKD8hFQLLmD1KnjinUcTaA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eAZWl7/S2HdQPmsQ4EY91S+B26rOXp+dBoThpr8tVdHvZY0Njxhb524MrO1vKpJIn4dd8NQJ8gDDOseoZYDKNS7DbvmEtAh/bps/NyObe4iqCpXOp7FJdr+IYzAfAcVV71Yy5+fNoTN2Bz8JyzW6IkTDN6KcfTwgHO83dAQh5H0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=fKEmRb0U; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6da202aa138so3088696b3a.2 for ; Mon, 05 Feb 2024 13:05:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707167104; x=1707771904; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; b=fKEmRb0UzhtVp0JQ8NPE8Nu+HOAPjAG3hq3TZ59qH5BWuUlO7T57l30gAyZKt0I78N /KVCinUWTW1V1w/CPrNKeq6NjggUecwxLP4eHJnRO6Gx4oySKg+57HefVME0hGpHrSd1 eQ2aw0CRvdXMzq5GkAh1ok76X1L+KpApRN8Qs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707167104; x=1707771904; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; b=cyA+jsVy2ERxmm0jpVqK2bi9uskzmqoAYg8AkZsHVO6myTDt8EQ/fW5YcSHnnyX96T rVBCog7NcLZs9KyYgO1EG8AEFoc1UE1QCRDBSh2Nh+CMrXuIZzWYLKqXKWxVV9nSZO9D HhuFT0igrFHhdIfrABcMbiMrFfA4Z4oTatDm0/1FXghnKVoTJariKSVFWMrEqrPJDBOj g7l+1jyiz684z2RTFTubN4Zu+EOX9O/AYXUbvdxR/VJWrcMQRd+R3wm5vablph0I+mui AA/mUiS4hGgnV8ri8mA0c3S/gNuYfwYB5Vi41JYMfwpi1YG9rSd6pWSzJoQOCJhEML+D 64cw== X-Gm-Message-State: AOJu0YwIvafB2rzxQElAaR5f1dQkMQn7IZR1/Sm2DeSlKue99mm7xh8v cYGHbsf+L4yj4jJEU3gCXlE5FuLhJptIwRgMUXOM1GPe8BcPRR6J+fojcSbXpZDkcChZ3A9oZ7/ CIM7MHcCeM6DsB1W+uDnWt+jv94AsUrB+clFIIVRECd+1HFKaLk+UuCMu9RRUA6gXl11JF8fMTi PMAidyoKljg09oBjbRVHoZ+lL/LFn7emwVqMlR1GA/mscIxw== X-Received: by 2002:aa7:8a5a:0:b0:6e0:3789:1d22 with SMTP id n26-20020aa78a5a000000b006e037891d22mr745135pfa.0.1707167104127; Mon, 05 Feb 2024 13:05:04 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWPCDwj2puJ8K8VTtMqxLjenbVPTnlb5l/s9MSgpQcJSphuEfojDN2xQ8wceg7xxhwdKnqgzZXtjqGEqcSQsFx40TTvbDOLhtxqAEKF6qejocHfQY2QJIYVhoEjUonK+O/KLFB+NCqEynMVOXKCSgzUaaNqICkui8Dk0VLLDBToef3tPycqOHfdjnxJQDy4Zd4+pdVP64r1tXr9BYhfqgaKTdnG1n21Yioi1vEKrjFk3eK202MWrCGVEP8PxR2bwnt1KJvyTcL+0Q8+Y8DZxrd6q/3Uke4FjrpcFxN/NeZzHr2f1+HlZ2EAxYvLkSv/Rz92o2TCWNkMPgX1y+OLczHDhccHd40xIPOb92n7ESGN8HAW6KeDpkL+ITHQz0mKaRXAb199+dsfJoOnrLgchGTtQmudj9BvEjtySid47JSg2WY8bu7Cm8wxA985K15oylghCstmbjyvBufle8nb3PfLKYFJI6akjuxYH3xZopZ6/0iLUoy4zswlgCg7phxlLlJZdlSeiH04Y5r8s4e0RtTseBiQOzbh2/78nkbTElh9yhTwuGB6h38x7rluWpmKq305dmTbo4IpL8Lg7slKVW2WvqmIuOeHKbrQQSGbcX/LmHqyJ/M5LOhIGTykw8I= Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id p9-20020aa79e89000000b006e03efbcb3esm315750pfq.73.2024.02.05.13.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 13:05:03 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v6 2/4] eventpoll: Add per-epoll busy poll packet budget Date: Mon, 5 Feb 2024 21:04:47 +0000 Message-Id: <20240205210453.11301-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240205210453.11301-1-jdamato@fastly.com> References: <20240205210453.11301-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790097807124274943 X-GMAIL-MSGID: 1790097807124274943 When using epoll-based busy poll, the packet budget is hardcoded to BUSY_POLL_BUDGET (8). Users may desire larger busy poll budgets, which can potentially increase throughput when busy polling under high network load. Other busy poll methods allow setting the busy poll budget via SO_BUSY_POLL_BUDGET, but epoll-based busy polling uses a hardcoded value. Fix this edge case by adding support for a per-epoll context busy poll packet budget. If not specified, the default value (BUSY_POLL_BUDGET) is used. Signed-off-by: Joe Damato Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet --- fs/eventpoll.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ce75189d46df..3985434df527 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -229,6 +229,8 @@ struct eventpoll { unsigned int napi_id; /* busy poll timeout */ u64 busy_poll_usecs; + /* busy poll packet budget */ + u16 busy_poll_budget; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -437,10 +439,14 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); + u16 budget = READ_ONCE(ep->busy_poll_budget); + + if (!budget) + budget = BUSY_POLL_BUDGET; if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, - BUSY_POLL_BUDGET); + budget); if (ep_events_available(ep)) return true; /* @@ -2098,6 +2104,7 @@ static int do_epoll_create(int flags) } #ifdef CONFIG_NET_RX_BUSY_POLL ep->busy_poll_usecs = 0; + ep->busy_poll_budget = 0; #endif ep->file = file; fd_install(fd, file); From patchwork Mon Feb 5 21:04:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 197050 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1166434dyb; Mon, 5 Feb 2024 13:58:35 -0800 (PST) X-Google-Smtp-Source: AGHT+IFzO8Q45360ZMyDv6HPNU7WOMtcdUsUTuhNRZVRkb6QP2uOemulgmEnGUksVuS5RQQ+GCoo X-Received: by 2002:a05:620a:146b:b0:783:bd6c:90cd with SMTP id j11-20020a05620a146b00b00783bd6c90cdmr811017qkl.19.1707170315528; Mon, 05 Feb 2024 13:58:35 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707170315; cv=pass; d=google.com; s=arc-20160816; b=Vx7BOnRzMBzyabK54te/ed/LOobGDdAowSQQEGjoAOcs5S/Z+LeqsgY0pALcFImpt5 I2hgzK+pnOFXV9Z60HIVMSDGM77fFgBIzGgbDQKv+xXE36mV0qeVsyYV9GTo61HXbc0B L6YWH2x9RgElxYOfT5V5hlujNbqasKZiFnTRKh55/zgL7K/Oskju8KFdR5sAQPotIbuN +cL3Rx0Yvl7YRT/fwjSWbr12hAfwnHv0Kt8YR7kMiNi78xIQ5bQr/NTW3AyciCKYs6Ad BRqts+x9d/cA1+XJprVSFZaGwvmbn1w+7EobVOubwpU/R0AwqNSUln4/r9RCCztbVdRn swIw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=wBiMdRH4c/8PwxTtqzx2lNrurPu3dag4D7tWOSrqvQs=; fh=T67cKxdGrtbhyljyNNXHwgM/vhqlIgTFbwUqRP5DwKQ=; b=p/aqhxq7x4Tk8Vkd0sdxKMDfbcGcygQZLbqeFGWZ620lmBGU2KGDWp7oYFhL/6AiKz 7C/mpDtKNux3xqyC7WlwTcYD3iR+2+oYUj5Ryg1Gq8nYqqcszyo+99nuoSSs7bAd38N7 HKPdqeq1FlE6yM/7+OwfhUYnTGEMn1RxnBFr4cMpE39ivBCyZ1MtNPcjakM3T7+dbcGc fRmTgPYwBEvxKVX4YJnkSVvDy0O1TVHWNLjMJhF5ulq9m0vmMdNQ6rNl9ICNbpY90NSq WHkE+UXv8QkzAIyl71SRIj+ZnZwNB5mQpFteZ/lZ3DsT9WurGXsY9yj5DVfGg01QZZPu 6pEA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=dANtzwMy; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53940-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53940-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCWAuRtCzq9yK38lpU7pjZREEDme0E+nfg1hdaS716OU8rdT9vebh4rnRFhDmf1cL2B476CdzjNfWhz6Om3ppq4QQeMNHA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id m20-20020ae9e014000000b007847b274ea0si798932qkk.298.2024.02.05.13.58.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 13:58:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-53940-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=dANtzwMy; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53940-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53940-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 45F951C28306 for ; Mon, 5 Feb 2024 21:58:35 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 52E9113E221; Mon, 5 Feb 2024 21:05:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="dANtzwMy" Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0212E13C1FC for ; Mon, 5 Feb 2024 21:05:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167109; cv=none; b=EKPfj1GFWqTjbGOIbS3Uf7n6PMB+jlNaGAbZ/2ePiFQjzMsZdTTPNSNgcy/67+dB03jxSZ6uI3A4KrwcHtJqrNVM4lkvOf15t1mDSNM2o9axXykNdN6aeH8AAyyukpVTy3VMzNZ3ce96eTCTANri+NiV8PphRlDau+B6CU5hXfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167109; c=relaxed/simple; bh=NOwj8c7OsDBEwdIAq+Wisv12nzJJCvXJamRUTIa+OYY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Cmy023u9IHawuxjB/ihHjwS8RzU//R6HPjnLFD6fO+saSWB1S5n2dKPuLKRKlw0aWJ4UIPkGBiZYEPJLS4jXC9DxfA55t90GN9SUNhWCPdbBW6X4nw5uhUVPV8IqMlaEdS9Be60oCN+xIU0Urkugr76fGE5cFbPrnu+nHUCAfPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=dANtzwMy; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6da9c834646so3920731b3a.3 for ; Mon, 05 Feb 2024 13:05:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707167106; x=1707771906; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wBiMdRH4c/8PwxTtqzx2lNrurPu3dag4D7tWOSrqvQs=; b=dANtzwMyTiUKYEANH6iYHxN5E0jb9AQ5LjPkqx9tb/WwixpwClUw6VoTkDuJaNBpDV 2fpfCSzWssPdIfwKbcRhptpfxiPDz+meBnzSmtjl3ElbNumQQgI0iVh3baH2J8UpKDsG vRZquuT9hL2IYDkVA5UUvSxNb/gx7ocksAWcQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707167106; x=1707771906; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wBiMdRH4c/8PwxTtqzx2lNrurPu3dag4D7tWOSrqvQs=; b=ZdSHeLbp9ryPKbjiF47OeEuioEFYDOVsEIYQFDO+SC9/CnZSOK8h07FP0lgf1zpBit pNrRvTj1X4ias7NUY0n3UMFBQAhKi/sfHbr+29kIhVnenosada/qSW9qKm/6Rk9T4h1Q E9rt28jRGKqwlWtM8WwVlB7fg2LdDZCRtea0ViSvoQ7yMnnZxKHGB1pwSs5qnWATjZsM dyqg9Q8ReDTqGIoD5ckkPJWmq29kw0NlAqBqCMdueemHhEy5maq7nhD0Jy3IP0m4n3TI qn3QmVsduJnStcEV8ZeyBVJsakwIxJOl7lYIYg3S6UHvdyPpdTx4ntWRxDi4n06h3g4/ M9cQ== X-Gm-Message-State: AOJu0YzbMGKSSLXsOdtFzhUXGyeo7Jrc/8lYWPI7LE5PqsvvUnuvcfkr zHqzPbCd+k9I5+xk3iCF3qT0knTDPWYfGtK3IQ85oaeXtGchFAhw+yGP8pBn7zQCHHWtdPVaVoC 09pCQyI3UVuqy3lBTouAOmp/RCqgE/Dy2reArBwGJ4bqwoh76W3f/XYtGcJa3qSmxVAqf22i8dZ eCcfF9vsmfNKP4dMP+f1aomuAj8AX28QuOR53N6CYq5KuUZg== X-Received: by 2002:aa7:8594:0:b0:6e0:540e:bf43 with SMTP id w20-20020aa78594000000b006e0540ebf43mr758751pfn.33.1707167106379; Mon, 05 Feb 2024 13:05:06 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXzF9zYuQBn2HijFNy9BeuTMFAR7WgvA6hcwexP03WGZparkd2SO67O5nwT73blaBRwmKgp9CMzTYv1RygO4ui1u741WPf/e+T8MWehMrXCuGivTWb0RItzr6I3uBklkx2WDa5M1f8InQjbB2NxtMuR/0qvS+YCiH8rAEx1mdxFNwpka7oW8vepfEEOvlvHuigNK6SnBz2PjWghEer5NuRNd74fpkguMCTCUemQP+vlfKX3Wb6i1kCXSWmiSaBGxNfhTG51ZnAif5+Y83TY0QidlK1NsBlo+HM07Yqfs/m9X0NfOZN8EhuUEk2zy0qs/kf7hRZPuvz9KK2nFOOqgedzpFLcv3DjnX0GTKQE7sxy1hjVgLN2Q6Wag8adDx141gXXDTvNJtyFr83GIe8msd1a/jqv0QOJRnxkhqxyTAysvNbqAaT3EV8qBMNna0xAOtkQTubWIpVZlS+0mTWExW4JJONa96R1MQuMSTCHlIiJkIcVUFa9ZaA3zCkohSw02Oa0RTdlRiVz9Vo/dXNQj+65IdwTBjPFzsWetoXumMARjYeTMlrbi7I87QyKnKGXe4lcIka43IQtwGe27s0jT8ujI1mLYiQmrMVIvYoBIZTL18fjiN92dXTVzkLGl/M= Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id p9-20020aa79e89000000b006e03efbcb3esm315750pfq.73.2024.02.05.13.05.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 13:05:05 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v6 3/4] eventpoll: Add per-epoll prefer busy poll option Date: Mon, 5 Feb 2024 21:04:48 +0000 Message-Id: <20240205210453.11301-4-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240205210453.11301-1-jdamato@fastly.com> References: <20240205210453.11301-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790097820706117217 X-GMAIL-MSGID: 1790097820706117217 When using epoll-based busy poll, the prefer_busy_poll option is hardcoded to false. Users may want to enable prefer_busy_poll to be used in conjunction with gro_flush_timeout and defer_hard_irqs_count to keep device IRQs masked. Other busy poll methods allow enabling or disabling prefer busy poll via SO_PREFER_BUSY_POLL, but epoll-based busy polling uses a hardcoded value. Fix this edge case by adding support for a per-epoll context prefer_busy_poll option. The default is false, as it was hardcoded before this change. Signed-off-by: Joe Damato Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet --- fs/eventpoll.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3985434df527..a69ee11682b9 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -231,6 +231,7 @@ struct eventpoll { u64 busy_poll_usecs; /* busy poll packet budget */ u16 busy_poll_budget; + bool prefer_busy_poll; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -440,13 +441,14 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); u16 budget = READ_ONCE(ep->busy_poll_budget); + bool prefer_busy_poll = READ_ONCE(ep->prefer_busy_poll); if (!budget) budget = BUSY_POLL_BUDGET; if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { - napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, - budget); + napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, + ep, prefer_busy_poll, budget); if (ep_events_available(ep)) return true; /* @@ -2105,6 +2107,7 @@ static int do_epoll_create(int flags) #ifdef CONFIG_NET_RX_BUSY_POLL ep->busy_poll_usecs = 0; ep->busy_poll_budget = 0; + ep->prefer_busy_poll = false; #endif ep->file = file; fd_install(fd, file); From patchwork Mon Feb 5 21:04:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 197071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:168b:b0:106:860b:bbdd with SMTP id ma11csp1180598dyb; Mon, 5 Feb 2024 14:17:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IGi6yB9vfO+9Vtw+n+LtbyKEPl2njND3w7AdO/M5sx7d96Cnf1fKM6Zn4TsnR4Q8ecQb4s0 X-Received: by 2002:a05:6a00:3903:b0:6dd:816a:94db with SMTP id fh3-20020a056a00390300b006dd816a94dbmr957222pfb.27.1707171465023; Mon, 05 Feb 2024 14:17:45 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707171465; cv=pass; d=google.com; s=arc-20160816; b=FnL9DprjGnOg9foX8+iFcjNrVEEO8JyQ9CMuJx4KQj1JSvT7DMBqfkI9FPVscaGg7z jmYoZ9cdHsVsBC9sEvcZvBmR7MJn2XtsgPHT8NByDM9lYe4obugMexkCLiBxZDF8Cxsr o98w5Yla9fBcU+m62t9llN9RzM0ZXUfXqPHrPvLuyZi5ppwcPye8WAqmlceHTtWvW/Ub +BsM0ajpamdkfOSleYYPD/kuGeLOomcGBl3eqDdXV0CM5cpF82G68QIKkZ4Js6D6H/mn RtprwMi+MYFCUiJYTCIqez6hIu9OR+S7aKkS3XiQaZzLwTK8CP0ifA9j9QCbSjB3jQo9 v4LA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=nOYgxaVhBBqkYTQLahlIJIPu6cEXeF8iHyq//4nC0Dw=; fh=OXxtco1ikiOIY0M2iaY2Vh9InZnaKLyg/nY481S5d1A=; b=O/DYdJUOWYf8U+ECmW8WFubPEAN7zAc5DFXnfhv3C51js7rGwz0DjNGh7q59XX0M6R 5pA9qw0mjuWvb+u4TrnDNLIJxWfAtXatJA7dLnMulkZ3hxXD/SDxy+GGlp6XnioLPXxQ OVSQzusOStfi7hhbJoReORsFBp6ebGuYbD98uKPhX0eQJOBNqr3ccunR+NCPsqGElMlO qV6U5RNOk7oNToTt++KRruMvlWcF/UMFe4aJXJ2S9tmHno1wdM17EU9Aiqvhy+Nh9ylS 1j90kjQ9HU7jKh0aBsjg5HuKTQJx/P8iJ3YWD3Do3o9+MCVpDOHwl02/ibXhQuQm5h10 uYZw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=Lp9+rMLu; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53941-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53941-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCUTo8GPGPTMuE2L08kCgryBwCJa26w1HflOawZCqTDSvGQsjPcBSXImGGGxjVz9682JodKfjjmWxVw7iRuyB7ifg4nfGw== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id s17-20020a635251000000b005cdff5fb163si522133pgl.333.2024.02.05.14.17.44 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 14:17:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-53941-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=Lp9+rMLu; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-53941-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-53941-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 93A4FB278CA for ; Mon, 5 Feb 2024 21:59:03 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F0E1E140787; Mon, 5 Feb 2024 21:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="Lp9+rMLu" Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D45BD13EFE1 for ; Mon, 5 Feb 2024 21:05:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167116; cv=none; b=S0oKkPTkr3BbzPUqo6+bSMNmxXj7f7nfDmui8wUOZJzt7m/lO/LIZ+dWBmk14rS9yCNa+CckQXnLFR1F2yAoIVPxZ5Fb2dKvrsnKxRN7gPItnjPrcAYCpmhgTNyozu5n9LKvNb9a3JmgkIjnfIYZTv7NBCRmaM0bAfL2pgzpQKc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707167116; c=relaxed/simple; bh=heAQsVdQyZX1S3QWfoEAK9WpamlcuDvtpD/1aq7LXwc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hZreRci86awYPHJjNiGS4pS/s5Zv9R0o1FuR3fvnY/6gw4LL9b8Pg8zysEYukKKs2ADbYDWPbKqucl1Kt5faA+12yYi/WBrUuT4YSfGqmInmxdr8+54Cx0StYhu+WU+VY58WDVrAEeO2cPiuvOZWAP+IX1zw8AQrzoxWzulGKZ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=Lp9+rMLu; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6e053b272b0so353660b3a.1 for ; Mon, 05 Feb 2024 13:05:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707167113; x=1707771913; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nOYgxaVhBBqkYTQLahlIJIPu6cEXeF8iHyq//4nC0Dw=; b=Lp9+rMLu+2bmP1Trs+oEouQO4pUbXiU4wDoVDJVjsMrrVe0igbsYsgU25t0NTNkLl5 E4aXA+HvovmeQ/62QJYvtrRsjanY1kEAPTWzHnHeeZZ+VCqzRZ+iqf9FEqI862zDLLS0 FwGGrzFKGIgdo21lpWE8VviJIShiJMpLnT7Ck= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707167113; x=1707771913; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nOYgxaVhBBqkYTQLahlIJIPu6cEXeF8iHyq//4nC0Dw=; b=RSlEW+W70HqsfTjOjro74inFHpWxlXnCHCPxp+C9pj1/CpnetqC0ooM1O2dyd+lHkX B0JMwe/dqij9fOKMLDzLRhf8JxppHFk+euw1srp+GuleFaOxXtku6CBub25VD0apobUv nL3AIHIlLNbQCiA2i8fmNGYszEoS1JNX2btdYmWpAMhum10DFXRXJhBV5iOe0ehZ8LDf zhKBQLW1L3NwNKqzr+b940gBtZce8pjUFtGcpSuAx5sPMYXcy812TnjplD6tOH0tP7R7 jPcMr5ds4jopgxRPP/9tvCkEls3vKmRD2BVwlEsSqiUo7ngeeFkco5LFLte99LaPkQuM A+gg== X-Gm-Message-State: AOJu0Yyl/g1lMks43ZwMC0/tIn2orvpbpc0TNqUBnmOLLLYeZhNc3lsi PlilI9LMAJiL01UZK7NSfU0Mru+7D3zUbfEo7dpCSTFnXN6FN9dC/VcMuzWFuvDn30apySAffi7 w83k31EBEJ0aTC0YUvRLWTsbbhQjezND3olvC6szPaaUjEWqj6haueszlt9IFMb6tNpkhuQWOaz YAX5lOJ94IB9I5tiV0IqFEblhzsGSHSjcm3kmNgZMGBF4V9Q== X-Received: by 2002:a05:6a00:1882:b0:6e0:4797:801e with SMTP id x2-20020a056a00188200b006e04797801emr951487pfh.12.1707167112887; Mon, 05 Feb 2024 13:05:12 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUXgARtOfY85fQ804fqgf9R3N0xD8xUK5NSW8I+9Y6M/3k+8ChC8R5MV2ubEiCHQXgUGHWFilMag26kloIdmvbD9DNBx86K/gl8d1bft6Nb583fIfQjKVW9R0y8f4J5+KhKgbtAUPhyZIRX/1T4HLEMMMKztQDj+Jd+ZiB84H3wbBaEgPrATM+47H8g0lag3VMroI0ZMIPKwrBkHMoh0qSU50c6/HfV8jfHaSK17tAcl0j6cPGL0t88IJlsr3+pjIv7qqdgPZ8IYpJrsu4POxQOsjEQOXt3GhgI5ovaLB2mK6DcEgkasP2zUj9CzTc6HsYuSnAvJ79QPDLOEt0u0JGm8YykxWinu8pHBBOqQ1k6Je+GQ4gLzc4yDwbmQX+14bUDRL16T/G+FIrsNw2Bv9JagyHbgTIrCRUvRG0NFHDzPZXPql61g2rQq8qVEdn/2HDRcO8y/2h9JFeseS0wbxPd6iHdco0jfe23hulnmdJcZEtw9aflDMfiNG00RXeaBvH/L/2E2nbzVn+s11S5vLOeE64mMY52ggp9vnPCWds0Kv425amQm3PMKwrbDVL1phxyhcOL6bv031sS04NBVrP3UME1/r4kL4XyZnQiK2+xjmhIvFdia4NwzMLjLcSfpboCdFCoYM4ChPQTdguCyaS+TItcwazJryiKCasKM8QLTBqnMutci0DaFPOlb0WkFfYVbdzs5dF2cIyA4ZEKAmfOFfrFGkTopmSxENnHPZxB3VX7aQ7FzFtY8/mBFipAq1mbVmc1CuHIR0xBBz88Dn8lDDh1D22FmkyGsE8U6ncOXbedp5aqBjkP0Ub9Ah+Q2jttXZ0l4F2G2UcRg8oeT5Z1DCkswlfqwWGpCK/e+YqzG1PCOEyVwtBTwCOZ5SkZ4TYiyMwtDNk7wKmQNNnO6We43Lf8Tujv/1xlYtjhQ05KMHdCty+C7O0cVdeXV5RaG2rAMK 0KMRCGzZ0YnIga05VUKAmY/H8enp6rxbu6VurdaXqikHPhoSOfDzWKQH2qdc8xi1WAiw== Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id p9-20020aa79e89000000b006e03efbcb3esm315750pfq.73.2024.02.05.13.05.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 13:05:12 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Jonathan Corbet , Alexander Viro , Jan Kara , Nathan Lynch , Michael Ellerman , Greg Kroah-Hartman , Namjae Jeon , Steve French , Thomas Zimmermann , Jiri Slaby , Julien Panis , Andrew Waterman , Palmer Dabbelt , Albert Ou , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v6 4/4] eventpoll: Add epoll ioctl for epoll_params Date: Mon, 5 Feb 2024 21:04:49 +0000 Message-Id: <20240205210453.11301-5-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240205210453.11301-1-jdamato@fastly.com> References: <20240205210453.11301-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790099025913053053 X-GMAIL-MSGID: 1790099025913053053 Add an ioctl for getting and setting epoll_params. User programs can use this ioctl to get and set the busy poll usec time, packet budget, and prefer busy poll params for a specific epoll context. Parameters are limited: - busy_poll_usecs is limited to <= u32_max - busy_poll_budget is limited to <= NAPI_POLL_WEIGHT by unprivileged users (!capable(CAP_NET_ADMIN)) - prefer_busy_poll must be 0 or 1 - __pad must be 0 Signed-off-by: Joe Damato Reviewed-by: Jiri Slaby --- .../userspace-api/ioctl/ioctl-number.rst | 1 + fs/eventpoll.c | 73 +++++++++++++++++++ include/uapi/linux/eventpoll.h | 13 ++++ 3 files changed, 87 insertions(+) diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 457e16f06e04..b33918232f78 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -309,6 +309,7 @@ Code Seq# Include File Comments 0x89 0B-DF linux/sockios.h 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range 0x89 F0-FF linux/sockios.h SIOCDEVPRIVATE range +0x8A 00-1F linux/eventpoll.h 0x8B all linux/wireless.h 0x8C 00-3F WiNRADiO driver diff --git a/fs/eventpoll.c b/fs/eventpoll.c index a69ee11682b9..8eb4ea2557af 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -37,6 +37,7 @@ #include #include #include +#include #include /* @@ -497,6 +498,50 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) ep->napi_id = napi_id; } +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct eventpoll *ep; + struct epoll_params epoll_params; + void __user *uarg = (void __user *) arg; + + ep = file->private_data; + + switch (cmd) { + case EPIOCSPARAMS: + if (copy_from_user(&epoll_params, uarg, sizeof(epoll_params))) + return -EFAULT; + + if (memchr_inv(epoll_params.__pad, 0, sizeof(epoll_params.__pad))) + return -EINVAL; + + if (epoll_params.busy_poll_usecs > U32_MAX) + return -EINVAL; + + if (epoll_params.prefer_busy_poll > 1) + return -EINVAL; + + if (epoll_params.busy_poll_budget > NAPI_POLL_WEIGHT && + !capable(CAP_NET_ADMIN)) + return -EPERM; + + ep->busy_poll_usecs = epoll_params.busy_poll_usecs; + ep->busy_poll_budget = epoll_params.busy_poll_budget; + ep->prefer_busy_poll = !!epoll_params.prefer_busy_poll; + return 0; + case EPIOCGPARAMS: + memset(&epoll_params, 0, sizeof(epoll_params)); + epoll_params.busy_poll_usecs = ep->busy_poll_usecs; + epoll_params.busy_poll_budget = ep->busy_poll_budget; + epoll_params.prefer_busy_poll = ep->prefer_busy_poll; + if (copy_to_user(uarg, &epoll_params, sizeof(epoll_params))) + return -EFAULT; + return 0; + default: + return -ENOIOCTLCMD; + } +} + #else static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) @@ -512,6 +557,12 @@ static inline bool ep_busy_loop_on(struct eventpoll *ep) { return false; } + +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return -EOPNOTSUPP; +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -871,6 +922,26 @@ static void ep_clear_and_put(struct eventpoll *ep) ep_free(ep); } +static long ep_eventpoll_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + int ret; + + if (!is_file_epoll(file)) + return -EINVAL; + + switch (cmd) { + case EPIOCSPARAMS: + case EPIOCGPARAMS: + ret = ep_eventpoll_bp_ioctl(file, cmd, arg); + break; + default: + ret = -EINVAL; + break; + } + + return ret; +} + static int ep_eventpoll_release(struct inode *inode, struct file *file) { struct eventpoll *ep = file->private_data; @@ -977,6 +1048,8 @@ static const struct file_operations eventpoll_fops = { .release = ep_eventpoll_release, .poll = ep_eventpoll_poll, .llseek = noop_llseek, + .unlocked_ioctl = ep_eventpoll_ioctl, + .compat_ioctl = compat_ptr_ioctl, }; /* diff --git a/include/uapi/linux/eventpoll.h b/include/uapi/linux/eventpoll.h index cfbcc4cc49ac..36a002660955 100644 --- a/include/uapi/linux/eventpoll.h +++ b/include/uapi/linux/eventpoll.h @@ -85,4 +85,17 @@ struct epoll_event { __u64 data; } EPOLL_PACKED; +struct epoll_params { + __aligned_u64 busy_poll_usecs; + __u16 busy_poll_budget; + __u8 prefer_busy_poll; + + /* pad the struct to a multiple of 64bits for alignment on all arches */ + __u8 __pad[5]; +}; + +#define EPOLL_IOC_TYPE 0x8A +#define EPIOCSPARAMS _IOW(EPOLL_IOC_TYPE, 0x01, struct epoll_params) +#define EPIOCGPARAMS _IOR(EPOLL_IOC_TYPE, 0x02, struct epoll_params) + #endif /* _UAPI_LINUX_EVENTPOLL_H */