From patchwork Fri Feb 9 21:15:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 199113 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1138020dyd; Fri, 9 Feb 2024 13:22:09 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVzkicoi0sikddLTgZZdBoFbGFjjA5UkTTfijx0wy/5CJPzgb8kqxv35QjgOAiCxYUPyYeAtR+PpSkESfIHoyE4wp+5Xg== X-Google-Smtp-Source: AGHT+IFye/TsDOjPkzWKP+2raVdenorZUj74HUG6+dspddcw4CuKPB35O+ASdpnQzPde9MC5bwsr X-Received: by 2002:a17:90a:fd0d:b0:290:bd97:aa6 with SMTP id cv13-20020a17090afd0d00b00290bd970aa6mr226561pjb.48.1707513729337; Fri, 09 Feb 2024 13:22:09 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707513729; cv=pass; d=google.com; s=arc-20160816; b=AGPlFnKbdtKm9TFVG8FXKlsZtiLTwpoWYP+ul7jXNkvb0/9gFY+xgykM+/b1dqjlHY gqxSba8vN3rxjF0WJ6lLF5sfg7wtJhfKUv8IoVRVbby68xmUzRufSz3atRsA2Soqnedx 47Jl2jLQky4xOXgsqvJe/wXQBdVE3HOFZGVz6yhc54Lk/WDp+FNlfLQm85FJao8X1Keo vFs21C6ifrEPoqdDwntsbfFJgP0rHc0SAR+hGP/tT0eLtxLnARQE4MaNzQF22y4Ivlm/ G+SpjI060Kz1BmkCB2dvcW5p6WpjsEwf9o/NfmUcGdzURyq0ePgMQjNQgpP1Om0o4/LT ytkQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=CnWe3jITTYQxWO7n4MTzpSC6BAe+szTuzlwikAcSvaM=; fh=8bL2Cfizv7A2oTtDDf7EhrHnT/hMCFvnth6W/aXTFjM=; b=BB5i7RQTXo5HMou/dX6mhITgpBVaEp02DA2CySv+Tat7iXmTOtMPpfyHNmF/cLIZrj trc/q+daXkGWSSPv113W0S9PBgQGMw2vdNyU3BnLijkOtplQ84y2N7P1z396aUWTgXWd +zFOrkesvkx3frec8AOsuDZuma+1UMq450sOvgSpZMH57w1WZLsnXYU7KOzRPeFCr4cE XKC3QKK/qmKo9OZqOxl3I52ss+ttpsENGv6GC72ogHZWrg4DQUVoqbJ8diYq2ipSz3Ad xvKL8GraDZ54u5WYsOv5MgazCmjUZfuBAOc1iOIkYCUCd3Pe/uNi1sckKvMSpNxhSzXl SHIA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=tYT0zxyG; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60019-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60019-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=2; AJvYcCWXhNYXVcupqiABEf0xfrkeA9p3S6R9xjK+rlLECtl/YqvZG/PwgqDR66US8MVxhFNxlxMyrZHpBlcG5uEY51Xtrkgi4A== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id kb18-20020a17090ae7d200b002943cd1b20asi2275263pjb.93.2024.02.09.13.22.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:22:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60019-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=tYT0zxyG; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60019-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60019-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id E8D54288A78 for ; Fri, 9 Feb 2024 21:21:40 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 4769D3A8C0; Fri, 9 Feb 2024 21:15:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="tYT0zxyG" Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E1E539851 for ; Fri, 9 Feb 2024 21:15:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513348; cv=none; b=Q5kELpEcvnzkNsIiEjzZKA8b4oJUyiMXd6d1FONQsj8vDtPqlR9d+OxeD4ByNxUuipl5/HxtyQd3ncseeZlYhjb/NA68p8kwMw9Zh1Tm3BMMd5VTSXZuJbQRiXhLhjlij3v6roAxfa1V/o25SHo9+tstv+GxZXExKndlHZ+7aMM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513348; c=relaxed/simple; bh=HZCLTuMgp/ckUlZ/e4J3shoTAMuEzpAwMA7BLa/9RiE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CuWXh5r3h/rqBIkJ7RbSzJHf66dJNkBN/25ephAkLDLN8h4pM74Owy70+VJSrJwiJxIA7wPHk0gG48LY8wGn/q91EPEDZpIShbLxD/2RdT4XxybG/aoAaDwpUZZDkO/+p/DwGV9bhtGGNAvnGYQknNKdCRWqNrC+22oTW3868nA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=tYT0zxyG; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-6d9f94b9186so1227873b3a.0 for ; Fri, 09 Feb 2024 13:15:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707513346; x=1708118146; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CnWe3jITTYQxWO7n4MTzpSC6BAe+szTuzlwikAcSvaM=; b=tYT0zxyGQALoutYHNBNhO5y5dZYLx2lgv1nqXnhIWkc76+UlnmfM6Fw9th4ZlAfatK xjzvDb4YzqRsFZvnvGncRJIIRKiawlyOdOoAwQ36Q4/yDxaHK0fgoUTreOygNbeLrH8q LArkw7y/XuC5RNRUWgP6M94Yg6jVPxr/lsCx0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707513346; x=1708118146; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CnWe3jITTYQxWO7n4MTzpSC6BAe+szTuzlwikAcSvaM=; b=ENHHvLD3V2IZkyOweq+tTw4xU2iAtq9fLpITrhQpAl6RABSoixq7+LdETz5pGv+Cvz Mv1kam+JUYmXbm+TKDm2scxfmMi7gXYL8N5gczq4l7UygZeIBGz7zWvPXObFWfDJctB8 4fsuHdL5s3BMBjhMV/tfE5+qTQRLhRKn4lXdFQldMsYiWlfJK4vQ8vA6ePovHAn0uAUX xKembSofZAXJzzQ/3zirzCMh0Rj7ghC+uG/omtgdPjsMEzRcGlJBuAt8fkrCJ189Scy/ 3FUlhKvHZhgaxZkJkD2CiyCgrWQKwdQ3vjkc0IolVhEUkHFQyyqdeOWrApXbGI3XX4VG 7sFg== X-Gm-Message-State: AOJu0Yy8VuJZRbT1xswY4n8jXvyU3KyFFSK3FjEmwHcGno/jP96+I5Ie Orml1OvRjiYk6ObHpAuWWlWh9chId71fZ6EDDzSDYw8CM/CNIdPpO1nZDL3yS8TFRh+hf4D5xHh j37/TT3fdQXx2Om+udNvBiatnmyIqHE+bSicq/TMD1L7O2by456/s0eRXfio2pgOhPP/O5VLkM7 91eSTEqy1EJeez7VSinkDfsggJSuK7O8TUdhU5LR0ZHSdl7g== X-Received: by 2002:aa7:8d0a:0:b0:6de:1e25:48e8 with SMTP id j10-20020aa78d0a000000b006de1e2548e8mr469606pfe.14.1707513345615; Fri, 09 Feb 2024 13:15:45 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCUZBSIfqNr68Owgav3RM3Xfph+ZAAxDrRGFRBDiw8MjSKXZJGtj8MF123MizOhU5Ok4S26YLAEomcDdR/fLxX0F+Z2siGroZptJ3px9fyzb50dnPIweDBnDdI5dkSY206V4FwgAihNo45vggdOuRM1dtVZtH4SO/wwTOsaGS7bhkx27e6C7Qjcx6+uF3eHNAkovl5ar26Cwo+E00/yAdpta9p6ILwwBfoC5Kd92oYw7KteYRqTngPeaTMRvPKExkG+apB00MviiOYjI1VOA6Kdz4c7WQv0c6eiwjQEqj1QZesDy/ASWH0as8V4jG9pxu1pXnJIazZwnYk1YoebWQjA/fzsBfJbMXYLNDBGgVAJ/seeLAtx+q6GztNJuFQ5aLv+4mjQ80NfN53enwBjgPQIAtoR6NVoyvKjdS+cQ1Ca64JGBZXvUlgx/YsF8nMIicKtIY6wjmxQaDG8wR/3LUP0xT4evp7XTND4o9KuqQv2p2OXrfX3bbPoOj480fWHEZBkHI+rMkCdp6uzEkJ5eTr2WXt0Jup+sOHeDBBbSO6cHWSOhuhILEjJ656tSmWaP0acDFOKHFEDyQCgTLl92MAuGKRA2l8GzcYn/pdICo1TXSWeAn5kW69X7xLF8PSw= Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id x23-20020aa79197000000b006e05c801748sm969629pfa.199.2024.02.09.13.15.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:15:45 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v7 1/4] eventpoll: support busy poll per epoll instance Date: Fri, 9 Feb 2024 21:15:21 +0000 Message-Id: <20240209211528.51234-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240209211528.51234-1-jdamato@fastly.com> References: <20240209211528.51234-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790457916315875828 X-GMAIL-MSGID: 1790457916315875828 Allow busy polling on a per-epoll context basis. The per-epoll context usec timeout value is preferred, but the pre-existing system wide sysctl value is still supported if it specified. busy_poll_usecs is a u32, but in a follow up patch the ioctl provided to the user only allows setting a value from 0 to S32_MAX. Signed-off-by: Joe Damato Acked-by: Stanislav Fomichev Reviewed-by: Eric Dumazet --- fs/eventpoll.c | 44 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3534d36a1474..401f865eced9 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -227,6 +227,8 @@ struct eventpoll { #ifdef CONFIG_NET_RX_BUSY_POLL /* used to track busy poll napi_id */ unsigned int napi_id; + /* busy poll timeout */ + u32 busy_poll_usecs; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -387,11 +389,41 @@ static inline int ep_events_available(struct eventpoll *ep) } #ifdef CONFIG_NET_RX_BUSY_POLL +/** + * busy_loop_ep_timeout - check if busy poll has timed out. The timeout value + * from the epoll instance ep is preferred, but if it is not set fallback to + * the system-wide global via busy_loop_timeout. + * + * @start_time: The start time used to compute the remaining time until timeout. + * @ep: Pointer to the eventpoll context. + * + * Return: true if the timeout has expired, false otherwise. + */ +static bool busy_loop_ep_timeout(unsigned long start_time, + struct eventpoll *ep) +{ + unsigned long bp_usec = READ_ONCE(ep->busy_poll_usecs); + + if (bp_usec) { + unsigned long end_time = start_time + bp_usec; + unsigned long now = busy_loop_current_time(); + + return time_after(now, end_time); + } else { + return busy_loop_timeout(start_time); + } +} + +static bool ep_busy_loop_on(struct eventpoll *ep) +{ + return !!ep->busy_poll_usecs || net_busy_loop_on(); +} + static bool ep_busy_loop_end(void *p, unsigned long start_time) { struct eventpoll *ep = p; - return ep_events_available(ep) || busy_loop_timeout(start_time); + return ep_events_available(ep) || busy_loop_ep_timeout(start_time, ep); } /* @@ -404,7 +436,7 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); - if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { + if (napi_id >= MIN_NAPI_ID && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, BUSY_POLL_BUDGET); if (ep_events_available(ep)) @@ -425,12 +457,12 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) */ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) { - struct eventpoll *ep; + struct eventpoll *ep = epi->ep; unsigned int napi_id; struct socket *sock; struct sock *sk; - if (!net_busy_loop_on()) + if (!ep_busy_loop_on(ep)) return; sock = sock_from_file(epi->ffd.file); @@ -442,7 +474,6 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) return; napi_id = READ_ONCE(sk->sk_napi_id); - ep = epi->ep; /* Non-NAPI IDs can be rejected * or @@ -2058,6 +2089,9 @@ static int do_epoll_create(int flags) error = PTR_ERR(file); goto out_free_fd; } +#ifdef CONFIG_NET_RX_BUSY_POLL + ep->busy_poll_usecs = 0; +#endif ep->file = file; fd_install(fd, file); return fd; From patchwork Fri Feb 9 21:15:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 199114 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1138158dyd; Fri, 9 Feb 2024 13:22:29 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCXzK13vreRMnyCJrtZQcJBvTy2v7+p9l/GWcnl4YZYLcxzrBSlJgEn2lngbZfVZR5aUSAfMi5bLN8zwhj8gg1bxKmYKcQ== X-Google-Smtp-Source: AGHT+IGZOAbJn8/jimLsTmP7ScRtzbv5OzTo7pP5W2odH24JIJ6Zv1VJsDYlgKEWxzFYomS4fx5e X-Received: by 2002:a17:906:e247:b0:a37:d5c7:406f with SMTP id gq7-20020a170906e24700b00a37d5c7406fmr218726ejb.75.1707513748875; Fri, 09 Feb 2024 13:22:28 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707513748; cv=pass; d=google.com; s=arc-20160816; b=ibO93A+dR39jSCsg8ZIByA/vZwwg3v5fmcvfEpqAPJ2Rt8NRt2Z7IJQFEve2T725DQ WucBpIzYYMSv6ZL8Z+b0QEx2gUc3WK2+znRq7YFL8r+YctZpqeWGFiqfgF/HOMsGLVkz k0joUezDjzvRHUcfDOU3zUo+Jei8FtijCxVCuA5rwT1jZG6HSmr8ovI35U+RaCCWJS0f C2SE1HuFGtG2jm2fGlwepHfdhuKN2Pn+AMY3O9Hugq7KzsclvdMEvzt+CSAA+/PNcLRf okzgzku+P8i0NUwgB9tZXCVcwcqQaHddCkEtvEzva4E/8TkLxSZ4FamlwNE238MbCWY/ k9Rg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=JFThVNztewuogOwemulUSfgykwCuDZCM+kDTpLgPy0E=; fh=i0ynaHhbCCwinWgKKStnNJbmp72vyoLC/BsAr3XTcHQ=; b=FHuWRgPPDB2HsW1j+9epNI4rNfgD11x1S7UBD+jUyy5rHi+fuXZLn8DqAftgVeL/QP cIPXiZsBe6VEjpd+unUkLQnoHZ+0U8auSgE8Rk+Czy9rWDCfbEphysdWTEEGS7UGAlQ0 xpvp3jGDM2SXnlPWtMCxYGvE9/0dkKjscehXk6jrziTRck1/HBL9MS3T1e+g9mxYDE/1 WMN8s8TElymRyeNkhLsYdzZrMt099+k1qj8WTz8JgwhT4puECTEfToOIWV0wmxstvjWk Kx/15UAv6BlivCuDXrpf03kW6rkwyoROQQEE9RhI82sn/UU+0Eq0/QWYIka6XLgRVpcB Bh4w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=b+oKolW1; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60020-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60020-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=2; AJvYcCXnfXd7D3ZQyieQjrvS3brl2Cxsr9Omb1sz+T3aQXH0WppRWzPaGocTqb4llIvywAtV0iQWXiWosZhNRdPjrcpuwpyZ9A== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id s16-20020a170906169000b00a384708a8e5si1206579ejd.589.2024.02.09.13.22.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:22:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60020-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=b+oKolW1; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60020-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60020-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id A12ED1F247A4 for ; Fri, 9 Feb 2024 21:22:10 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id EC04D3B1A4; Fri, 9 Feb 2024 21:15:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="b+oKolW1" Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5545339AF2 for ; Fri, 9 Feb 2024 21:15:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513350; cv=none; b=CqS+KLqckHSmPosJ17o5gQvwQWofJ7lSx6n61LrK0QPXzm16pkSLwpnIsTTnddOkz6Qmlz6riRd4jPmVyxfKRbxGmwzKMaHgcFQIbe5ZjzStwoaDhLcdzOqcnFGK7MBF7dG5p4RgoWrEQiwyppUI/82hBoZGawsNzfQ1BBTjdmA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513350; c=relaxed/simple; bh=L4RwZXaFjOs7euXjMxwYiABkrXXZbZcKloyh3rbhzu8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t6BB34qGvURvX2iy6AyA+wdrE3LUBNmVoq8cpJPiqmQfVyBHZEy2EAXhrMxVscG4TtJ3DOnNA328oirvSYQk/T6z0hN/VD4x2vgmsqnRSUmdbZqjDanl1SLcIzVtwDEypvbnVBvQR/CBjsEVFrilWk8HkWnTLJt4/7JC7gAMtww= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=b+oKolW1; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6e06725a08dso1057098b3a.3 for ; Fri, 09 Feb 2024 13:15:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707513348; x=1708118148; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JFThVNztewuogOwemulUSfgykwCuDZCM+kDTpLgPy0E=; b=b+oKolW1DTy82nN0oHPSubUTCq0lfavtXVbB+bq7BhtLyG5UQz2/UGVrWFoqxLXtXZ dVRnG/xGHm/t8rC6wEND/pRFlk4EvZLpzlsB3pJmN3pukc4dAwBYqSP8jIr99g7Olovb 5lzKrpzLmclX6vrZj38Tco/LFCzUmVtqNHkfc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707513348; x=1708118148; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JFThVNztewuogOwemulUSfgykwCuDZCM+kDTpLgPy0E=; b=bHitKsVNPD/hLvSCX/8g1P+Ls6Q3SR2cil5iilECAVBPdWTi0P9UkwjsS93JT/Ci0L jGiFk4QNpn9kRJUswzK2Bnu5QSuWMmGlON9P+i9u7r97fKk/CqJkEg3HWJGfZJk7XiCt taemAswDDAbTojBiRa7NzoQfgsX2cKi2rrmHxBJLzFFc1mGdaB5JgK21Mh8eaMJQUj64 8akIYAHNzexQl7mv73aQhOb5rl6JI33GqN2nt7/zIH4m+cHygHxrWfQcGwZtx/XUeFvq 93KOBCM293AQH23ENfhDpEW1WmdSrgSQvAkxtiJG7dG4KNVh3PvHe//bj30f5fZbBX7D Ri6w== X-Gm-Message-State: AOJu0YwbNE5pk1f/VkAd/JlM/dc+Rz8RAXg+UCTrSLhmZxu1EHctMhh6 7Z4O2LObaooAzqoCqX55zwxuKvJOgkwSpXYmvD4nJ/Y6yBo6v/pUkptWz0rW1J/G3pGSfRtBBbP 57KL6+atoEcYdY2xpJQmAGgVmenN2d2dgZvvrW1Y/dyb4J1J4MmGZznB7pMk3CPS21vALOqFwch JGYenOJWZXFX4MDWZXqlStdxxIUJQfnWxqq7AnVj/T3jU4TA== X-Received: by 2002:a05:6a20:d708:b0:19c:8d73:721e with SMTP id iz8-20020a056a20d70800b0019c8d73721emr464952pzb.36.1707513347879; Fri, 09 Feb 2024 13:15:47 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVdfNV3zIRMqCFWm0pUC8hu56lzq0vEgBCLGbtY0asu7ymZ5imx6EuZrSCehX/vvQ8oqWC6oUjGT+11xrnw2ltOPIfx0BIPn9mTkH3XYri9Y4KwD9wFMlJxRkPFBBcOUb7GxFHqJRwHwwkUc286hkZznY+jNz1pSnmV74R9Kv3uv+DskkGQvb2XrOU+AtdzbMwdVM970wFrPdGDXbeNfsY1KkeDRZL2iONNd75Q18hZhliWAZsjdRFiDV2z3qEh2Sm1keYNhjbpAYwb+wCXbTwMnbYvZhq+6njlY8t0iTq+EAaaUaq2Drpc+Sb0zwS6FPNBNVFXr570xg7UPO+pPusZj3GSrjKB/Dpr0eNDdQ0BKl0BF7l6CzXd5b9anDK/ltX3YwdBp8r8bdIs1M5TNd1E24E9RicTb7jvvmDgL51zokAZzyzqyUhz3Qvu7Qs0x0KjL+8L1PCUYwBkh0whJK1UTAt3iJXUjxzhP5T1sGJlqJ+djLuJGYx05CE6B+dOhdxGV0/vHZ3prPDyNTtUcrIfrX+mNXG1G5KmREXRS40SyQOzcumij41wqEK/HkeXUXPF9GUZeGUDLlklma/nmL3I1IBFeGTrC2y53gyOLsmEBfPSw87XKwJPdDFTgEU= Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id x23-20020aa79197000000b006e05c801748sm969629pfa.199.2024.02.09.13.15.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:15:47 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v7 2/4] eventpoll: Add per-epoll busy poll packet budget Date: Fri, 9 Feb 2024 21:15:22 +0000 Message-Id: <20240209211528.51234-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240209211528.51234-1-jdamato@fastly.com> References: <20240209211528.51234-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790457936502890867 X-GMAIL-MSGID: 1790457936502890867 When using epoll-based busy poll, the packet budget is hardcoded to BUSY_POLL_BUDGET (8). Users may desire larger busy poll budgets, which can potentially increase throughput when busy polling under high network load. Other busy poll methods allow setting the busy poll budget via SO_BUSY_POLL_BUDGET, but epoll-based busy polling uses a hardcoded value. Fix this edge case by adding support for a per-epoll context busy poll packet budget. If not specified, the default value (BUSY_POLL_BUDGET) is used. Signed-off-by: Joe Damato Acked-by: Stanislav Fomichev Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet --- fs/eventpoll.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 401f865eced9..ed83ae33dd45 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -229,6 +229,8 @@ struct eventpoll { unsigned int napi_id; /* busy poll timeout */ u32 busy_poll_usecs; + /* busy poll packet budget */ + u16 busy_poll_budget; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -435,10 +437,14 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); + u16 budget = READ_ONCE(ep->busy_poll_budget); + + if (!budget) + budget = BUSY_POLL_BUDGET; if (napi_id >= MIN_NAPI_ID && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, - BUSY_POLL_BUDGET); + budget); if (ep_events_available(ep)) return true; /* @@ -2091,6 +2097,7 @@ static int do_epoll_create(int flags) } #ifdef CONFIG_NET_RX_BUSY_POLL ep->busy_poll_usecs = 0; + ep->busy_poll_budget = 0; #endif ep->file = file; fd_install(fd, file); From patchwork Fri Feb 9 21:15:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 199115 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1138198dyd; Fri, 9 Feb 2024 13:22:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IEy6aB4ivxGvsKc25s9boPqPbKOa94ZypkdAK37fIOOoDLF+6qustLBj+By+se5NOacfpS4 X-Received: by 2002:a50:fc16:0:b0:560:d8e6:8da6 with SMTP id i22-20020a50fc16000000b00560d8e68da6mr100010edr.3.1707513754188; Fri, 09 Feb 2024 13:22:34 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707513754; cv=pass; d=google.com; s=arc-20160816; b=irxToO+SqJ19p+Hm3RSG2N6w8UxfTfpj9vL0+TC0zifmkXER6yueYdSG4tKT4wq+qq yJSod5Sp5fqac+oAHd1mcUa711GVVOtt7wECWkdYDEW8RhOLmGBP0U7ae5QhLBwkIZrZ WP994x476t+DuAUg9c1i0v07RovWuRESfegabxKgqIhg4x0H2hrYPTvsSFo5YPxvjdzx 73Kz9dTbKvAjV8lvOETgPh2VQqE7iTj1Ggh5GJ5T2ZIXUifVfTBrelNDHe8XXqieIqy4 QX6QJgT3eVumH+alSyiWFWzsQqG1w2rtKIVR4Kha75qzo3wmDGr5Kraem6Gd9Q8GcY15 9fUQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=G0NsVP2+DJYJlYo79AHfvXsFPLMqC3dLNa8zH3nE+As=; fh=djdNnzYjz/4yR7JXCYgc3C3fqB4XSGEes39lqDDrw9k=; b=Bm+jzUiDKVUBA0SyBbCPhmKNUfUJxk6pjG61jHj0E4pqsOz053oE+eeQaejLP8D62o Y22y0xTq9K+wyNNh6h85TDCArfiyo0QzR2s6BEaLMO4Kh1DFvAIT4WSv+b0w6oojAhwb wGjnQiq18wIjYbkhlDVBA4sSncfauFn0Z9BqLXnD34bbW3Dl7kZOrHWAFAJ+xjQDe7Zb ejK8lzYXmzEMvezJ985aA9G278X1PftlhRxdlgZjaoEdERHW4E4oqXqbk/n856qCD2FW cKpYq/nAAa0NHwJYZbrDtxGt/ivyZc2pnJHiBpWOQxFkRL2HpQ0Tq1aWTh1zDYM42eay qAPw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=tG2UUo5K; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60021-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60021-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=2; AJvYcCVBzOMha/uUO51T6nRzCRsunwYz7IAXNwhj6BD++0HzBQ9L9cviOUnNHcBdDH8X02jQdAK9M7SlUt1MMHVfgm/VETFa8w== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id y19-20020a056402135300b0056022a0d6e5si123389edw.466.2024.02.09.13.22.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:22:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60021-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=tG2UUo5K; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60021-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60021-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 2994D1F29E75 for ; Fri, 9 Feb 2024 21:22:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 828AF3B79E; Fri, 9 Feb 2024 21:15:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="tG2UUo5K" Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 00D8E3A1B1 for ; Fri, 9 Feb 2024 21:15:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513352; cv=none; b=pZKWyuI1EvPSAfhRdAOSqLO+YYq6IKlKa514FdcyntUERspKYv9PnXBd49QaAEgOna8bKJk93yC61LSIOQb1LnRYw460DqcV/VOs5ulCmYSQkrot4CRd002QpIDwid50zkC199C2IU9KQ59yTDTvDr6pGbhBbT0dW8OeeCCLy9Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513352; c=relaxed/simple; bh=CJVUUyQixO3GY5CVmRJIGygCqjREOllOxrIr3Eh94fc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jvpoxNgr1VwZWcYTrFV0ZzZWAHJ/fTqOGtkIq8BLTUeQJgAITuAVrOoRVlK37/9kyf0i27eDxTg1FAG67si/qiu35NnfrfQuys8vvCrSbjvGywn25mzf6RJInNZC345itTv8FOmb3ScLpPK77bNWnqbsg8F5UWQagwQRyjeiGhs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=tG2UUo5K; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-5d8ddbac4fbso1194116a12.0 for ; Fri, 09 Feb 2024 13:15:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707513350; x=1708118150; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G0NsVP2+DJYJlYo79AHfvXsFPLMqC3dLNa8zH3nE+As=; b=tG2UUo5Kpulb48owiZcJFB8pxTuNhFU9SFG3y6Pp2+m3iTqxdP0TNvApcT4I8+TCqd g46fYzuQUTpOtXqJJQuOh1qkYpoEarnlL0Fna8bntuGSxxCWfXSbQkAY4Ft4wi+GltSY VWO9pMWohX3ut8JEMS147DOZyzh0Q4/O3Y8UU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707513350; x=1708118150; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G0NsVP2+DJYJlYo79AHfvXsFPLMqC3dLNa8zH3nE+As=; b=sqR+lvHq9QTD7Lwf351BF3kO/DPieiBZ9BHtxg+Olf3C2lWx+NgybItnQsIAO6/03m IWrISxyJTaENcrAVTu/AqljxrcY19qtuqNCSXovngQ66Qn4KRrrV70w39VQjNSUMDjv5 yQ3V9QR5JvtZrqmly3jWoyUX4toBfx2jk614dLkxdcEbqI7Fmy/oQQTfXzPhtQt3M3VD 4n8xtRdZprDKfRCe1dlsBc8+RaeZ2lio0C/xOiBEOV/VfvhCFE/gXeyo12M2KO2Bexmb s2CkJowvP1owsb3a9q7YKVlpuA8T/4F5sWw2lgtHe+9ksf+pnupXXEbJBaw+lbjoutG+ 4I5Q== X-Gm-Message-State: AOJu0Yw+e1svK4w26G4aGQAmMYT+hpA1X3eKCmWRigPw0Iq9nj3Tj1hX 1rK4VEoK+Z+Y6rP8enhHg2OOW0Z3bmnEwsX9Y+ywSVM8KUvqUYgwZs/xbJdOkcMYjuK5FfZsjxU sc2K7h+Vv23jcdZWDWajRXDhuU3gwfso9U3Fbthh/UKQFMx0KLibfxherzuElKp1Kglylklf3IM 9BHkWBsxbBjhng7o1HbrMnYgixawel7aYCODFXvgmPOC1+6Q== X-Received: by 2002:a05:6a20:9c99:b0:19e:9a59:20df with SMTP id mj25-20020a056a209c9900b0019e9a5920dfmr417488pzb.9.1707513349972; Fri, 09 Feb 2024 13:15:49 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCWNGCpYwEjdWOp9Af1u8tW2O5xfC8VpboIT36q3QoQQ0xH8ctFz596nOAMoJ6F9eyhTK4CEvJ51uFuMx4RHYRgPLExDttotnTGSfTdhT23/b36g8AFG7+uss8UnzMKMa53LRAHzowhtf0al3hzyCvrkY11h/lWzYIPynIHTbIN6G5UcjMAIPlzOVTX095Co2AVRH9cDkWKI8/1rFzGFHNlXbp85kHT81yA+mrIfbgPMAMw+U1nAeAtmV5ebnFKXhLt/PI9r5tcZhGMuVg1ZfywC+HV8IxiRKzPV/lxrB8MPOR8HgvmRPf2u0u/QI5ISX9m31W8z6zinbpMD8h2Hb4jl7zx3p4ZPGSM7XSYfzsXJNLqAmY9wa0hkB+0hnDhEMgvImlirN4gAJGlPng+ARn+AZpzmpIk4WS++qFSfmnCroCn4gOE4F+T2RUvPsGLi+7fMs3YNc5Z747JAwSsALvsjuGQ1fvuK/rQO4ph0xMzubDjDQzWiPoXGGnNdIa7pyamFRUmN5+5V2DdYXB7H1i20ZEHsDOqik5LK4JATJ3l08TuBrqHTaHawQ8+MnQomcrLFsBX9LzSWPMf7UK9UZzaVwSRWUnGTI+hnps+26D5KUGMH5xl8Lr3OSaL/c/Q= Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id x23-20020aa79197000000b006e05c801748sm969629pfa.199.2024.02.09.13.15.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:15:49 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v7 3/4] eventpoll: Add per-epoll prefer busy poll option Date: Fri, 9 Feb 2024 21:15:23 +0000 Message-Id: <20240209211528.51234-4-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240209211528.51234-1-jdamato@fastly.com> References: <20240209211528.51234-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790457942272223327 X-GMAIL-MSGID: 1790457942272223327 When using epoll-based busy poll, the prefer_busy_poll option is hardcoded to false. Users may want to enable prefer_busy_poll to be used in conjunction with gro_flush_timeout and defer_hard_irqs_count to keep device IRQs masked. Other busy poll methods allow enabling or disabling prefer busy poll via SO_PREFER_BUSY_POLL, but epoll-based busy polling uses a hardcoded value. Fix this edge case by adding support for a per-epoll context prefer_busy_poll option. The default is false, as it was hardcoded before this change. Signed-off-by: Joe Damato Acked-by: Stanislav Fomichev Reviewed-by: Jakub Kicinski Reviewed-by: Eric Dumazet --- fs/eventpoll.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ed83ae33dd45..1b8d01af0c2c 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -231,6 +231,7 @@ struct eventpoll { u32 busy_poll_usecs; /* busy poll packet budget */ u16 busy_poll_budget; + bool prefer_busy_poll; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -438,13 +439,14 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); u16 budget = READ_ONCE(ep->busy_poll_budget); + bool prefer_busy_poll = READ_ONCE(ep->prefer_busy_poll); if (!budget) budget = BUSY_POLL_BUDGET; if (napi_id >= MIN_NAPI_ID && ep_busy_loop_on(ep)) { - napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, - budget); + napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, + ep, prefer_busy_poll, budget); if (ep_events_available(ep)) return true; /* @@ -2098,6 +2100,7 @@ static int do_epoll_create(int flags) #ifdef CONFIG_NET_RX_BUSY_POLL ep->busy_poll_usecs = 0; ep->busy_poll_budget = 0; + ep->prefer_busy_poll = false; #endif ep->file = file; fd_install(fd, file); From patchwork Fri Feb 9 21:15:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 199116 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:50ea:b0:106:860b:bbdd with SMTP id r10csp1138313dyd; Fri, 9 Feb 2024 13:22:49 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCW8Jx9cgD/1f2GIYAquNFqCtRwe5PAPmRhA4fGF5QgLtKp1Q+9RZE1fri5iIlqzTQW3dW/lXFlP6Mypu9n1LxSaM55vsA== X-Google-Smtp-Source: AGHT+IHr6H7EUnMSocGexFa08jAvYe9m3hF+H07VtdWcKoSeoZHXbr+0qSOYpXMgmq8tTTE8SILt X-Received: by 2002:a05:6a20:6f9c:b0:19e:abbf:a47d with SMTP id gv28-20020a056a206f9c00b0019eabbfa47dmr345648pzb.6.1707513769615; Fri, 09 Feb 2024 13:22:49 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707513769; cv=pass; d=google.com; s=arc-20160816; b=NzLmoWmGcmljGcnbbtY55C0bgQhUnnsYmGqoTAZQ/GHanrwukNpm2T+/caL46qjviL OAxB4Dxwy4pmYiI4/4KSGvK/YPCtPz3td+t5gzRc9nEDmks0sedquuzF+BhRPP99lYJ2 swnhlI/CE8tf+14FzbJxo0nqiPHvnkDkai1hPAl48afwSHFrCat5+VC6ya+UtT00CLr/ CN8IJGRwFtGD4D5t7HOttQaNbnUcuQQobFgj3uv+iViplKlz22+Gy1Xih0f7RUE4M9Ny DSNpXWmkG2Dv8nKF/cgg/TLGtEuMRUhJ3bNFxwf+wBPTFCcIzIwOWdvlhRRo4ciAlu6/ doEg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=8T72+tfYxIVMKkrQX69I/sG/Ln/429Cv0P7tqGCS/cM=; fh=w9J29KvElaMGmCNkCG2a59JfI4DNHgPdu6nZkaSg1BY=; b=DOkbcMPRLJ8E8MpQPAoW/HlUA9/QBu3XjPI6bBUzGgJCVnnJMOjf42DaAFn4WEmVHf I7lOuF28OnHDF2VOpcsxP0m5PemAVeyGmfFw6NiTaA+NeKPN0rZ7/iaMv7bItbNeAjpe b6Yd65+UxIwsMD9Chw2srHG1no7cW2z0F++S1V0cFCTq/eP6R6TlE56cYQB2l4dR1Me6 5TmJg3FtkKvyWWmCXdk5qIkB4kEJAoJvEFYbhBQ+vxMPdq4HIiFWWQeS6UOXrO63snqy HCLFLywAWh4S0aOYTuw4lUqPzf/Su2Gtvgjb16WTVI2xfYtvyHe/k6VzB3yszRfLNx2m cT9A==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=wLoX768r; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60022-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60022-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=2; AJvYcCWUd6CSluW1TeUZj1IZJXNGO0RY2aMz29XL8xxRsgcvL3fOf9sZWl+KLkK0vC4ZN1reAKpW+QVMSYIB1eG/IiQ5D/Ke9g== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id l29-20020a63ba5d000000b005d8bbf7e78bsi2371636pgu.307.2024.02.09.13.22.49 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:22:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-60022-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=wLoX768r; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-60022-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-60022-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 64624280DCD for ; Fri, 9 Feb 2024 21:22:49 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D5DD53D0B6; Fri, 9 Feb 2024 21:16:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="wLoX768r" Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7B063BB25 for ; Fri, 9 Feb 2024 21:15:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513359; cv=none; b=qlH2qgJ5fHddsbGpE5tsDgJy9GWlxkH7s30OtO5GUfiPin6rb+EwwyVHivTUKXrHBznYxbvwMj9a1jD9FBQBoo3xywmxuLGDOfmqo0uQG3tQf6PIZv+j6drgmQ0N6z1tDMpzHJpHHONDf4K61X2YTqe/xe/hhdir1XUD8Gt1tAo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707513359; c=relaxed/simple; bh=bU1lRcpmwwquP/3WT3NzDHAkVOQAiLuggZfIHJAVD+s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E4NwwezXNGc7He6DwWFIrK00tQduhbTzojlDeVWYCPxpoqXkX7fMqLacflLkSn8agSGbljMeFnXG8Tj64sKXAtPYF2KvazVBieJ9CJZLIxeGYjTo3biBrTdzoHg23k3DkkwEuJtJezASPxYj7U7IYvqd+Qw+j80Gn4wG3t12Gac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=wLoX768r; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6da6b0eb2d4so1279253b3a.1 for ; Fri, 09 Feb 2024 13:15:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1707513356; x=1708118156; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8T72+tfYxIVMKkrQX69I/sG/Ln/429Cv0P7tqGCS/cM=; b=wLoX768rhfA6KiMLsxoXieDDIgiLiuhFmxIyhI6YSIG+EK1zztLADr3sCzO3Ug0ajd InGT2QhFIXO0/IBTPq+qbKMx3k0ba1LnfNqpKknZ5Y8YDVf0sJmpF3N0X1+4b4LQtGKQ /6PXRPFTtWM1DH8u//n8KiSrtV7cVTAGB7drU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707513356; x=1708118156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8T72+tfYxIVMKkrQX69I/sG/Ln/429Cv0P7tqGCS/cM=; b=q1/KzVG04JRYVAVqbiiexlsCkoWCs5e7jvS/rOeWq7owQcgYklOM72luZkVgdqzLH2 XFRN4J9ZqGwX80nj0H7WRlj8Vq9fU42tOELpSJ6RDP44OD0nIAFSk8/wmtcvPKpQ6Upx 0nXHSJsnqm1BCLlpTskIEG9kF0lJRb1qJBH91gx6UAkXJY1MqNw93pZ7IPeDfEjesish UDi2ODEVobOa7BVqQy2fcJWnYlH16KM0g1DHN7487KyMPEplIs+WQtCYEVwKA+She1bK gG0HWjwjNOCFMPJqvPy7f963KVrtLaWrSji7LWOswkV9+BqIkt+I4xF5KLuvGV7siRUg xb5w== X-Gm-Message-State: AOJu0Yxo+YoOnvkHhm+WjB6Iiphlc476U+3DhkD3yxDt/h0ogxJi84i3 1wjUDRhP8g5ACzhqn9VgPkBUeGvytBQmpmqxD73km4WPz7n+v+ehJwKDtwD5jKAZf1mpLXl5WgH gpNkpyy/5CsVIRJiHc1Tlr7R+O4qICpl4aCcUMYPwDfTJpKH6DKRW2Ah8KMkDarPeX9ylUg9B8Y dgSWbKDDaGQ+ThvmT7xExQFgY7S7f9hHG+1nIcKfhlE6t4VQ== X-Received: by 2002:a05:6a20:43ab:b0:19e:97ee:af55 with SMTP id i43-20020a056a2043ab00b0019e97eeaf55mr416941pzl.1.1707513356321; Fri, 09 Feb 2024 13:15:56 -0800 (PST) X-Forwarded-Encrypted: i=1; AJvYcCVJj4LR8eeRuam1BjuM1EtiGu4vRjwhKwhCP2VDo1jULvf+oH9l/0VIv52J3i3gAldtueQNp1HaQlSWVqUAOltAJbLlwiAvsey9QvL1B7bO02YsgDNJopPwshTnQ3tyFQ6D5kt8M3NjEpwPS6FLo2uH+M+2v0a/RdTTzAQMW2Wu/fkW2IxjaJRXRaCPKig0A7c+szKH2aD5loF5hu1ERoEbVmJPJxxXdkT/Vt0PKlnyYneCEtS7ltvzrsljk8C3Bpg+W3KkH9i4zF7se9Mg7h2ixJKrXbXUfB1SPt3wfJNyvDnaj3Ske0SQb3UB66GJk2kpsCuTlpKGzysTVLj+0RNmpNSkFGflrBDALZ8AHclwtbHrFS46Qk4vju1eWXlcXIBFaO4B4M+QJBI3TStBkygryk8choXmBnwFue451x0uWFjJ/hyUtYHRnl06YSLsOdi3sq93YtBuSxTZTNm9dO0nB9E6hPNoFgQg7TjU/KepJumiDHhPjJRqkhepXJ5Rr0gucnslzFbw8Vt8wGJsMuP+fLBrI5uQMlx1h9mMGYoJsNFKMlsh/rX9ssjr5PWAD1Pe8L4cPxjWGwdpeTPzN6lFYbsHnTGqTBZ5Di6PHpnyv3VSDX2xhwCPnwEV9nbI9Nb0MIS3t+QdYy2jHaEyiWwsB3MwHluQ5zsHDIv2nfbSanPshj/sbNHgLmrqsc0bdMD/Zzr8YqNOwI9PyrBc6ntkp7F+p9jl4i2svMK7kg+UZztPA4V5jfaqDTHVD4BRKLWaoaRI/F9Vas3w1UwNh9r1dc0VeTJBmuvQ317k9SaD1ICDR2/yT5Dut0Qp54NyDdhHTFSeCMLiSy9u4SMwngtb4Hifi+a71p9k1vZLWaUxxbA0TdxSCaPzXni33Ur8qjXbknmy6anUxmQ1COpjI4kmgzhjw9T8U6GhwFufx6XZF5IaHrynioWVsudVbIGracylX7 a02wPNR2PPU0KDXcJqz/4oACS54pay537goYpUUFa77oBslUA1Ow== Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id x23-20020aa79197000000b006e05c801748sm969629pfa.199.2024.02.09.13.15.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Feb 2024 13:15:55 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, sdf@google.com, amritha.nambiar@intel.com, Joe Damato , Jiri Slaby , Jonathan Corbet , Alexander Viro , Jan Kara , Michael Ellerman , Nathan Lynch , Greg Kroah-Hartman , Maik Broemme , Steve French , Thomas Zimmermann , Julien Panis , Thomas Huth , Albert Ou , Palmer Dabbelt , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v7 4/4] eventpoll: Add epoll ioctl for epoll_params Date: Fri, 9 Feb 2024 21:15:24 +0000 Message-Id: <20240209211528.51234-5-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240209211528.51234-1-jdamato@fastly.com> References: <20240209211528.51234-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790457958683475022 X-GMAIL-MSGID: 1790457958683475022 Add an ioctl for getting and setting epoll_params. User programs can use this ioctl to get and set the busy poll usec time, packet budget, and prefer busy poll params for a specific epoll context. Parameters are limited: - busy_poll_usecs is limited to <= s32_max - busy_poll_budget is limited to <= NAPI_POLL_WEIGHT by unprivileged users (!capable(CAP_NET_ADMIN)) - prefer_busy_poll must be 0 or 1 - __pad must be 0 Signed-off-by: Joe Damato Acked-by: Stanislav Fomichev Reviewed-by: Jiri Slaby --- .../userspace-api/ioctl/ioctl-number.rst | 1 + fs/eventpoll.c | 72 +++++++++++++++++++ include/uapi/linux/eventpoll.h | 13 ++++ 3 files changed, 86 insertions(+) diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 457e16f06e04..b33918232f78 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -309,6 +309,7 @@ Code Seq# Include File Comments 0x89 0B-DF linux/sockios.h 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range 0x89 F0-FF linux/sockios.h SIOCDEVPRIVATE range +0x8A 00-1F linux/eventpoll.h 0x8B all linux/wireless.h 0x8C 00-3F WiNRADiO driver diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 1b8d01af0c2c..aa58d42737e6 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -37,6 +37,7 @@ #include #include #include +#include #include /* @@ -494,6 +495,49 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) ep->napi_id = napi_id; } +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct eventpoll *ep = file->private_data; + void __user *uarg = (void __user *)arg; + struct epoll_params epoll_params; + + switch (cmd) { + case EPIOCSPARAMS: + if (copy_from_user(&epoll_params, uarg, sizeof(epoll_params))) + return -EFAULT; + + /* pad byte must be zero */ + if (epoll_params.__pad) + return -EINVAL; + + if (epoll_params.busy_poll_usecs > S32_MAX) + return -EINVAL; + + if (epoll_params.prefer_busy_poll > 1) + return -EINVAL; + + if (epoll_params.busy_poll_budget > NAPI_POLL_WEIGHT && + !capable(CAP_NET_ADMIN)) + return -EPERM; + + ep->busy_poll_usecs = epoll_params.busy_poll_usecs; + ep->busy_poll_budget = epoll_params.busy_poll_budget; + ep->prefer_busy_poll = epoll_params.prefer_busy_poll; + return 0; + case EPIOCGPARAMS: + memset(&epoll_params, 0, sizeof(epoll_params)); + epoll_params.busy_poll_usecs = ep->busy_poll_usecs; + epoll_params.busy_poll_budget = ep->busy_poll_budget; + epoll_params.prefer_busy_poll = ep->prefer_busy_poll; + if (copy_to_user(uarg, &epoll_params, sizeof(epoll_params))) + return -EFAULT; + return 0; + default: + return -ENOIOCTLCMD; + } +} + #else static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) @@ -505,6 +549,12 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) { } +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return -EOPNOTSUPP; +} + #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -864,6 +914,26 @@ static void ep_clear_and_put(struct eventpoll *ep) ep_free(ep); } +static long ep_eventpoll_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + int ret; + + if (!is_file_epoll(file)) + return -EINVAL; + + switch (cmd) { + case EPIOCSPARAMS: + case EPIOCGPARAMS: + ret = ep_eventpoll_bp_ioctl(file, cmd, arg); + break; + default: + ret = -EINVAL; + break; + } + + return ret; +} + static int ep_eventpoll_release(struct inode *inode, struct file *file) { struct eventpoll *ep = file->private_data; @@ -970,6 +1040,8 @@ static const struct file_operations eventpoll_fops = { .release = ep_eventpoll_release, .poll = ep_eventpoll_poll, .llseek = noop_llseek, + .unlocked_ioctl = ep_eventpoll_ioctl, + .compat_ioctl = compat_ptr_ioctl, }; /* diff --git a/include/uapi/linux/eventpoll.h b/include/uapi/linux/eventpoll.h index cfbcc4cc49ac..4f4b948ef381 100644 --- a/include/uapi/linux/eventpoll.h +++ b/include/uapi/linux/eventpoll.h @@ -85,4 +85,17 @@ struct epoll_event { __u64 data; } EPOLL_PACKED; +struct epoll_params { + __u32 busy_poll_usecs; + __u16 busy_poll_budget; + __u8 prefer_busy_poll; + + /* pad the struct to a multiple of 64bits */ + __u8 __pad; +}; + +#define EPOLL_IOC_TYPE 0x8A +#define EPIOCSPARAMS _IOW(EPOLL_IOC_TYPE, 0x01, struct epoll_params) +#define EPIOCGPARAMS _IOR(EPOLL_IOC_TYPE, 0x02, struct epoll_params) + #endif /* _UAPI_LINUX_EVENTPOLL_H */