From patchwork Wed Jan 31 18:08:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 194892 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp2069664dyb; Wed, 31 Jan 2024 10:11:02 -0800 (PST) X-Google-Smtp-Source: AGHT+IGAdNNaAg7AB7uXcpB5U9ebPPNzWcEy+xafbsTgsDprXs8GMLsdB0idiWwaDxbVkWXVt07q X-Received: by 2002:a05:6402:430a:b0:55f:a33a:6ad2 with SMTP id m10-20020a056402430a00b0055fa33a6ad2mr660631edc.24.1706724662401; Wed, 31 Jan 2024 10:11:02 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706724662; cv=pass; d=google.com; s=arc-20160816; b=L9FAwaW9XvrBTjWhmOuMAtf3Ouce0piGNiMcfFa1usPhfn0KYPMczD1x5695KXNqSC 702ispbdLJauKSb3loFIqGdnYevd+bojGtgHd050qKnCBwMJMeuTOIIuIx2+5sWjJOnd MojRSQYZNETB7uGXsa3JM2p6Zargiw5azu4pTP0fJxadSs/t7uTgEf5i8MvY1S/2i+Dm RoMVnbfwdnomDHgIQ0H0oLCxwh3WE5haOf73nCxRX/OL+/6T4ezZP5K6MxBmgbukd5RJ nneEncQZ6q2ONoN2eY44DhkAbfgKbEpGrVV52PdUPgNWH0pT7OuWShRp+OHnkqfetaQK zDDg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=XPyBFpA03VgPUcSSFnhqRnjQDUKtK8Zok+qzBXzC1y0=; fh=lanZMMQSJbBWwXfb+Qr5p9tCBtUegS2BzYZa2TAAp3s=; b=dTCXucTYoYATk9jS0EkKXyvTJalI/6skDg7BKbFzniT/ANu0pihK6gXnMqquhICAYF 40FbX9lnufcVieSp+B0OySGe9UnF1/KZ8J5DLr6KLejG2TrHoojxyYZsSXkIhnIy8C12 /5ix/uW1GBYKzO2mNAO1Goh5x30NNUkWkyz0SyytGIJ+zG+MvdtKt3EWTbbkZ5mQioWc m0zTLxbHsevfIWZYJc7D5JOfMdz2KjT13i8AV3SgToUQSDNQkjjtLWIT0xw8FO0zd8nE qxmTRrvuRT560vve1N40ASu7sSDa7R8E+29ghryfqN9w7lKR791jWQCTBNDSt0gu9v7u 5ELw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b="QN/mleoj"; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-46954-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46954-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCUyXOajOkH1pwalZmotGfc1j/IMOv0VM+UxudfD88OK7kW5DUuUK0DFY8UGNuJqol22hunyKyLl4HUuK5fMFefdW0UYMw== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id 13-20020a0564021f4d00b0055c8e01428dsi5947007edz.123.2024.01.31.10.11.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 10:11:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-46954-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b="QN/mleoj"; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-46954-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46954-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id D0B441F26068 for ; Wed, 31 Jan 2024 18:11:01 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 3B6F613BE94; Wed, 31 Jan 2024 18:09:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="QN/mleoj" Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46571130E35 for ; Wed, 31 Jan 2024 18:09:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706724582; cv=none; b=c1tNh7BDF1deeNkwYQyo09y0LI4fTTxMHaB6g+ATyvRuRoiIIXgJmVOfKS4FwP4GMsRZCAJ2akcDL9TUGSpy5cIi0NFB3HzIR9ZuyUvYRisIRHJnKqNEWikJhqoODHA481ioVNeXAIvlkzeq6nNYaQIixxZ1dln8d/LuubqO1Ko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706724582; c=relaxed/simple; bh=iOH3rDvMU6l8EibI+PkwMyfqV5hY0EoK3y01y1v3/4g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iKR4xbjhK2x0jEVr8PyuehbiWu9bHXkZ84IGwgsY7usiIOQMmASyWCiCTiyb/gCpM4zgbF6C7ZawWEEcfsntUTNiR3T4bqbZqmQ3HgzzxweMN+aHu49N/c6WTfywPDcRgfOO2EDjiS5R/V82Th8Vtu/zHdZOiH50X0zV7JBPAbE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=QN/mleoj; arc=none smtp.client-ip=209.85.166.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-io1-f48.google.com with SMTP id ca18e2360f4ac-7c009016d5bso1946039f.1 for ; Wed, 31 Jan 2024 10:09:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1706724578; x=1707329378; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XPyBFpA03VgPUcSSFnhqRnjQDUKtK8Zok+qzBXzC1y0=; b=QN/mleojEw9HkQpWKu/XT/shVQnhEDCSJnfeqqIU1XjaWrNL1oKAVNbbR1q4FVA8Rp 9SGHlk517ohEulqpzVOU7svJFvmmTmjqvvc0Uf/UThQfSjVkZWhax/Vdx25yyBrf5T6Z ZG87YKxa0QRTrX3AMqbkiWkfms8jWWpSJzNms= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706724578; x=1707329378; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XPyBFpA03VgPUcSSFnhqRnjQDUKtK8Zok+qzBXzC1y0=; b=Gt865RnLr7sbgj2XD700iDU9JWb2N/RPu76RZiAKVSJ0ib8tuoINiDPgl3gFqjlq7u pSqWVf+sSmho+q9jZLFj7hwM40Rjj9OauZrqyKq7dhEsf6QG/eqJJht45Ia//Cp2dHxK j0p1gbHKs/nJLGaL+j0W49azGF4/5TIt7aVu0RB92esAdmFA3m6wktYUxK2ymYTlkCC+ tLCaZ4QvLf8S3kPw6So4nP2+AuR9FJqC+Z4wHhSy8N2nrmVi7M3SE/t/XtZwLV5h05og DALAWPkwzfuUMYqBAfzoHkiZCGR9Fv8ZiqjcQIHT3riEuJ8M5z3XsQI1hPfNbL7Qg7vn ZMag== X-Gm-Message-State: AOJu0YwQiYsbrwM8JwdYVVVPT6Eqp3FDdEZu7svkXbfB4vkHo2bSIGhC wssnKWv3tqw6jG/1G1+JZn26vBw4rkByrpUOkxtfkf8KrOC3CfPKLMBEGhhkdnweJcsbqZVT1CT +6eEivWqOoDFEI7L0pSStRH/ZMpBJWJPj/OAd2bhHDGiNJVLYpHI8c6WPvrD9OEXp1xSMpIcWkZ 3CqScIWGKMfsuQxYvv8CnK705rSF82ycPmRPRKzApQdyMhmA== X-Received: by 2002:a05:6e02:1807:b0:363:792c:ac43 with SMTP id a7-20020a056e02180700b00363792cac43mr3605310ilv.3.1706724578489; Wed, 31 Jan 2024 10:09:38 -0800 (PST) Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id v24-20020a634818000000b005cfbf96c733sm10876004pga.30.2024.01.31.10.09.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 10:09:38 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v5 1/3] eventpoll: support busy poll per epoll instance Date: Wed, 31 Jan 2024 18:08:03 +0000 Message-Id: <20240131180811.23566-2-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240131180811.23566-1-jdamato@fastly.com> References: <20240131180811.23566-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789630519784839771 X-GMAIL-MSGID: 1789630519784839771 Allow busy polling on a per-epoll context basis. The per-epoll context usec timeout value is preferred, but the pre-existing system wide sysctl value is still supported if it specified. Note that this change uses an xor: either per epoll instance busy polling is enabled on the epoll instance or system wide epoll is enabled. Enabling both is disallowed. Signed-off-by: Joe Damato --- fs/eventpoll.c | 49 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3534d36a1474..ce75189d46df 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -227,6 +227,8 @@ struct eventpoll { #ifdef CONFIG_NET_RX_BUSY_POLL /* used to track busy poll napi_id */ unsigned int napi_id; + /* busy poll timeout */ + u64 busy_poll_usecs; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -386,12 +388,44 @@ static inline int ep_events_available(struct eventpoll *ep) READ_ONCE(ep->ovflist) != EP_UNACTIVE_PTR; } +/** + * busy_loop_ep_timeout - check if busy poll has timed out. The timeout value + * from the epoll instance ep is preferred, but if it is not set fallback to + * the system-wide global via busy_loop_timeout. + * + * @start_time: The start time used to compute the remaining time until timeout. + * @ep: Pointer to the eventpoll context. + * + * Return: true if the timeout has expired, false otherwise. + */ +static inline bool busy_loop_ep_timeout(unsigned long start_time, struct eventpoll *ep) +{ +#ifdef CONFIG_NET_RX_BUSY_POLL + unsigned long bp_usec = READ_ONCE(ep->busy_poll_usecs); + + if (bp_usec) { + unsigned long end_time = start_time + bp_usec; + unsigned long now = busy_loop_current_time(); + + return time_after(now, end_time); + } else { + return busy_loop_timeout(start_time); + } +#endif + return true; +} + #ifdef CONFIG_NET_RX_BUSY_POLL +static bool ep_busy_loop_on(struct eventpoll *ep) +{ + return !!ep->busy_poll_usecs || net_busy_loop_on(); +} + static bool ep_busy_loop_end(void *p, unsigned long start_time) { struct eventpoll *ep = p; - return ep_events_available(ep) || busy_loop_timeout(start_time); + return ep_events_available(ep) || busy_loop_ep_timeout(start_time, ep); } /* @@ -404,7 +438,7 @@ static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); - if ((napi_id >= MIN_NAPI_ID) && net_busy_loop_on()) { + if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, BUSY_POLL_BUDGET); if (ep_events_available(ep)) @@ -430,7 +464,8 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) struct socket *sock; struct sock *sk; - if (!net_busy_loop_on()) + ep = epi->ep; + if (!ep_busy_loop_on(ep)) return; sock = sock_from_file(epi->ffd.file); @@ -442,7 +477,6 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) return; napi_id = READ_ONCE(sk->sk_napi_id); - ep = epi->ep; /* Non-NAPI IDs can be rejected * or @@ -466,6 +500,10 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) { } +static inline bool ep_busy_loop_on(struct eventpoll *ep) +{ + return false; +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -2058,6 +2096,9 @@ static int do_epoll_create(int flags) error = PTR_ERR(file); goto out_free_fd; } +#ifdef CONFIG_NET_RX_BUSY_POLL + ep->busy_poll_usecs = 0; +#endif ep->file = file; fd_install(fd, file); return fd; From patchwork Wed Jan 31 18:08:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 194893 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp2069751dyb; Wed, 31 Jan 2024 10:11:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IFSM/fHFDqYym4rhYrMSJh0QHivVBelG2BGeDZbaoxIwDGGJEBwAFXyP2Dp7HIruFsD//WO X-Received: by 2002:ac8:4e56:0:b0:42a:a88b:9ba1 with SMTP id e22-20020ac84e56000000b0042aa88b9ba1mr84105qtw.21.1706724670169; Wed, 31 Jan 2024 10:11:10 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706724670; cv=pass; d=google.com; s=arc-20160816; b=CVfNKfU6u3cl1JmDJ/2jx3Vzu/LarADZAO/izqGpA9cJYpPhYPS6ff8glHH5hklKyH hpVroMNAdaLs0OQ61+G788cV+ZQSyy67sLY3Epvn/4b41gJiRTR+P9Ws5O8qv/5VN3T3 qJhD0SyM45u0Q1iu0rt4TWWrA820gnArm3YYCEhnpwcJHpG5aeDop9x8OIrM6Dd9dekG nGfvjCP/YqaU3CP2x98IP08CslPlfQvqbeh2QBeBIIdblpdtRVc0cVD1ndy4fZFqaIDA CNT+56R9wIzpf2oXHpKdnELuJAg5QJl3WpsKEmBo3p4VnGpDwBDTR/wWNv5ab4G/gdx+ ZZBg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; fh=VdM51TEBxh2WibK9su0aPW+z0JGEPgswLbdWq9srmro=; b=dpnIVxfJIWwFaZpQ7p5uBGVo9MAHTHmoGux1PfUuU1iOmuO3+8Hb1m3H4yiT3/WBI5 BYtzUT3hoztoGkxSfgtbow6DbhD8nbH/oz/Habc7tS7tpyZyySvACyMs/fCK58omMYER ftiGSLwPCXdPdXq7hf+g7USDfve4ITjMsOZeVxaRnZr1MNQUKzkmHR9QuPJ/2iS06wcc nKJsECr6kS4veL8fzlT5fkh+/+1m1iNeaZMMUHgmdqhac7aKXXlhwH94E6+z2VtNw29c TLWURKVgse2IGbHZEWH9sVdEDO3Y4VMelySZ/cvNX8LUIxG1Ao85dFfVIB3cY+b6B16+ R66w==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=KnHVwhWv; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-46955-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46955-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCWO5YPMim7/Tdb+igy2y44XWQ7lV0Os1WjsBJ63yJ47LcawWnTeBaLEXacwgW+aKtpIwXU5lP3egsO9FXbh4qKi9Kd4YA== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id r19-20020ac85c93000000b0042a1138b665si12468990qta.783.2024.01.31.10.11.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 10:11:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-46955-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=KnHVwhWv; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-46955-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46955-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id E49181C20C9F for ; Wed, 31 Jan 2024 18:11:09 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 476DF13BEAB; Wed, 31 Jan 2024 18:09:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="KnHVwhWv" Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D95FD1386D7 for ; Wed, 31 Jan 2024 18:09:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706724583; cv=none; b=k71CF0m5cmr1iCbaj+6rTgz0kB3HGeNw+v7FSqhHuXPAeZMMr2RPpzV/MpGWF2cNr9xw/s9PgdhYImmFhOzvmdwNezgOwKFP+eCL7QuF2chtR2Qw+hFvQ21KDlesKej7rvkiIyQZiJdk8jG1duXxOjYddcMdmko/oalybCGYf+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706724583; c=relaxed/simple; bh=+NvwukKrlDHz6eylGdtOKZKD8hFQLLmD1KnjinUcTaA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cnu+uQ/WDx+flwoPk03E2KJcuxp5Rbb5iwE6QHSoIICwdKoIRFk7TMEXnwxK1c5sk1xHwqnNPx3Elvs4bIY6603ufze96roi4fQ/tSIM90IpOWdZs11Ufm0No5GYs+58j3ZnfQCtDrzgHJk7xiAxnJ6aBWgkQiS3MTzhrLn2GaA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=KnHVwhWv; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-598699c0f1eso52061eaf.2 for ; Wed, 31 Jan 2024 10:09:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1706724580; x=1707329380; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; b=KnHVwhWvgJevhVlcnikP7QUDjrpfFLbgEZhFUoIjHIFCR/MNbfMrJXrQ4IkSM5TrqK 2UDPfy1N3PUya8cOMl+64EFU1OoCdR0ueVBFh3poG/6pfowuA0LDBbpd2CEe4mxyO5Ik 4vT9JFBioIWaYPE9QI3DKf+tA05SI0ri3vzZQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706724580; x=1707329380; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dis0Zk83NVRgFQ7j5PMchP4NSpK2O2E2OiTy+PShPHQ=; b=n3lpDt61isn6LfJ3kpFLqjiQrKfK+G74K+tbGP8R9kZKRENJRzoF8W/qJ3E9wRiJlb lNqkgK8M0SnkUkWN7K6QyKDV1eI40Up+VZ+Y8IIvbRXkcq88g9rb/gTvG7HbOGzdQGQf eWVudv2ojM/t65bVwLitsXNekKEA+9apcciROhqRGz2TRYUzMyCwOb4XiV+lv9s63HFe hH1TS1LPHsJ1jJFCIhyDT4yy2ZyO4QtQP5ZQ2T+tuEmO9MSAOBU6H4gh3GTCbiHTnS9L cVAGgY+0fODos7IuuwXFlnZ42WuWnMenB9eYr4NyUHnorVkcXd30ZshkBqaNieKc7XHV i4AA== X-Gm-Message-State: AOJu0Yxjvq40F9uPTcFuyRyPOkqp+g/2A9W0Hc3Qz1c9lRYbOWEBwX6f 3XTzMQYoYaqgvwHGKZ9X0wvi+19iBZC9psKPOPVCQSX1f5HzOK2h7H9CyOYCGZUFIkAZ+9Cz7Ik iEcWGr/mIF7q32cTZAnSQSmlMHbzuvB2xOMR8TimKcCoD/EOrf9rJ9gk3ZkbLwOCB++t797evwm ZrBH/+GOf+IGaqjq+sjDutDfoSZOjn9EoEw+ePy5ao7IAb3A== X-Received: by 2002:a05:6358:4b4b:b0:176:915a:c7b7 with SMTP id ks11-20020a0563584b4b00b00176915ac7b7mr32979rwc.31.1706724580488; Wed, 31 Jan 2024 10:09:40 -0800 (PST) Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id v24-20020a634818000000b005cfbf96c733sm10876004pga.30.2024.01.31.10.09.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 10:09:40 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, Joe Damato , Alexander Viro , Jan Kara , linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v5 2/3] eventpoll: Add per-epoll busy poll packet budget Date: Wed, 31 Jan 2024 18:08:04 +0000 Message-Id: <20240131180811.23566-3-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240131180811.23566-1-jdamato@fastly.com> References: <20240131180811.23566-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789630528046013767 X-GMAIL-MSGID: 1789630528046013767 When using epoll-based busy poll, the packet budget is hardcoded to BUSY_POLL_BUDGET (8). Users may desire larger busy poll budgets, which can potentially increase throughput when busy polling under high network load. Other busy poll methods allow setting the busy poll budget via SO_BUSY_POLL_BUDGET, but epoll-based busy polling uses a hardcoded value. Fix this edge case by adding support for a per-epoll context busy poll packet budget. If not specified, the default value (BUSY_POLL_BUDGET) is used. Signed-off-by: Joe Damato --- fs/eventpoll.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/fs/eventpoll.c b/fs/eventpoll.c index ce75189d46df..3985434df527 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -229,6 +229,8 @@ struct eventpoll { unsigned int napi_id; /* busy poll timeout */ u64 busy_poll_usecs; + /* busy poll packet budget */ + u16 busy_poll_budget; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC @@ -437,10 +439,14 @@ static bool ep_busy_loop_end(void *p, unsigned long start_time) static bool ep_busy_loop(struct eventpoll *ep, int nonblock) { unsigned int napi_id = READ_ONCE(ep->napi_id); + u16 budget = READ_ONCE(ep->busy_poll_budget); + + if (!budget) + budget = BUSY_POLL_BUDGET; if ((napi_id >= MIN_NAPI_ID) && ep_busy_loop_on(ep)) { napi_busy_loop(napi_id, nonblock ? NULL : ep_busy_loop_end, ep, false, - BUSY_POLL_BUDGET); + budget); if (ep_events_available(ep)) return true; /* @@ -2098,6 +2104,7 @@ static int do_epoll_create(int flags) } #ifdef CONFIG_NET_RX_BUSY_POLL ep->busy_poll_usecs = 0; + ep->busy_poll_budget = 0; #endif ep->file = file; fd_install(fd, file); From patchwork Wed Jan 31 18:08:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joe Damato X-Patchwork-Id: 194894 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp2070101dyb; Wed, 31 Jan 2024 10:11:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IGdpw6j88MMwrENH3YgLmbG+N+3p77DmnydPKIXuXgmZuD+TrXlZLLTZgoDcyFq0kQQwz8m X-Received: by 2002:ac8:7fc6:0:b0:42b:f044:27eb with SMTP id b6-20020ac87fc6000000b0042bf04427ebmr1056158qtk.2.1706724698858; Wed, 31 Jan 2024 10:11:38 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706724698; cv=pass; d=google.com; s=arc-20160816; b=Yv5UndksIsdt0Hn28c54mdk73Rmyp2kkoK8wbkGKMGCQHIsVx73KPQVT6XGekMSRzo 3wEcxrFIzL5yTEBMTwvvH4vkMLn1rN6+FQrv/06SH4uvU4faEpIAm2enPNWLMbnXjX+1 kM6KEBQqHsG2Y4H2ZK3suiAnwQ+UkdNxFRPwlzr3fMI49Yywks3f0NoI1Er82SW0vqMX a77ZKM9XJODT6kmk6UxqD0WZoZej6g/6GcgXSDvvuLherjEtwsmuY8VfwJf7wpfWlD8h L7o8IeQ0TRSNDek9B/C4Rskbf/YITJl2wZKZ7yhTq+SNlNsA5b6kbY6vaW3Sy2Ijol7L FPaA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=eL7KL2gLPbAb4hjqxQ9bqO5kTIW0debt+d1tm+r3LDo=; fh=DXqQh9cSQlK9fa4r4fMWXgaL+5Ur9Sjgej/WfPXz6Ao=; b=W+bxL2Rk5+Hp4tmnge6tWEdqBk7+DN5W/u+it5C764csF/16UNftqRmwifosjVcxWP Fq8Ywp7BlxaWNRZJCGT+WbPxr5mzRUTopp2ztvolnUOeTKskTgjzr0dcqG2snRY1n6Rh mGYOyC69etOfSeeqgDsunbkeLG+kkn/aLAZj1zpDKYQv38LD/KX7c1tRP4F3vUf2glzH YwiUW1lpeY+WkR+bP8RW6WjAExvUPZ6T1+4Qwypkj4F70dtdPg3SUdZQl0LyVNzwhYbt Qcdhdm78FDJ075sF2zOnEoxDdfyN6wIdkc4LSQq767GatgM+2xf0zqky3XYbZ7brWRMW F1qw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=PMtahE2n; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-46956-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46956-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com X-Forwarded-Encrypted: i=1; AJvYcCWuMCoO+F1x0fMENuDNiQN2AtvX0YoYKA73qvA/+lq1TpPsTGRUZ17RgbruZshgdU6zcDnJvrpVCarMr2nXiZisyiFo0w== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id p15-20020a05622a13cf00b0042a86d14547si11763280qtk.691.2024.01.31.10.11.38 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 10:11:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-46956-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@fastly.com header.s=google header.b=PMtahE2n; arc=pass (i=1 spf=pass spfdomain=fastly.com dkim=pass dkdomain=fastly.com dmarc=pass fromdomain=fastly.com); spf=pass (google.com: domain of linux-kernel+bounces-46956-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46956-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=fastly.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 819991C20753 for ; Wed, 31 Jan 2024 18:11:38 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8E6A413D50B; Wed, 31 Jan 2024 18:09:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b="PMtahE2n" Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 680B713BEAF for ; Wed, 31 Jan 2024 18:09:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706724590; cv=none; b=h56z2cNjniMALUwWVnmZWqDnbq8Fyx9vvnIZ7+xbvkZaX16r4iBYdoVThdRYn8h/t55h3wWfUT4uenPWmqWTXJxJmVDf4lVD8wdtpzHxcn9K8dayEaZK5T8qQKMbL7MmJVgKT2/kcAVWT+65n+kjd1cQ1XP9v9IZxKcITUjuON0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706724590; c=relaxed/simple; bh=7A7sHroL9Ow9RB5SlrBbD5eZxpWYWwUbOvldYd9/zgE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Adc7eyRGUq5v15G0T39achKMt5JdRGKYwsLAJlCgBOmlL9q4HOBacVTa+1clx7P4yavwhsMyjr84fYlf8ia39M2sa4+7w6uujAA+7T8kUT7+5UvvIAT3dPBMjBRT5GkGDfOwnNq5q+3I/G8r3hWqPHgOuhlKOom56tnqp+d3FCY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com; spf=pass smtp.mailfrom=fastly.com; dkim=pass (1024-bit key) header.d=fastly.com header.i=@fastly.com header.b=PMtahE2n; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=fastly.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=fastly.com Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-599f5e71d85so44271eaf.3 for ; Wed, 31 Jan 2024 10:09:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastly.com; s=google; t=1706724586; x=1707329386; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eL7KL2gLPbAb4hjqxQ9bqO5kTIW0debt+d1tm+r3LDo=; b=PMtahE2nzTI6NrcNsv9ux64CWk/ga8Ey3k8fsDCJ0ka8O1FdG2frROY36f64eLQRJ/ qekG4muZzU9V9FEvI5Yg+W0zvkg56aWQotkl/G0V3QG+ifqSLBdvS2sQqPcj/IKFPRbm d//41RqxlWTDNRZYpxI8z5FEEPwkD5Tt3cmw0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706724586; x=1707329386; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eL7KL2gLPbAb4hjqxQ9bqO5kTIW0debt+d1tm+r3LDo=; b=OrExe0Li+R3LYsXTlc52giSRKcmhfENnT3/WP7sSvwd+w81npmoaBdoVVKD9c2fMru LGgK74AZF6EL19P3X9GHEnrm7J7WuNPwHQA+epcWuKv1DI2JtbWxrYMDcuzWp9t/4yff PFnXBCYdrR0PZKDP2W5zdquJh1vjtYvl/9vmHdgU5DRpYkG6hFaFLAhoI/Rki3ysw/Pc TncVF+51S7iScC9ay3dczaakGyPtDVZl74HEcn9bV0az7z2u0l9/DX8NohlRlX0OfKb5 tBf7HBu+x92ki6wXy9CBf4DbYuKl1LKRVaQxQgbLrzDTE+W6argdhY3uLlDT4Qk4Yibs 2stw== X-Gm-Message-State: AOJu0Yw+N/CNtJC1oxv5uFm3xSuDz8wP59nyAog62F3sbz0FILOqe/r9 BBe+PM3gM6tRG5p7bnQ2P0hZuWTEv4ljS6jxixRafhW/kKgdC4T61rEoaeQQGAhpslq2nBm2oSg aMRc0FNnWhP1lsNlRhZikpS9O2NzHXHzSso27jA47Lge0YDjEbCDtz47LWBqEuYdjNOSi3o9Eht S7TCwwat8wPXOPA5eFMXzGFh4XmM88jZjuqSY/BwQArsls0Q== X-Received: by 2002:a05:6870:8a1f:b0:218:d3a2:2430 with SMTP id p31-20020a0568708a1f00b00218d3a22430mr1705910oaq.22.1706724586595; Wed, 31 Jan 2024 10:09:46 -0800 (PST) Received: from localhost.localdomain ([2620:11a:c018:0:ea8:be91:8d1:f59b]) by smtp.gmail.com with ESMTPSA id v24-20020a634818000000b005cfbf96c733sm10876004pga.30.2024.01.31.10.09.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 10:09:46 -0800 (PST) From: Joe Damato To: linux-kernel@vger.kernel.org, netdev@vger.kernel.org Cc: chuck.lever@oracle.com, jlayton@kernel.org, linux-api@vger.kernel.org, brauner@kernel.org, edumazet@google.com, davem@davemloft.net, alexander.duyck@gmail.com, sridhar.samudrala@intel.com, kuba@kernel.org, willemdebruijn.kernel@gmail.com, weiwan@google.com, David.Laight@ACULAB.COM, arnd@arndb.de, Joe Damato , Jonathan Corbet , Alexander Viro , Jan Kara , Greg Kroah-Hartman , Nathan Lynch , Michael Ellerman , Dominik Brodowski , Steve French , Julien Panis , Thomas Zimmermann , Jiri Slaby , Thomas Huth , Albert Ou , Andrew Waterman , Palmer Dabbelt , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-fsdevel@vger.kernel.org (open list:FILESYSTEMS (VFS and infrastructure)) Subject: [PATCH net-next v5 3/3] eventpoll: Add epoll ioctl for epoll_params Date: Wed, 31 Jan 2024 18:08:05 +0000 Message-Id: <20240131180811.23566-4-jdamato@fastly.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240131180811.23566-1-jdamato@fastly.com> References: <20240131180811.23566-1-jdamato@fastly.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789630557620615663 X-GMAIL-MSGID: 1789630557620615663 Add an ioctl for getting and setting epoll_params. User programs can use this ioctl to get and set the busy poll usec time or packet budget params for a specific epoll context. Parameters are limited: - busy_poll_usecs is limited to <= u32_max - busy_poll_budget is limited to <= NAPI_POLL_WEIGHT by unprivileged users (!capable(CAP_NET_ADMIN)). - __pad must be 0 Signed-off-by: Joe Damato --- .../userspace-api/ioctl/ioctl-number.rst | 1 + fs/eventpoll.c | 68 +++++++++++++++++++ include/uapi/linux/eventpoll.h | 12 ++++ 3 files changed, 81 insertions(+) diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index 457e16f06e04..b33918232f78 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -309,6 +309,7 @@ Code Seq# Include File Comments 0x89 0B-DF linux/sockios.h 0x89 E0-EF linux/sockios.h SIOCPROTOPRIVATE range 0x89 F0-FF linux/sockios.h SIOCDEVPRIVATE range +0x8A 00-1F linux/eventpoll.h 0x8B all linux/wireless.h 0x8C 00-3F WiNRADiO driver diff --git a/fs/eventpoll.c b/fs/eventpoll.c index 3985434df527..96efca6a9238 100644 --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -37,6 +37,7 @@ #include #include #include +#include #include /* @@ -495,6 +496,45 @@ static inline void ep_set_busy_poll_napi_id(struct epitem *epi) ep->napi_id = napi_id; } +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct eventpoll *ep; + struct epoll_params epoll_params; + void __user *uarg = (void __user *) arg; + + ep = file->private_data; + + switch (cmd) { + case EPIOCSPARAMS: + if (copy_from_user(&epoll_params, uarg, sizeof(epoll_params))) + return -EFAULT; + + if (memchr_inv(epoll_params.__pad, 0, sizeof(epoll_params.__pad))) + return -EINVAL; + + if (epoll_params.busy_poll_usecs > U32_MAX) + return -EINVAL; + + if (epoll_params.busy_poll_budget > NAPI_POLL_WEIGHT && + !capable(CAP_NET_ADMIN)) + return -EPERM; + + ep->busy_poll_usecs = epoll_params.busy_poll_usecs; + ep->busy_poll_budget = epoll_params.busy_poll_budget; + return 0; + case EPIOCGPARAMS: + memset(&epoll_params, 0, sizeof(epoll_params)); + epoll_params.busy_poll_usecs = ep->busy_poll_usecs; + epoll_params.busy_poll_budget = ep->busy_poll_budget; + if (copy_to_user(uarg, &epoll_params, sizeof(epoll_params))) + return -EFAULT; + return 0; + default: + return -ENOIOCTLCMD; + } +} + #else static inline bool ep_busy_loop(struct eventpoll *ep, int nonblock) @@ -510,6 +550,12 @@ static inline bool ep_busy_loop_on(struct eventpoll *ep) { return false; } + +static long ep_eventpoll_bp_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + return -EOPNOTSUPP; +} #endif /* CONFIG_NET_RX_BUSY_POLL */ /* @@ -869,6 +915,26 @@ static void ep_clear_and_put(struct eventpoll *ep) ep_free(ep); } +static long ep_eventpoll_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + int ret; + + if (!is_file_epoll(file)) + return -EINVAL; + + switch (cmd) { + case EPIOCSPARAMS: + case EPIOCGPARAMS: + ret = ep_eventpoll_bp_ioctl(file, cmd, arg); + break; + default: + ret = -EINVAL; + break; + } + + return ret; +} + static int ep_eventpoll_release(struct inode *inode, struct file *file) { struct eventpoll *ep = file->private_data; @@ -975,6 +1041,8 @@ static const struct file_operations eventpoll_fops = { .release = ep_eventpoll_release, .poll = ep_eventpoll_poll, .llseek = noop_llseek, + .unlocked_ioctl = ep_eventpoll_ioctl, + .compat_ioctl = compat_ptr_ioctl, }; /* diff --git a/include/uapi/linux/eventpoll.h b/include/uapi/linux/eventpoll.h index cfbcc4cc49ac..98e5ea525dd0 100644 --- a/include/uapi/linux/eventpoll.h +++ b/include/uapi/linux/eventpoll.h @@ -85,4 +85,16 @@ struct epoll_event { __u64 data; } EPOLL_PACKED; +struct epoll_params { + __aligned_u64 busy_poll_usecs; + __u16 busy_poll_budget; + + /* pad the struct to a multiple of 64bits for alignment on all arches */ + __u8 __pad[6]; +}; + +#define EPOLL_IOC_TYPE 0x8A +#define EPIOCSPARAMS _IOW(EPOLL_IOC_TYPE, 0x01, struct epoll_params) +#define EPIOCGPARAMS _IOR(EPOLL_IOC_TYPE, 0x02, struct epoll_params) + #endif /* _UAPI_LINUX_EVENTPOLL_H */