From patchwork Thu Dec 22 22:12:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 35971 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp16028wrn; Thu, 22 Dec 2022 14:16:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXu4EXjWKTtFxQ/ijOYpk1Xjdx+MkFxk8+/BoRhp2DNUazgmuPKxM7LQrnx81dkstLWHr5T7 X-Received: by 2002:aa7:924d:0:b0:57e:866d:c095 with SMTP id 13-20020aa7924d000000b0057e866dc095mr8015005pfp.25.1671747390029; Thu, 22 Dec 2022 14:16:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671747390; cv=none; d=google.com; s=arc-20160816; b=oXeM/JER/5jaUl1jSSryMSB+8OwDVHNwdYitSn0QUlwA4lDMFopICLw6QzxAx9+fmy 0WsshZ32Jyv3ZSSw8CehW8OOv1AB9UtHrJJ0kpyrM21WVnfxI7qhE1k/B5EpuLPGuqDm TY0GmHIfeiDlhY9jUpEDwNSyJwwkq2xNGnQS0spSoyOIMi5978rVI4xPFFfNAL7hWrZL 5mDLUJ9orTQ6r1JE/pmgk9On2Xiow7yfLt00J2WBkYqjkSlkH7k/NERmKt5gWvICSqrR PXJj+VUfge7696jjQN6Q9/yimVpd6f+QI2XAwD+HIJTwpToLQNsvlWm2zow+GVuFFkzP rJkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JSmXL/dXjNPK8Y3ZkG4X3YTX6GbgYrDbG5RWIoLienk=; b=uGDaDMb4r5aUqeMoReeuFhakkKw6miALEQi8NVct3yXqmvQFy4Tu2g04HUqrI5VnBl 8at1jA4cLoXuSlmrGDWktcFKpon5HduYFQmCXtrGlwyJXGnblM0Oyf/Eqy6KeQOSubCo pxcobKQYWB5Nnb+mkFaJxI+IpnIggbVa4ltdV7k1wanna4C9evAQMqfay6iOn9WMAAuF RJkGG8WQWL+uCwAK5ElhQlMTp+CVoh/35yA+Add2ZE4KRSJYAb9Tq9wIOLY7T/aouDlF 3P2I30a1jpissf6ulVVDQcsG2MoV0WQAAxUuiYAQOPkLPv6ucqemG24m2zlTwnGe+WFa lHYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=axJZ+bqr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 10-20020a62140a000000b00577d10d6eaesi1560777pfu.169.2022.12.22.14.16.16; Thu, 22 Dec 2022 14:16:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=axJZ+bqr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229951AbiLVWMv (ORCPT + 99 others); Thu, 22 Dec 2022 17:12:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229627AbiLVWMt (ORCPT ); Thu, 22 Dec 2022 17:12:49 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEB8121E28; Thu, 22 Dec 2022 14:12:48 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6209861DAE; Thu, 22 Dec 2022 22:12:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D6ABC433EF; Thu, 22 Dec 2022 22:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1671747167; bh=6/3Z9VJ1qfYuDDjUhrIE3LXc4NT3VetuZ5KotX+3+58=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=axJZ+bqrgkFrKPJvPBzSSfx/QTBG34Fme1UeVDMTF+N3TEa3nbuov7nBj8Ff9ycRL rKXiubu2M5nCya5ElgsumAzZ8YvjVl9bCqLjXpolV+b+VVJbmqkOW2A/o2XqWbuYev G1cnGBCZtFIj10TMPRE6di3qTSGhHpYUnTfiPSedQO66GibXATNTcr+SXJU9x+9pCx tS8C8JHNfMPDaana1h1HVYkwpMkjKV7pDa7IlxIfK8PjQ5b4/9ViyMcQHqofW4V0Fd AtlnEVtE8ijGsEws3uF2IkfeITjTReDOUXi8bDcQ79F0soE1ZyMy3DPaWJyN6Vs8lf sMTspFS7s76Nw== From: Jakub Kicinski To: peterz@infradead.org, tglx@linutronix.de Cc: jstultz@google.com, edumazet@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 1/3] softirq: rename ksoftirqd_running() -> ksoftirqd_should_handle() Date: Thu, 22 Dec 2022 14:12:42 -0800 Message-Id: <20221222221244.1290833-2-kuba@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221222221244.1290833-1-kuba@kernel.org> References: <20221222221244.1290833-1-kuba@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752954191472704631?= X-GMAIL-MSGID: =?utf-8?q?1752954191472704631?= ksoftirqd_running() takes the high priority softirqs into consideration, so ksoftirqd_should_handle() seems like a better name. Signed-off-by: Jakub Kicinski --- kernel/softirq.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..00b838d566c1 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -86,7 +86,7 @@ static void wakeup_softirqd(void) * unless we're doing some of the synchronous softirqs. */ #define SOFTIRQ_NOW_MASK ((1 << HI_SOFTIRQ) | (1 << TASKLET_SOFTIRQ)) -static bool ksoftirqd_running(unsigned long pending) +static bool ksoftirqd_should_handle(unsigned long pending) { struct task_struct *tsk = __this_cpu_read(ksoftirqd); @@ -236,7 +236,7 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt) goto out; pending = local_softirq_pending(); - if (!pending || ksoftirqd_running(pending)) + if (!pending || ksoftirqd_should_handle(pending)) goto out; /* @@ -432,7 +432,7 @@ static inline bool should_wake_ksoftirqd(void) static inline void invoke_softirq(void) { - if (ksoftirqd_running(local_softirq_pending())) + if (ksoftirqd_should_handle(local_softirq_pending())) return; if (!force_irqthreads() || !__this_cpu_read(ksoftirqd)) { @@ -468,7 +468,7 @@ asmlinkage __visible void do_softirq(void) pending = local_softirq_pending(); - if (pending && !ksoftirqd_running(pending)) + if (pending && !ksoftirqd_should_handle(pending)) do_softirq_own_stack(); local_irq_restore(flags); From patchwork Thu Dec 22 22:12:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 35972 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp16492wrn; Thu, 22 Dec 2022 14:17:46 -0800 (PST) X-Google-Smtp-Source: AMrXdXtO63LP3h/F28I8bH4aPyBPOC5+INCvjjKmakO04T73nBMwIIlNkY7/ZTkQffOuLeCJ/QIs X-Received: by 2002:a17:902:cec2:b0:185:441e:6ef3 with SMTP id d2-20020a170902cec200b00185441e6ef3mr10719904plg.61.1671747465738; Thu, 22 Dec 2022 14:17:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671747465; cv=none; d=google.com; s=arc-20160816; b=XxGWF+7GYmgDpoG/U91y98Rm3RqZ5xdm+uoBewWm66zp1/r/UMAd3eLMzUh0YlTD3p TO3tcFlcTEkBESXG6T2RdVuxbuVQ1Ky2MT5f7XiNrVOjLK1GGIjb4IlLsY7rkzGxVGO4 IweNcSEHEJ80SxnpWBq95ZFU5Z0T32+KbL9EvdMbIic9L0nsAm1wbqSifLJTOy5z5DLJ Y6MrjtuzAtmAdXc2ajYRmHPP6QNKwJ0m8hnGAP8aQb92VlyLiWeH/xJt8ueIvW/qr/BJ /RPGMIItj3Puqzy0FqpezsGv4R6X4Cf9FOm3L0Xv7pBpjE/TSdN1i4e7Tm4BdidogrC0 wPUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Gdrqk6CHEokwmtF3LJJlAyk9Cea6yBlXw6WVFiPdmno=; b=Iwa9wqEE/dmvJ9mWMyYYn1tvfP1y5jops20TVNL/AR4zeUo26tRYomLowKl/LMjOk1 vuJF8cIPScKlPIERX8wJm0VbsMVe/9Ra3hagJ8+tUzi2JOq2XgQ0RBtzCXEU3OEManUp zY66BWUFNY9pqNA+FbPtCbqDksDqwW2kMDPE0STZDgg57l3ux4N2fZcLvjnPP2wBmP04 fikwdkKBjyU3BpFDxgNnnm53rMyGXVqAzbMzNjwoTaPqzFv8WE4wk9KNs7RFgL9V6JNW BBCkFhaH7v1WFf+CLDCDt6IS66V4DXr4X31G8UHUM4Dn+7RRjda7JndOhDcWedxzRdwX 6MZQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="N/XYYW2G"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c18-20020a170902c1d200b00189de92ab32si1584344plc.22.2022.12.22.14.17.31; Thu, 22 Dec 2022 14:17:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b="N/XYYW2G"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230347AbiLVWMz (ORCPT + 99 others); Thu, 22 Dec 2022 17:12:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229666AbiLVWMt (ORCPT ); Thu, 22 Dec 2022 17:12:49 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 35BF524BCD; Thu, 22 Dec 2022 14:12:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BFA8461DB1; Thu, 22 Dec 2022 22:12:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E8AA8C43392; Thu, 22 Dec 2022 22:12:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1671747168; bh=D5KSQncHbyZRvS0KOGabofpFDBc6Ejaiaupvz+j2wyA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N/XYYW2GuzaLDUnRG8/SOo/WZrhBXMZ9XxOtpoxV40Qx0lWnlbL4ZRqqyp8pyLo9A 2YPpejBNuVp0k0bzm826qR9jwXAuAD5Wq/nBU82scAxNYZf2LgXb5bxUivd7Nh33Rr Z17UpYi+6RtO4litzdZ2sVIcXTgxM0NCNeiSaI8l7Vitnxx0IWACG1bs99DDV9lgg+ +1f1zQ5KTTv8zLk5tarwUYsN4pB9TsaSGPfAOIXbq1T5vvA9MjlF6BF8JN69ZL9w3a dj1CE0klsCxKDMRIErGSwQzwWdY75b//Q1mmg2ra57QjUUOI9GXCTqLDyDksTLHwmi oi+8El0DcWohg== From: Jakub Kicinski To: peterz@infradead.org, tglx@linutronix.de Cc: jstultz@google.com, edumazet@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 2/3] softirq: avoid spurious stalls due to need_resched() Date: Thu, 22 Dec 2022 14:12:43 -0800 Message-Id: <20221222221244.1290833-3-kuba@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221222221244.1290833-1-kuba@kernel.org> References: <20221222221244.1290833-1-kuba@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752954270554657401?= X-GMAIL-MSGID: =?utf-8?q?1752954270554657401?= need_resched() added in commit c10d73671ad3 ("softirq: reduce latencies") does improve latency for real workloads (for example memcache). Unfortunately it triggers quite often even for non-network-heavy apps (~900 times a second on a loaded webserver), and in small fraction of cases whatever the scheduler decided to run will hold onto the CPU for the entire time slice. 10ms+ stalls on a machine which is not actually under overload cause erratic network behavior and spurious TCP retransmits. Typical end-to-end latency in a datacenter is < 200us so its common to set TCP timeout to 10ms or less. The intent of the need_resched() is to let a low latency application respond quickly and yield (to ksoftirqd). Put a time limit on this dance. Ignore the fact that ksoftirqd is RUNNING if we were trying to be nice and the application did not yield quickly. On a webserver loaded at 90% CPU this change reduces the numer of 8ms+ stalls the network softirq processing sees by around 10x (2/sec -> 0.2/sec). It also seems to reduce retransmissions by ~10% but the data is quite noisy. Signed-off-by: Jakub Kicinski --- kernel/softirq.c | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index 00b838d566c1..ad200d386ec1 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -59,6 +59,7 @@ EXPORT_PER_CPU_SYMBOL(irq_stat); static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp; DEFINE_PER_CPU(struct task_struct *, ksoftirqd); +static DEFINE_PER_CPU(unsigned long, overload_limit); const char * const softirq_to_name[NR_SOFTIRQS] = { "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "IRQ_POLL", @@ -89,10 +90,15 @@ static void wakeup_softirqd(void) static bool ksoftirqd_should_handle(unsigned long pending) { struct task_struct *tsk = __this_cpu_read(ksoftirqd); + unsigned long ov_limit; if (pending & SOFTIRQ_NOW_MASK) return false; - return tsk && task_is_running(tsk) && !__kthread_should_park(tsk); + if (likely(!tsk || !task_is_running(tsk) || __kthread_should_park(tsk))) + return false; + + ov_limit = __this_cpu_read(overload_limit); + return time_is_after_jiffies(ov_limit); } #ifdef CONFIG_TRACE_IRQFLAGS @@ -492,6 +498,9 @@ asmlinkage __visible void do_softirq(void) #define MAX_SOFTIRQ_TIME msecs_to_jiffies(2) #define MAX_SOFTIRQ_RESTART 10 +#define SOFTIRQ_OVERLOAD_TIME msecs_to_jiffies(100) +#define SOFTIRQ_DEFER_TIME msecs_to_jiffies(2) + #ifdef CONFIG_TRACE_IRQFLAGS /* * When we run softirqs from irq_exit() and thus on the hardirq stack we need @@ -588,10 +597,16 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) pending = local_softirq_pending(); if (pending) { - if (time_before(jiffies, end) && !need_resched() && - --max_restart) + unsigned long limit; + + if (time_is_before_eq_jiffies(end) || !--max_restart) + limit = SOFTIRQ_OVERLOAD_TIME; + else if (need_resched()) + limit = SOFTIRQ_DEFER_TIME; + else goto restart; + __this_cpu_write(overload_limit, jiffies + limit); wakeup_softirqd(); } From patchwork Thu Dec 22 22:12:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 35973 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp16747wrn; Thu, 22 Dec 2022 14:18:35 -0800 (PST) X-Google-Smtp-Source: AMrXdXsbtA2b4k0yRxif8uFuMNT1BEWgx/ZilUIVdwOI1cKmxGiq5TzjUDO+5Tjbr5fO+XPBH1JW X-Received: by 2002:a05:6a20:93a6:b0:ad:b14c:9e85 with SMTP id x38-20020a056a2093a600b000adb14c9e85mr9955560pzh.46.1671747515310; Thu, 22 Dec 2022 14:18:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1671747515; cv=none; d=google.com; s=arc-20160816; b=Jf6q0//sP2yU1SY4ZlRLMgT1dVafo8ehqPJ8PZCSvuArfLS6oTkKVgnBZiLFFzLKYq MIYiIjcq7YS97e+0xv2iQR8N8KeDN8xY5MC91kdQR1cLtjJRvE/jZrVeDTo8epIlblRj c3uvgo+gpxZaLUV4KNhthvTGujg/4daA6fYXGRF0dLL+GNoFUdN4GjdLaEuX2GXiFEpR 7gFxV1JVYZ6UL5Qzi3E4fBFQ9Shv7+/n7E0azq/x5CSB1zJOJzfpWF3ctaq8A2W1OMgA yn85Gh2whGUNglxekBp3QT90GJIDsG7yzAVbXixapQLoTy36oL2xVkUZT/ZSCdReomTT Rk0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CJO9JeOiIBCfR6ZrnazWXarOCGmD89IQ5WpffzXjH+8=; b=Zf16l+NZ3b/vd9KQwedzpaRdz6KB09TZ8mGEoYnRc9FO/qED4rVSbYiTO9gtWR0dpY bwHonizaIZFZocW+/k11XkD7nK+c0o1KahX2p6KTleb5NMs80EMPLGVezkxEPgG0q6in cHHzfWU2B4fS/6Uo7+S0hOZM2f+EqiI3WXEtdo9eYhONlU8WyGlSjBr5OqcrTQ6BBX8i 4lgOgYWbKwyCtkL1JCh0cuD4gpBt/Ua7Z4RM88WUVxmrokYsBnuWxSEfOUXtJNMJx/V+ 58lvGcrbLDZCUGw4fyaegaeJVf6yNm78aepKlR6Ag/hWIWf2b877X7nhrE93ZX/F52Qo 888Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=XQqMj6Z4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f5-20020a056a00228500b00574fcd99824si1898072pfe.73.2022.12.22.14.18.17; Thu, 22 Dec 2022 14:18:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=XQqMj6Z4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235669AbiLVWM7 (ORCPT + 99 others); Thu, 22 Dec 2022 17:12:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbiLVWMu (ORCPT ); Thu, 22 Dec 2022 17:12:50 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B86A10FDB; Thu, 22 Dec 2022 14:12:49 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2377C61DB4; Thu, 22 Dec 2022 22:12:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 518A6C433F2; Thu, 22 Dec 2022 22:12:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1671747168; bh=mQD2ZYw4YRduH/pzuxEiNaEpTsT1YNKaQ62QM19DZGQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XQqMj6Z4it5dSFIYft/H2Xzfs/wXS3w0orSN/1N6zaT/JSyNX+MYxUMbSYWJb9qYs MSKBbg7QoMfXqkYroZVfTbb7AAbk30WPU5M1BbT3jayFvOocStnYrnMcrVyj0QPZB0 9wn1QMIuHbt1FRas8Tec/sEx7+abSc2//V5Qaep3OMpz2jNJt4jWu4RjRr1z3kzjcm K6l5kMN0ViZaNaWxzH8bc65XUL49e+TYdhLQnVgqmjx6Op02k6yrBr5FfRDBqPzDBl BopxF+W1aJmuDKVqwz7T09xdXnCm+8FgHIjxNkfLtL8byI++EUno7vjf2agYZZ66yP Z35qEt0tuQneg== From: Jakub Kicinski To: peterz@infradead.org, tglx@linutronix.de Cc: jstultz@google.com, edumazet@google.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Jakub Kicinski Subject: [PATCH 3/3] softirq: don't yield if only expedited handlers are pending Date: Thu, 22 Dec 2022 14:12:44 -0800 Message-Id: <20221222221244.1290833-4-kuba@kernel.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221222221244.1290833-1-kuba@kernel.org> References: <20221222221244.1290833-1-kuba@kernel.org> MIME-Version: 1.0 X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752954322449672695?= X-GMAIL-MSGID: =?utf-8?q?1752954322449672695?= In networking we try to keep Tx packet queues small, so we limit how many bytes a socket may packetize and queue up. Tx completions (from NAPI) notify the sockets when packets have left the system (NIC Tx completion) and the socket schedules a tasklet to queue the next batch of frames. This leads to a situation where we go thru the softirq loop twice. First round we have pending = NET (from the NIC IRQ/NAPI), and the second iteration has pending = TASKLET (the socket tasklet). On two web workloads I looked at this condition accounts for 10% and 23% of all ksoftirqd wake ups respectively. We run NAPI which wakes some process up, we hit need_resched() and wake up ksoftirqd just to run the TSQ (TCP small queues) tasklet. Tweak the need_resched() condition to be ignored if all pending softIRQs are "non-deferred". The tasklet would run relatively soon, anyway, but once ksoftirqd is woken we're risking stalls. I did not see any negative impact on the latency in an RR test on a loaded machine with this change applied. Signed-off-by: Jakub Kicinski --- kernel/softirq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index ad200d386ec1..4ac59ffb0d55 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -601,7 +601,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) if (time_is_before_eq_jiffies(end) || !--max_restart) limit = SOFTIRQ_OVERLOAD_TIME; - else if (need_resched()) + else if (need_resched() && pending & ~SOFTIRQ_NOW_MASK) limit = SOFTIRQ_DEFER_TIME; else goto restart;