From patchwork Wed Nov 16 07:59:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 20765 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp14357wru; Wed, 16 Nov 2022 00:03:08 -0800 (PST) X-Google-Smtp-Source: AA0mqf6ChLmPwidnv87RuyxKAfMUUSpNboJRUImK/bO/IMwzIIpChV40ywBmO787GT+PZNg7M8LJ X-Received: by 2002:a17:90a:67c8:b0:210:9858:2b2c with SMTP id g8-20020a17090a67c800b0021098582b2cmr2517953pjm.191.1668585788256; Wed, 16 Nov 2022 00:03:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668585788; cv=none; d=google.com; s=arc-20160816; b=zXHNvOJzVFOXZhoWLST5ExhxdS9XlAiVBZJDF5lDYuAETdP5IH1OtXB6EDqxzEQ+lK hiXSN6MfPbb9+T7saGCj0cRpn0X9eeIF2kKT9MVbm68H26wlMylqgGxJkMcFBCGWpvvz 3lYqNMyF2hqwYj6nzdSmKBnFKJq6asawBXpOOA3xMs5rQuEUYIExnmN/d/iS7BAiBsZA 7C6RVonMWW/MyhKG/3uC8ZiepKq4kbvGcflj/WA0UmjUgrRzRoKs1Y+XXhLKhhOIMbD0 ScypHQGOSlKgsWEMjvMYIzM3gYh092xPJa6kBzO7M2KXS9UsCWw0JwyeV7M9gsoWOtdS j10Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=7v71ZLMMw9Ypk/qBIbo+9W6doSfdrcneW8rCdbD8mEA=; b=PkQCwm8llTlzkmgTzMt9+o/oelez5IKCpfanTofpirla7koehhlQWPgnHmUUWopHUa DHwaC/rqOyZPoSL9Fhi+y0Sn2OunOkPPZuns0KtbS1/ww7z/N6jxLCMzK1GUTezYTK8W +kH6+sVkMGzJMj6Xfgx8Xd4Ja5Z8icP8AQIGfwfAFnRKNLWlfJCKP1/pz7rdb7wH3Vty XLdjBpeB+yG1YdPr342U0mYuQsewZlfE0AfU7SDbYJ0a+MKzS/fpmA1qFvcTxRZjwpRv SKAmJBbPbXjs511mCpQVuPrE1mmY8llXqgyMHXo0wKiALt4UpoS7pU1dK0nJ+iiVjcQx AnIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=aVyjfoFa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a11-20020a170902eccb00b001729cfb3cb5si17161009plh.610.2022.11.16.00.02.54; Wed, 16 Nov 2022 00:03:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=aVyjfoFa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232867AbiKPH7m (ORCPT + 99 others); Wed, 16 Nov 2022 02:59:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232346AbiKPH7i (ORCPT ); Wed, 16 Nov 2022 02:59:38 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 801F1C7 for ; Tue, 15 Nov 2022 23:59:37 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id om10-20020a17090b3a8a00b002108b078ab1so1167530pjb.9 for ; Tue, 15 Nov 2022 23:59:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7v71ZLMMw9Ypk/qBIbo+9W6doSfdrcneW8rCdbD8mEA=; b=aVyjfoFajZKLOg+r2F+ZQPrKrjySyvIWcFKpJ4d1e/CHJ+bdY58v7gW+f+Uko6nH3q RP7ZRjV924KcAXvTR872pThBb1PBfZDwI7ClhwK6GnQXqxyGtxoLvgHaaQnmdTqo3wuF 0geX7hpH8jUu5+RwxicO37D214kmNIZ77WZQsMgg/OUarnykVJDAR5LSF6U2kPfeyvG8 tTzIZfYtvRtuBhA69pk4ncCYsPHVvjW1YNHQI6BN8e2pKkTolxwRmWGq6FF2A2xn2g96 ysvANC+S6GVrtMlLqB6D/tngj+IkyohmnBIV5/Oxeb1stneOUaM93jc8AJk9FPlW6aYM MSPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7v71ZLMMw9Ypk/qBIbo+9W6doSfdrcneW8rCdbD8mEA=; b=K8RH5Ww4LEbl/qx6nQIRw+h0xgfOtFaQ7Y30nhdxNn6P4xzUKT5Ai2w/TETKeHCWLq U6GVer2wh0S/AE1pf6VOBsJRBTU1O45QnBY0Kt06UXP1PHiZzTvTHeF7HXPsitFcWG++ cBIRSBTOrWf9FxmSszRVuYVGJJBa65OdlIgF3pV9VwOLAbUyJ3HPBoeRZ3H0XqOuYSBJ 2ofWvaB+0vXEv4svKN2s9kKqobPRiHaWNivIFOcaVFTcSbeujmfZ0Hw1EYjginrSxDAZ gJ83hCZOACuu6UakbG4yMJcqt1X5WSAnsbirKzpw9MyCR9YUwtPANysGSuTAVkNNTIQz w9/g== X-Gm-Message-State: ANoB5pmqt+eQEwnAmhl03leP5yijnD5KrSMarJNS7Z6xz8ythv1DZ8G7 vDebokOf0tKy2sOx/7oX7nakB2a0uZb0dX+sBr6TweX8E6ww7NEZHfbGgyG3dC96lT3+KvIumYJ a2pHNaMx89bFDRs5vOiofn4dsANmVQfDXFAJtRDfcp0w1a3sKjTwLkfrR6krSBGEbdSV5AnE= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:902:8d90:b0:187:3bc8:fc56 with SMTP id v16-20020a1709028d9000b001873bc8fc56mr7837879plo.83.1668585576869; Tue, 15 Nov 2022 23:59:36 -0800 (PST) Date: Wed, 16 Nov 2022 07:59:26 +0000 In-Reply-To: <20221116075929.453876-1-jstultz@google.com> Mime-Version: 1.0 References: <20221116075929.453876-1-jstultz@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221116075929.453876-2-jstultz@google.com> Subject: [PATCH v5 1/3] softirq: Add generic accessor to percpu softirq_pending data From: John Stultz To: LKML Cc: John Stultz , John Dias , "Connor O'Brien" , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , Joel Fernandes , Alexander Gordeev , kernel-team@android.com, kernel test robot X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749639011154109964?= X-GMAIL-MSGID: =?utf-8?q?1749639011154109964?= In a previous iteration of this patch series, I was checking: per_cpu(irq_stat, cpu).__softirq_pending which resulted in build errors on s390. This patch tries to create a generic accessor to this percpu softirq_pending data. This interface is inherently racy as its reading percpu data without a lock. However, being able to peek at the softirq pending data allows us to make better decisions about rt task placement vs just ignoring it. On s390 this call returns 0, which maybe isn't ideal but results in no functional change from what we do now. TODO: Heiko suggested changing s390 to use a proper per-cpu irqstat variable instead. Feedback or suggestions for better approach here would be welcome! Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Joel Fernandes Cc: Alexander Gordeev Cc: kernel-team@android.com Reported-by: kernel test robot Signed-off-by: John Stultz --- arch/s390/include/asm/hardirq.h | 6 ++++++ include/linux/interrupt.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/arch/s390/include/asm/hardirq.h b/arch/s390/include/asm/hardirq.h index 58668ffb5488..cd9cc11588ab 100644 --- a/arch/s390/include/asm/hardirq.h +++ b/arch/s390/include/asm/hardirq.h @@ -16,6 +16,12 @@ #define local_softirq_pending() (S390_lowcore.softirq_pending) #define set_softirq_pending(x) (S390_lowcore.softirq_pending = (x)) #define or_softirq_pending(x) (S390_lowcore.softirq_pending |= (x)) +/* + * Not sure what the right thing is here for s390, + * but returning 0 will result in no logical change + * from what happens now + */ +#define __cpu_softirq_pending(x) (0) #define __ARCH_IRQ_STAT #define __ARCH_IRQ_EXIT_IRQS_DISABLED diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a92bce40b04b..a749a8663841 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -527,6 +527,17 @@ DECLARE_STATIC_KEY_FALSE(force_irqthreads_key); #define set_softirq_pending(x) (__this_cpu_write(local_softirq_pending_ref, (x))) #define or_softirq_pending(x) (__this_cpu_or(local_softirq_pending_ref, (x))) +/** + * __cpu_softirq_pending() - Checks to see if softirq is pending on a cpu + * + * This helper is inherently racy, as we're accessing per-cpu data w/o locks. + * But peeking at the flag can still be useful when deciding where to place a + * task. + */ +static inline u32 __cpu_softirq_pending(int cpu) +{ + return (u32)per_cpu(local_softirq_pending_ref, cpu); +} #endif /* local_softirq_pending */ /* Some architectures might implement lazy enabling/disabling of From patchwork Wed Nov 16 07:59:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 20766 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp14454wru; Wed, 16 Nov 2022 00:03:22 -0800 (PST) X-Google-Smtp-Source: AA0mqf6OT8nv9106dYlGmYczPO4gTMXvE9r6uaAJhRn9GXuWsBKimLlY4ITMtC86DSvD1t+lnber X-Received: by 2002:a63:e48:0:b0:46f:f26e:e877 with SMTP id 8-20020a630e48000000b0046ff26ee877mr18968840pgo.545.1668585802539; Wed, 16 Nov 2022 00:03:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668585802; cv=none; d=google.com; s=arc-20160816; b=taFgKjrnWMAHjz8Rd1AmeehKxBBY8tYuIsHdHH9yI8cEBbCfBLke0zXibBi0LZCGkc ZiHcn9V4PxOT6r30W6tpJclx5i01bfu4+84DWIIyBFW9eBD8fzQfhMwHIKd6xE14Eo+J Qv2AaZmD5FvyE1/OL3k2n5zPA6uGkxEAcXrg/nrMzLgYF6K+XLehvB0n9+UT5ZdliT7b UxOIc7aQUgwdYKG5Z8z8IT4Yln9YTYQkneMoyRqM+rlACJUAhhlAaBHAB/32Ersig+f2 Ia4A7F/piSY56/ijqeP/No5IyHRvfnFadwTS6kqs7D4GIo+Jcu8mDF8SLjlhEGwT9MTt JKig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=1Yx3hruW+9J71DEKe9UfpU1h/qKMqxTS0D2Vr6oFr1g=; b=t4tyac4TherdJX3V4yogCLEe0jCGVeX2EQ/6Fp5ebG8vPWFoKfjV+WWNwT/fHSQ+rA Uxq7nSMo0PCuea/3EtJzYa5De9O4Q9Lre99UIUoJ6HnJwHWGlwlISr5kuBh0sMbVyelk 7tYn9RXyEAf3KEC3fhrj+wwTHhPiqFBY0xwpq1CXFOAWhh+s3JsBOb1YKrtrx9FEMLOF CyjbB4U7jt/Iw+zw484AcP8Kv3g2pgPO60Oesxs1cJn2nL0E8QWu4JKJulVSYt1vW7lZ TdLcy2hqUDBj41FvNDaAawqzyjez4Tf//OyF40lhSnpG69ashEMRfbreva64K0ksMw0t ZJ+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BeKPVtw9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i189-20020a6387c6000000b00476e640ddfesi1477346pge.80.2022.11.16.00.03.08; Wed, 16 Nov 2022 00:03:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BeKPVtw9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232954AbiKPH7t (ORCPT + 99 others); Wed, 16 Nov 2022 02:59:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232845AbiKPH7k (ORCPT ); Wed, 16 Nov 2022 02:59:40 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44FA6C7 for ; Tue, 15 Nov 2022 23:59:39 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id x11-20020a056a000bcb00b0056c6ec11eefso9352997pfu.14 for ; Tue, 15 Nov 2022 23:59:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1Yx3hruW+9J71DEKe9UfpU1h/qKMqxTS0D2Vr6oFr1g=; b=BeKPVtw938FgpE2fj5tbX0/xEYCMV7j4HPSZT9yK4HfbwlBapDncQLYGz0imOhWMRo nZ23AgSsBznN+yiJ4hnMJgwNvo8W3lmSmCVppq9zzTAdcjIs9VOHrTDisssRdWupkUiN GfG+lIvkJGiJ3kpfkc2xrnx9AebesJNqVqY6f0ovXKFavDz4SclJCEUeaF3sMtRm6DBh p/0ZhckxElVomontjl4KA9uZ9+y64hN8H9pdQP3FbgiwFsFKpl6rgYL7BSFpw3vBjd/v r4wJ9qGpBddA1NG9ocHfHdaM+6pQVlHzGnlwaxzxbQCIAwssvG2H2QV70xnmXU9luvVn G7jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1Yx3hruW+9J71DEKe9UfpU1h/qKMqxTS0D2Vr6oFr1g=; b=jkruDA4bmNf6P6Kt42Y88A6lACz+5zMJjq2I8B7wkPt7NfDvAt+p/Aoit1g0T46RO7 NoBG5igJpvg0pkj4cKZQLBrhvaNaYFhF++XpdaO+04c7Q2t0lE8ukWdFljs6hC3jVXsn Q4H/FWjEOX9RtDz4VpfgB5DYe/t+yh7BhqJ6ZsGGd3qtMW6MmAfEjy/jE6etB86FRk+Q HuIGE9ggO2QMzRJxl6hcGSGR5RW3AUyXx/lRubxSNzgVztrMCSzo0+SNtcqbEf/3mKR+ nOWukmZrLC9PDD2iGo/bRRDSHZDKqiUqlOSkEv+iSqPMA2U/z0r8btlud3av3tZjzumC yfmQ== X-Gm-Message-State: ANoB5pnika6DibfLvtZiKELRCDCTOw2t4vZ8YhN+aKEwd34xRzhX9/oT nKRq5xSbgP1qeA8YuwzoWaaoSIrOTvmVGyE8NeWTCCyagMpBtMuaIM1Rt/O5jmgfrAf6IMhAd9j Psm2rRL9fiky754gNnrNv2LXEqqnMN8nfr9tHDG1BscInlV3icl9vX/2RgOC5lFRMfL3e9SQ= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:902:b183:b0:186:d5b9:fbcd with SMTP id s3-20020a170902b18300b00186d5b9fbcdmr8083893plr.64.1668585578685; Tue, 15 Nov 2022 23:59:38 -0800 (PST) Date: Wed, 16 Nov 2022 07:59:27 +0000 In-Reply-To: <20221116075929.453876-1-jstultz@google.com> Mime-Version: 1.0 References: <20221116075929.453876-1-jstultz@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221116075929.453876-3-jstultz@google.com> Subject: [PATCH v5 2/3] sched: Avoid placing RT threads on cores handling long softirqs From: John Stultz To: LKML Cc: "Connor O'Brien" , John Dias , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , Joel Fernandes , Alexander Gordeev , kernel-team@android.com, "J . Avila" , John Stultz X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749639026194392364?= X-GMAIL-MSGID: =?utf-8?q?1749639026194392364?= From: Connor O'Brien In certain audio use cases, scheduling RT threads on cores that are handling softirqs can lead to glitches. Prevent this behavior in cases where the softirq is likely to take a long time. To avoid unnecessary migrations, the old behavior is preserved for RCU, SCHED and TIMER irqs which are expected to be relatively quick. This patch reworks and combines two related changes originally by John Dias Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Joel Fernandes Cc: Alexander Gordeev Cc: kernel-team@android.com Signed-off-by: John Dias [elavila: Port to mainline, amend commit text] Signed-off-by: J. Avila [connoro: Reworked, simplified, and merged two patches together] Signed-off-by: Connor O'Brien [jstultz: Further simplified and fixed issues, reworded commit message, removed arm64-isms] Signed-off-by: John Stultz --- v2: * Reformatted Kconfig entry to match coding style (Reported-by: Randy Dunlap ) * Made rt_task_fits_capacity_and_may_preempt static to avoid warnings (Reported-by: kernel test robot ) * Rework to use preempt_count and drop kconfig dependency on ARM64 v3: * Use introduced __cpu_softirq_pending() to avoid s390 build issues (Reported-by: kernel test robot ) v4: * Drop TASKLET_SOFTIRQ from LONG_SOFTIRQS (suggested by Qais) * Depend on !PREEMPT_RT (Suggested by Qais) * Larger simplification of logic (suggested by Qais) * Rework LONG_SOFTIRQS to use BIT() macros * Rename task_may_preempt() to cpu_busy_with_softirqs() v5: * Conditionalize active_softirqs handling (suggested by Alexander Gordeev ) * Reorder rt_task_fits_cpu to have the "fast" function first (Suggested by Alexander Gordeev ) * Fix bug I introduced in v2 condensing task_thread_info(task)->preempt_count to preempt_count() (Reported-by: Alexander Gordeev ) * Tweak comment discription to remove the vauge "slow" descriptor of softirqs being run by ksoftirqd (Suggested by Alexander Gordeev ) * Switch to using CONFIG_RT_SOFTIRQ_AWARE_SCHED (suggested by Joel Fernandes ) * Simplify cpu_busy_with_softirqs() logic as pointed out by Alexander Gordeev * Switch to using IS_ENABLED rather then defining my own macro (suggsted by Joel Fernandes ) --- include/linux/interrupt.h | 9 +++++++ init/Kconfig | 10 ++++++++ kernel/sched/rt.c | 49 ++++++++++++++++++++++++++++++++------- kernel/softirq.c | 17 ++++++++++++++ 4 files changed, 76 insertions(+), 9 deletions(-) diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index a749a8663841..7d09eb998d4c 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -582,6 +582,11 @@ enum * _ IRQ_POLL: irq_poll_cpu_dead() migrates the queue */ #define SOFTIRQ_HOTPLUG_SAFE_MASK (BIT(RCU_SOFTIRQ) | BIT(IRQ_POLL_SOFTIRQ)) +/* Softirq's where the handling might be long: */ +#define LONG_SOFTIRQ_MASK (BIT(NET_TX_SOFTIRQ) | \ + BIT(NET_RX_SOFTIRQ) | \ + BIT(BLOCK_SOFTIRQ) | \ + BIT(IRQ_POLL_SOFTIRQ)) /* map softirq index to softirq name. update 'softirq_to_name' in * kernel/softirq.c when adding a new softirq. @@ -618,6 +623,10 @@ extern void raise_softirq(unsigned int nr); DECLARE_PER_CPU(struct task_struct *, ksoftirqd); +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +DECLARE_PER_CPU(u32, active_softirqs); +#endif + static inline struct task_struct *this_cpu_ksoftirqd(void) { return this_cpu_read(ksoftirqd); diff --git a/init/Kconfig b/init/Kconfig index abf65098f1b6..ce0f0be5759c 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -1291,6 +1291,16 @@ config SCHED_AUTOGROUP desktop applications. Task group autogeneration is currently based upon task session. +config RT_SOFTIRQ_AWARE_SCHED + bool "Improve RT scheduling during long softirq execution" + depends on SMP && !PREEMPT_RT + default n + help + Enable an optimization which tries to avoid placing RT tasks on CPUs + occupied by nonpreemptible tasks, such as a long softirq or CPUs + which may soon block preemptions, such as a CPU running a ksoftirq + thread which handles slow softirqs. + config SYSFS_DEPRECATED bool "Enable deprecated sysfs features to support old userspace tools" depends on SYSFS diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index ed2a47e4ddae..152347c4394c 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1595,6 +1595,32 @@ static void yield_task_rt(struct rq *rq) #ifdef CONFIG_SMP static int find_lowest_rq(struct task_struct *task); +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +/* + * Return whether the given cpu is currently non-preemptible + * while handling a potentially long softirq, or if the current + * task is likely to block preemptions soon because it is a + * ksoftirq thread that is handling softirqs. + */ +static bool cpu_busy_with_softirqs(int cpu) +{ + u32 softirqs = per_cpu(active_softirqs, cpu) | + __cpu_softirq_pending(cpu); + + return softirqs & LONG_SOFTIRQ_MASK; +} +#else +static bool cpu_busy_with_softirqs(int cpu) +{ + return false; +} +#endif /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ + +static bool rt_task_fits_cpu(struct task_struct *p, int cpu) +{ + return rt_task_fits_capacity(p, cpu) && !cpu_busy_with_softirqs(cpu); +} + static int select_task_rq_rt(struct task_struct *p, int cpu, int flags) { @@ -1633,22 +1659,24 @@ select_task_rq_rt(struct task_struct *p, int cpu, int flags) * This test is optimistic, if we get it wrong the load-balancer * will have to sort it out. * - * We take into account the capacity of the CPU to ensure it fits the - * requirement of the task - which is only important on heterogeneous - * systems like big.LITTLE. + * We use rt_task_fits_cpu() to evaluate if the CPU is busy with + * potentially long-running softirq work, as well as take into + * account the capacity of the CPU to ensure it fits the + * requirement of the task - which is only important on + * heterogeneous systems like big.LITTLE. */ test = curr && unlikely(rt_task(curr)) && (curr->nr_cpus_allowed < 2 || curr->prio <= p->prio); - if (test || !rt_task_fits_capacity(p, cpu)) { + if (test || !rt_task_fits_cpu(p, cpu)) { int target = find_lowest_rq(p); /* * Bail out if we were forcing a migration to find a better * fitting CPU but our search failed. */ - if (!test && target != -1 && !rt_task_fits_capacity(p, target)) + if (!test && target != -1 && !rt_task_fits_cpu(p, target)) goto out_unlock; /* @@ -1890,14 +1918,17 @@ static int find_lowest_rq(struct task_struct *task) return -1; /* No other targets possible */ /* - * If we're on asym system ensure we consider the different capacities - * of the CPUs when searching for the lowest_mask. + * If we're using the softirq optimization or if we are + * on asym system, ensure we consider the softirq processing + * or different capacities of the CPUs when searching for the + * lowest_mask. */ - if (sched_asym_cpucap_active()) { + if (IS_ENABLED(CONFIG_RT_SOFTIRQ_AWARE_SCHED) || + sched_asym_cpucap_active()) { ret = cpupri_find_fitness(&task_rq(task)->rd->cpupri, task, lowest_mask, - rt_task_fits_capacity); + rt_task_fits_cpu); } else { ret = cpupri_find(&task_rq(task)->rd->cpupri, diff --git a/kernel/softirq.c b/kernel/softirq.c index c8a6913c067d..dd92ce8f771b 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -60,6 +60,21 @@ static struct softirq_action softirq_vec[NR_SOFTIRQS] __cacheline_aligned_in_smp DEFINE_PER_CPU(struct task_struct *, ksoftirqd); +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +/* + * active_softirqs -- per cpu, a mask of softirqs that are being handled, + * with the expectation that approximate answers are acceptable and therefore + * no synchronization. + */ +DEFINE_PER_CPU(u32, active_softirqs); +static inline void set_active_softirqs(u32 pending) +{ + __this_cpu_write(active_softirqs, pending); +} +#else /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ +static inline void set_active_softirqs(u32 pending) {}; +#endif /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ + const char * const softirq_to_name[NR_SOFTIRQS] = { "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "IRQ_POLL", "TASKLET", "SCHED", "HRTIMER", "RCU" @@ -551,6 +566,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) restart: /* Reset the pending bitmask before enabling irqs */ set_softirq_pending(0); + set_active_softirqs(pending); local_irq_enable(); @@ -580,6 +596,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) pending >>= softirq_bit; } + set_active_softirqs(0); if (!IS_ENABLED(CONFIG_PREEMPT_RT) && __this_cpu_read(ksoftirqd) == current) rcu_softirq_qs(); From patchwork Wed Nov 16 07:59:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 20767 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp14669wru; Wed, 16 Nov 2022 00:03:52 -0800 (PST) X-Google-Smtp-Source: AA0mqf7PvaBpsUCg/pR37lt95nH3ZLhEAry4wdiQ6enRpJt1lE1dQ4Wlrl9Q/GWIIFe8j32kAejo X-Received: by 2002:a17:90a:9c8:b0:212:e521:7cbd with SMTP id 66-20020a17090a09c800b00212e5217cbdmr2453108pjo.230.1668585831847; Wed, 16 Nov 2022 00:03:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1668585831; cv=none; d=google.com; s=arc-20160816; b=C0XfrZwXcRMz8vBminImGbO5T8yW1uql/JgLhDPCAuIfjP3h7CbZoci1Qa9X8UEShx kZO9zPiGaDkOhxwIfukqeu7Dg/Ij4JtuTL6Kov14iQ5d4u6eQurwHVGQnfSn4ccR0UuH AwEAK1gZWLuedWz5seh/nXqotfOreHbVCtHD4QkrtNoXqMztnCqx3+8jyZZWfpQB34tJ ZJHdYu9rcvrpLkIbnCUvoRMdPxFti16D0rnSb9XhBVdbimO29i6t7m5e7iC7IhTCQUB0 muUsmhX9V9y08UkGBEqKdS0n6zAfwPhpxIWomLmRSD8QtXD+hR9FO0jcNrOa5JXn+y6W 08Hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=LdOYk9RhgVfGcLslo0PFudrqwmqg3H4kC0w388sZncY=; b=dVFczWvJbvRIVWA61p7HnrQD8VWb9RUL5bJs7Fc9rgqemDExq8NEKn7mlLRZoaTJpP JshNhugGzuEJjDVuadZtol7S5Cny+L9yTgBPgep497khFSSQDN90HtfCHFROh1QP46uv JO/aWsoqyTXJzWZjCs3n5pmPX6Bp+o4CI1kpPXR36SvoYx2pt7SIgrdoGlFmq6YSl3YA rwU/hJlpqBnU96O12gR0v69N/fE8YUOs4zQA8EM1k5oDj1BX/vtYNOqwnBw/tou7OJXB KMZ9Ud9XffnYkcC4ZjiPsw1JH6s1UIhrT5ZS3KCnzLhkEnjZWbBADJf0CA2T1VUcYfQY tgVQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="fr/JCq7r"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b6-20020a170902d88600b00188c26e5bcdsi7596464plz.531.2022.11.16.00.03.38; Wed, 16 Nov 2022 00:03:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="fr/JCq7r"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232829AbiKPIAA (ORCPT + 99 others); Wed, 16 Nov 2022 03:00:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232908AbiKPH7m (ORCPT ); Wed, 16 Nov 2022 02:59:42 -0500 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10FC5C7 for ; Tue, 15 Nov 2022 23:59:42 -0800 (PST) Received: by mail-pg1-x549.google.com with SMTP id e128-20020a636986000000b0047075a1c725so8926470pgc.19 for ; Tue, 15 Nov 2022 23:59:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LdOYk9RhgVfGcLslo0PFudrqwmqg3H4kC0w388sZncY=; b=fr/JCq7r62KE0Vzw76H1bT9MaRv3VI3U4VSrGRUA3D9YlXCe2DUz51VM2lWrXOMUxV YVvBZTX2gKnpizrEuUVasgONXKXLzliBHpflI08C9h2pwa2NzO4q064Jgg6ZKBz6igZS I9JwaIyxHq8NVtKRz0QSHSCkkbKYzCXQIsScAG1CA3z51Y67t22Hx7StMioevPluwQDa 882XH8O/VafbYarNIkrtfLoqikiFo3UDW0rnkG0mNz8KsvhslrVhuOyDLWy5QEH8/pzd HHHIfpyQnTGTWeZAjbB1M+aC4xSsTawDgHC1jpHvqvNdu5z+IuWLtIrY3uupyafBSqv0 ifUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LdOYk9RhgVfGcLslo0PFudrqwmqg3H4kC0w388sZncY=; b=dPhPrn8VdVuiKAWl13KtVGuUsWLWjTqgls5xBROGoXBc3RPmZqTDg8ljvpKbaGSqs/ sS51DcUQ3AoiLpsdef3bNdACo3ILbOtQQzLKeLLBVl1ZB1YxUz2br/iY2LImD/hWm15o CP+n9p9Aq1d+3FM0rB1Uxi+SLkyTuKDzC0HG87VCJky1k/HfRV0yWqNX62rpE7vwWAWC rExbZQrDSgWWdBKKaY6FI7L8TXIH7SIYZQKN9iokyx4COp0F42uSVFBUBRXkQijr/QNf IDMbGEKEkWIc+tB3mRQKyl3yurcW6+jSJ90FNLj6YjugvKYPStBJ1KndqvLbMoIpaU2K TCBg== X-Gm-Message-State: ANoB5pmg+IhiwPYYzdjJ4/L7yspZ/g00/XX1XgQIk4RTvZc38jncyC+Z SkCHM7wVArN3RwgknsTpDPBye2hvILPMDONKqu+pe+c3NKxdw+67HKMoXkFyYUuFGHaat6Wjkzg rB5SqVFILSXXFfqPGYkquhwMSv7rX3fi0H9TqwHIdhjzgYZ1HiuzLNwnWhzWMrvSL+73D1lU= X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a17:90a:a392:b0:20a:fee1:8f69 with SMTP id x18-20020a17090aa39200b0020afee18f69mr225380pjp.0.1668585580398; Tue, 15 Nov 2022 23:59:40 -0800 (PST) Date: Wed, 16 Nov 2022 07:59:28 +0000 In-Reply-To: <20221116075929.453876-1-jstultz@google.com> Mime-Version: 1.0 References: <20221116075929.453876-1-jstultz@google.com> X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221116075929.453876-4-jstultz@google.com> Subject: [PATCH v5 3/3] softirq: defer softirq processing to ksoftirqd if CPU is busy with RT From: John Stultz To: LKML Cc: Pavankumar Kondeti , John Dias , "Connor O'Brien" , Rick Yiu , John Kacur , Qais Yousef , Chris Redpath , Abhijeet Dharmapurikar , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Thomas Gleixner , Heiko Carstens , Vasily Gorbik , Joel Fernandes , Alexander Gordeev , kernel-team@android.com, Satya Durga Srinivasu Prabhala , "J . Avila" , John Stultz X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1749639057082724914?= X-GMAIL-MSGID: =?utf-8?q?1749639057082724914?= From: Pavankumar Kondeti Defer the softirq processing to ksoftirqd if a RT task is running or queued on the current CPU. This complements the RT task placement algorithm which tries to find a CPU that is not currently busy with softirqs. Currently NET_TX, NET_RX, BLOCK and IRQ_POLL softirqs are only deferred as they can potentially run for long time. Additionally, this patch stubs out ksoftirqd_running() logic, in the CONFIG_RT_SOFTIRQ_AWARE_SCHED case, as deferring potentially long-running softirqs will cause the logic to not process shorter-running softirqs immediately. By stubbing it out the potentially long running softirqs are deferred, but the shorter running ones can still run immediately. This patch includes folded-in fixes by: Lingutla Chandrasekhar Satya Durga Srinivasu Prabhala J. Avila Cc: John Dias Cc: Connor O'Brien Cc: Rick Yiu Cc: John Kacur Cc: Qais Yousef Cc: Chris Redpath Cc: Abhijeet Dharmapurikar Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Thomas Gleixner Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Joel Fernandes Cc: Alexander Gordeev Cc: kernel-team@android.com Signed-off-by: Pavankumar Kondeti [satyap@codeaurora.org: trivial merge conflict resolution.] Signed-off-by: Satya Durga Srinivasu Prabhala [elavila: Port to mainline, squash with bugfix] Signed-off-by: J. Avila [jstultz: Rebase to linus/HEAD, minor rearranging of code, included bug fix Reported-by: Qais Yousef ] Signed-off-by: John Stultz --- v4: * Fix commit message to accurately note long-running softirqs (suggested by Qais) * Switch to using rt_task(current) (suggested by Qais) v5: * Switch to using CONFIG_RT_SOFTIRQ_AWARE_SCHED (suggested by Joel Fernandes ) --- kernel/softirq.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/kernel/softirq.c b/kernel/softirq.c index dd92ce8f771b..5db2afd0be68 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -95,6 +95,7 @@ static void wakeup_softirqd(void) wake_up_process(tsk); } +#ifndef CONFIG_RT_SOFTIRQ_AWARE_SCHED /* * If ksoftirqd is scheduled, we do not want to process pending softirqs * right now. Let ksoftirqd handle this at its own rate, to get fairness, @@ -109,6 +110,9 @@ static bool ksoftirqd_running(unsigned long pending) return false; return tsk && task_is_running(tsk) && !__kthread_should_park(tsk); } +#else +#define ksoftirqd_running(pending) (false) +#endif /* CONFIG_RT_SOFTIRQ_AWARE_SCHED */ #ifdef CONFIG_TRACE_IRQFLAGS DEFINE_PER_CPU(int, hardirqs_enabled); @@ -540,6 +544,21 @@ static inline bool lockdep_softirq_start(void) { return false; } static inline void lockdep_softirq_end(bool in_hardirq) { } #endif +#ifdef CONFIG_RT_SOFTIRQ_AWARE_SCHED +static __u32 softirq_deferred_for_rt(__u32 *pending) +{ + __u32 deferred = 0; + + if (rt_task(current)) { + deferred = *pending & LONG_SOFTIRQ_MASK; + *pending &= ~LONG_SOFTIRQ_MASK; + } + return deferred; +} +#else +#define softirq_deferred_for_rt(x) (0) +#endif + asmlinkage __visible void __softirq_entry __do_softirq(void) { unsigned long end = jiffies + MAX_SOFTIRQ_TIME; @@ -547,6 +566,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) int max_restart = MAX_SOFTIRQ_RESTART; struct softirq_action *h; bool in_hardirq; + __u32 deferred; __u32 pending; int softirq_bit; @@ -558,14 +578,16 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) current->flags &= ~PF_MEMALLOC; pending = local_softirq_pending(); + deferred = softirq_deferred_for_rt(&pending); softirq_handle_begin(); + in_hardirq = lockdep_softirq_start(); account_softirq_enter(current); restart: /* Reset the pending bitmask before enabling irqs */ - set_softirq_pending(0); + set_softirq_pending(deferred); set_active_softirqs(pending); local_irq_enable(); @@ -604,13 +626,16 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) local_irq_disable(); pending = local_softirq_pending(); + deferred = softirq_deferred_for_rt(&pending); + if (pending) { if (time_before(jiffies, end) && !need_resched() && --max_restart) goto restart; + } + if (pending | deferred) wakeup_softirqd(); - } account_softirq_exit(current); lockdep_softirq_end(in_hardirq);