From patchwork Tue Jul 18 00:59:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Finn Thain X-Patchwork-Id: 121722 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:c923:0:b0:3e4:2afc:c1 with SMTP id j3csp1459459vqt; Mon, 17 Jul 2023 19:12:35 -0700 (PDT) X-Google-Smtp-Source: APBJJlGuiFvuXedYdOKFE7UZGo45Vr7UQDEld/kZyfh8zDNKnVnLHrE3Ye2U1JcFpxqwcy0sKpnL X-Received: by 2002:a2e:9695:0:b0:2b7:1c0f:f20f with SMTP id q21-20020a2e9695000000b002b71c0ff20fmr8645164lji.7.1689646354764; Mon, 17 Jul 2023 19:12:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689646354; cv=none; d=google.com; s=arc-20160816; b=To8x1L0xgbMHVgFYwmbT/VLL1GNN96dIs0gD+n3kztc7gWwslX1ynGnVDNCPcojlee Jy5KJ5tqRoGPRvOS4JiIuiD/MSduR61Zgthrx/+HyjS/VO4vYRXb38JslVht3LfU3APW JIex2muSE9r4p9f5xDF2IFq+648M7d81bQuVN8PWqsSRcOMXETPUbSkP6lL/mGkuA6A2 4isshDIr+XMBNlDoP+w+hK1JcJFVWsxbqRyh4XQqw9g8M24WhOovCSC7evNo4s7KOk2z UKbJHMAEAcGYOIxRs8f0FnuAMgL9TbdByw/EY7YZ5dWVUUtKXkg2hHuS6bf8WZletY+l S83A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:date:subject:from:message-id:cc:to:feedback-id :dkim-signature; bh=kMkiwW1YZ5sB+UDs0gfnkYhphzsmIRLkHogP74cDU/o=; fh=WmxStnWLlGQNg81gf+AP0K/yG+Kde8rgGBJ6ak0nqEM=; b=chdLpDgcje+vMXFyB+3RmfbbA6iM6hzfYJmkMhhY1NR9vVH6fAOSr/EEBvn3nGO9g1 MJaZuPi+kUKfjakB3nBH6QJdYe9q3GztzeCMN+bL8Aii8fIDhglaUPMu4tn5SsG8Iz0p 4QMkVvhL6qSZfpZyT7+3fDOggEdY8+cUvJvzTrxvlCkKBsE5CwYF5SLUJ5nmnObHG/Xj t4vRLGaHWOErYjJFWpInci3upCYopUStDaFhsbP8wK5pAXMrxSkNXF3DZwBOpeW4SR3q sFf19jMRR5Oz9cE9BEVkJGEy0e2s5E4GWFc2bi8RFVai6rGt7uGLCnJRFOWYdor9whKQ bwoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm3 header.b=Rg+K0GO9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id li13-20020a170906f98d00b0099325564b77si369533ejb.548.2023.07.17.19.12.10; Mon, 17 Jul 2023 19:12:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm3 header.b=Rg+K0GO9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230318AbjGRBDf (ORCPT + 99 others); Mon, 17 Jul 2023 21:03:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230103AbjGRBDe (ORCPT ); Mon, 17 Jul 2023 21:03:34 -0400 Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E405695 for ; Mon, 17 Jul 2023 18:03:32 -0700 (PDT) Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id A64F15C019B; Mon, 17 Jul 2023 21:03:28 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 17 Jul 2023 21:03:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:reply-to:sender :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; t=1689642208; x=1689728608; bh=k MkiwW1YZ5sB+UDs0gfnkYhphzsmIRLkHogP74cDU/o=; b=Rg+K0GO9qaEKmeh7v aK1WZg3Um0FfuBs7apkhzLcGjA588KOI/5/1kjCvqjpclz2T0DWFSsz+xjSOXCHd yXT/tPg0ctjyFpF+GAc0dqDYaBdPHStMFYE4Bel1H2r6tsEP3RCWnOuPlQer9ZsP bkKbBs9SZ/fPUsXp8V7nsEz3OGn2Hiq5DBalCVr5TxKXMMgMQRHPT4TIZnoWAqMy uLLhZbM4ms7+dtnwzwZ+NAj/k0/leabtvTVMU3lKFNspw0ybCEHDanuW0xBhzq9i A5RyuSBH4NiRdGb3kdO74AANr4FWmYVxUp+yNza0/Cq6tQEV8ip4sMlW7Dt9ubia +mzCg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrgeefgdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefvvefkhffuffestddtredttddttdenucfhrhhomhephfhinhhnucfvhhgrihhn uceofhhthhgrihhnsehlihhnuhigqdhmieekkhdrohhrgheqnecuggftrfgrthhtvghrnh epheffgfegfeevgeevtdeiffefveeutdeghfeuheeiteffjeefgfegveefuedvudelnecu vehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepfhhthhgrih hnsehlihhnuhigqdhmieekkhdrohhrgh X-ME-Proxy: Feedback-ID: i58a146ae:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 17 Jul 2023 21:03:25 -0400 (EDT) To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Frederic Weisbecker Cc: "Thomas Gleixner" , linux-kernel@vger.kernel.org Message-Id: <453f675efb082e08068736bf69293d48ff3129a7.1689641959.git.fthain@linux-m68k.org> From: Finn Thain Subject: [PATCH v2] sched: Optimize in_task() and in_interrupt() a bit Date: Tue, 18 Jul 2023 10:59:19 +1000 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL, SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771722615997761870 X-GMAIL-MSGID: 1771722615997761870 Except on x86, preempt_count is always accessed with READ_ONCE. Repeated invocations in macros like irq_count() produce repeated loads. These redundant instructions appear in various fast paths. In the one shown below, for example, irq_count() is evaluated during kernel entry if !tick_nohz_full_cpu(smp_processor_id()). 0001ed0a : 1ed0a: 4e56 0000 linkw %fp,#0 1ed0e: 200f movel %sp,%d0 1ed10: 0280 ffff e000 andil #-8192,%d0 1ed16: 2040 moveal %d0,%a0 1ed18: 2028 0008 movel %a0@(8),%d0 1ed1c: 0680 0001 0000 addil #65536,%d0 1ed22: 2140 0008 movel %d0,%a0@(8) 1ed26: 082a 0001 000f btst #1,%a2@(15) 1ed2c: 670c beqs 1ed3a 1ed2e: 2028 0008 movel %a0@(8),%d0 1ed32: 2028 0008 movel %a0@(8),%d0 1ed36: 2028 0008 movel %a0@(8),%d0 1ed3a: 4e5e unlk %fp 1ed3c: 4e75 rts This patch doesn't prevent the pointless btst and beqs instructions above, but it does eliminate 2 of the 3 pointless move instructions here and elsewhere. On x86, preempt_count is per-cpu data and the problem does not arise presumably because the compiler is free to optimize more effectively. Cc: Thomas Gleixner Fixes: 15115830c887 ("preempt: Cleanup the macro maze a bit") Signed-off-by: Finn Thain --- This patch was tested on m68k and x86. I was expecting no changes to object code for x86 and mostly that's what I saw. However, there were a few places where code generation was perturbed for some reason. Changed since v1: - Added a comment that was requested by Frederic. --- include/linux/preempt.h | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/include/linux/preempt.h b/include/linux/preempt.h index 1424670df161..d2ac07abf225 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -99,14 +99,21 @@ static __always_inline unsigned char interrupt_context_level(void) return level; } +/* + * Redundant invocations of preempt_count() are to be avoided because that + * function is commonly implemented using READ_ONCE. Hence, the optimal + * definition of irq_count() or in_task() depends on CONFIG_PREEMPT_RT. + */ + #define nmi_count() (preempt_count() & NMI_MASK) #define hardirq_count() (preempt_count() & HARDIRQ_MASK) #ifdef CONFIG_PREEMPT_RT # define softirq_count() (current->softirq_disable_cnt & SOFTIRQ_MASK) +# define irq_count() ((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | softirq_count()) #else # define softirq_count() (preempt_count() & SOFTIRQ_MASK) +# define irq_count() (preempt_count() & (NMI_MASK | HARDIRQ_MASK | SOFTIRQ_MASK)) #endif -#define irq_count() (nmi_count() | hardirq_count() | softirq_count()) /* * Macros to retrieve the current execution context: @@ -119,7 +126,11 @@ static __always_inline unsigned char interrupt_context_level(void) #define in_nmi() (nmi_count()) #define in_hardirq() (hardirq_count()) #define in_serving_softirq() (softirq_count() & SOFTIRQ_OFFSET) -#define in_task() (!(in_nmi() | in_hardirq() | in_serving_softirq())) +#ifdef CONFIG_PREEMPT_RT +# define in_task() (!((preempt_count() & (NMI_MASK | HARDIRQ_MASK)) | in_serving_softirq())) +#else +# define in_task() (!(preempt_count() & (NMI_MASK | HARDIRQ_MASK | SOFTIRQ_OFFSET))) +#endif /* * The following macros are deprecated and should not be used in new code: