From patchwork Sun Nov 6 09:51:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Mailhol X-Patchwork-Id: 16106 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1416750wru; Sun, 6 Nov 2022 01:54:22 -0800 (PST) X-Google-Smtp-Source: AMsMyM7kjyvXuBgxg+lKYyfTLMHz+ssRtUdd/5j9yzItZ+z7KY29a0pOg7Yjtci8OXf0DGRHyHzr X-Received: by 2002:a65:480c:0:b0:458:88cd:f48 with SMTP id h12-20020a65480c000000b0045888cd0f48mr36839918pgs.529.1667728462123; Sun, 06 Nov 2022 01:54:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1667728462; cv=none; d=google.com; s=arc-20160816; b=bnEYkjHh1r2xwFSKTa1oMmM4kNxP/biRaQE3VNZRrSeOdcGDblHpmc2nyoGzlRJDbF MdaRxvXrSJ8iljHpy14i91VTjkFCoN9awe4yH0L61f0ClpsR0xKhRIVL/GdaXVD4lH5w +mlQt+GQcMSDNdOeyPfBV91LrwathZXJqd+DHoBDKq6gf4b9exIvSUxSzWNymWzoelRh avnVj+IjSGZsEwaPOdyJmMukeGo3aoEbnYpWGRHy6WWjNE81sVAN62an3qXBA9jSVjw5 9IZzVCEiP/EipTRpS6tmd2VWXpYQIO7+TaBoq8+Y9rneEo9yTHY70OCcckRUhAnQ/kO7 13Kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=tnyJavHPkbHwX7+XS5T05mPkmW2Gv1owKlEwtY5yB8s=; b=ha0uOaxiT/d59OnSRS9e/QGxyinxbRa8VkMC/8yMbJjweFaqRhDP7HtSLgHtL/bSk8 /4EFdffQ3CTPQlbk7kj/g9tbnvwcJmaRg7qeI+Ut+soiiot+CZi4d+lW+VpDewS72tAd tIv9el/djnYYSMXa3d5ReVVcQ2rZdqEczzwYIPFZiVmHdPVW9aG9wyiRILiWyDmdkD2I S4CsmV7PHxMOaqqzmoh6O6Zri1WHy4kWNzWAJxq93CNRcnycKabl900OO/ftsxxxm/5O OUnYV6PUvhM3qxrb6Tout7g7FtGy1Y6cVlSqo5fZB23nDEojQzehxNbdHlcndFn/MwLj Xv+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=EGDlpI4m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id il18-20020a17090b165200b002136ad22b32si8156609pjb.39.2022.11.06.01.54.08; Sun, 06 Nov 2022 01:54:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=EGDlpI4m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229821AbiKFJv3 (ORCPT + 99 others); Sun, 6 Nov 2022 04:51:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229783AbiKFJv1 (ORCPT ); Sun, 6 Nov 2022 04:51:27 -0500 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD6F0E017 for ; Sun, 6 Nov 2022 01:51:26 -0800 (PST) Received: by mail-pg1-x536.google.com with SMTP id e129so8003643pgc.9 for ; Sun, 06 Nov 2022 01:51:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=tnyJavHPkbHwX7+XS5T05mPkmW2Gv1owKlEwtY5yB8s=; b=EGDlpI4moUq2A6m7KudPbv/MhIVZ8nGbetGSacPWCapDt+bJN5bWh0NvT2KzL05wAW dHj2IK70k94AuZwQl1YZqgHBWP8Xak6m4PV128avPnBPcH6CzVgxADpz2Yi+u3N7jsbU LoDwwNUnurtjlEap0kie2cdbx+9s3cFGUQmt+MQBVRlDjicQ7p6Ix102pirz9Oeu5/Aj EKaPKKmsO7ywKz6Fn2Fr9V16xXqSXYlCM0gTrSB+K3VFEOKAOMzUAaMidUsjO8kgylIu icFbRlz9pQFjRgoIYZJzPKg93ukIgWp9rcrUqImfhOcRMouj2duI9ViSNqsUFFRlUQOn ossA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=tnyJavHPkbHwX7+XS5T05mPkmW2Gv1owKlEwtY5yB8s=; b=5GJmTM4YO/BWRI4ZKoz0vkXDwG/vVxqeuJcrWqe9wC/mZoJJt2NUnfCtJn/Dw48Ei0 XsoEAa4pXh/4Xn83wBId8kMGmcf8h668qIJALreS4OTSkLWfVW32LnBGaK0rkE4Bw5H4 GRGKcJTHZSsTKhZEPW3hwGIRtqxDJUsm7mO+d0s5l3h60bGbeHMwbQMbQfvmfYdfGUFW GxApMmI2NvyvxyeEKRSb7eMApaQTaWqNUVsh4TR5pRiHcWJvvlAWy7be6F2+cFXKtNwv gFYaaS5AABUqBoI6xELAgUf5UO8IHBYTDvrknDcWUU6+RvRgTPeLZA00y2R+VE6RWcyC rcBQ== X-Gm-Message-State: ACrzQf2+p9qyfyuGZFyFl/pWaAQo9GC5l/WWpwhdJ0zihQJHGOV6fm50 Lh61DtNXjrBAWQwEGA/ks54= X-Received: by 2002:a05:6a00:e1b:b0:537:7c74:c405 with SMTP id bq27-20020a056a000e1b00b005377c74c405mr44307387pfb.43.1667728286198; Sun, 06 Nov 2022 01:51:26 -0800 (PST) Received: from localhost.localdomain (124x33x176x97.ap124.ftth.ucom.ne.jp. [124.33.176.97]) by smtp.gmail.com with ESMTPSA id b5-20020aa79505000000b0056bfd4a2702sm2368677pfp.45.2022.11.06.01.51.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Nov 2022 01:51:25 -0800 (PST) Sender: Vincent Mailhol From: Vincent Mailhol To: x86@kernel.org, Ingo Molnar , Borislav Petkov Cc: Nick Desaulniers , Thomas Gleixner , linux-kernel@vger.kernel.org, Yury Norov , llvm@lists.linux.dev, Vincent Mailhol , Borislav Petkov Subject: [PATCH v1 1/2] x86/asm/bitops: Replace __fls() by its generic builtin implementation Date: Sun, 6 Nov 2022 18:51:05 +0900 Message-Id: <20221106095106.849154-2-mailhol.vincent@wanadoo.fr> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> References: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748740039693708765?= X-GMAIL-MSGID: =?utf-8?q?1748740039693708765?= Below snippet: #include unsigned int foo(unsigned long word) { return __fls(word); } produces this on GCC 12.1.0: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 48 89 fb mov %rdi,%rbx d: e8 00 00 00 00 call 12 12: 48 0f bd c3 bsr %rbx,%rax 16: 5b pop %rbx 17: 31 ff xor %edi,%edi 19: e9 00 00 00 00 jmp 1e and that on clang 14.0.6: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 50 push %rax b: 48 89 fb mov %rdi,%rbx e: e8 00 00 00 00 call 13 13: 48 89 1c 24 mov %rbx,(%rsp) 17: 48 0f bd 04 24 bsr (%rsp),%rax 1c: 48 83 c4 08 add $0x8,%rsp 20: 5b pop %rbx 21: c3 ret The implementation from [1] produces the exact same code on GCC and below code on clang: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 48 89 fb mov %rdi,%rbx d: e8 00 00 00 00 call 12 12: 48 0f bd c3 bsr %rbx,%rax 16: 5b pop %rbx 17: c3 ret The builtin implementation is better for two reasons: 1/ it saves two instructions on clang (a push and a stack pointer decrement) because of a useless tentative to save rax. 2/ when used on constant expressions, the compiler is only able to fold the builtin version (c.f. [2]). For those two reasons, replace the assembly implementation by its builtin counterpart. [1] https://elixir.bootlin.com/linux/v6.0/source/include/asm-generic/bitops/builtin-__fls.h [2] commit 146034fed6ee ("x86/asm/bitops: Use __builtin_ffs() to evaluate constant expressions") CC: Borislav Petkov CC: Nick Desaulniers CC: Yury Norov Signed-off-by: Vincent Mailhol Reviewed-by: Nick Desaulniers --- arch/x86/include/asm/bitops.h | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index 2edf68475fec..a31453d7686d 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -285,19 +285,7 @@ static __always_inline unsigned long variable_ffz(unsigned long word) (unsigned long)__builtin_ctzl(~word) : \ variable_ffz(word)) -/* - * __fls: find last set bit in word - * @word: The word to search - * - * Undefined if no set bit exists, so code should check against 0 first. - */ -static __always_inline unsigned long __fls(unsigned long word) -{ - asm("bsr %1,%0" - : "=r" (word) - : "rm" (word)); - return word; -} +#include #undef ADDR From patchwork Sun Nov 6 09:51:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Mailhol X-Patchwork-Id: 16107 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp1417050wru; Sun, 6 Nov 2022 01:55:32 -0800 (PST) X-Google-Smtp-Source: AMsMyM4h22HpC9jv0W8uHCLvx2wdT3Ri4p73VCcI13FcEHcDjLMqX14G9+tKSn/yjCZFmuWVaL9p X-Received: by 2002:a17:903:1c2:b0:187:feb:1f31 with SMTP id e2-20020a17090301c200b001870feb1f31mr41495192plh.92.1667728531935; Sun, 06 Nov 2022 01:55:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1667728531; cv=none; d=google.com; s=arc-20160816; b=kTg99YZvd/BZVdYErxM7tYQM5M0ThxvTD+kYydqTkfxXM+Jk23NVm19zGvPi42qiwB HKiV6c6rEpyII+Ln3qTkJwRPhCU492YT+3FIw0e4sBTVkQfIvGhbSGyXV4SFM4SuNFDi udkhFSkEkI3Zxx1q4GwJBid7pcrera1xPL7fY6slyOIsafvOBFarT8f8Se+H7utAh1Fh Smd7Ujr4jlXbynWk4fMAsE/GeQc51FZQNTTVtYIaByJKjvHsiP+4cGFDH1VLMQA+g3+y 10h+jcYS4hjUHzGOlo1Cb0UupIOtG/N57UwPjVYNTbt4lumdtin90RmA9XogXZSpV7M/ exQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=ypcKLldwiqaDsOlKIEGYcVxJUId8HFUjb6S7/qxsdjE=; b=bYeSm4JAYX/h4ZDGhn/5ZBzjaxba0iYHcVUCAHrqRCFOBiiUSDVNqniDp+RcOkSUJC Gd2L9pICcqIaB3VOL7eG8k46J0Tl7rGrjP/HclUjQwE8gyHgEGiuLkz5qCWq93Pz4esa W2o3HXc39nb8dCOrh0CGIGKPsc9eVFFKQS8RwQIpO6zBhwN7oilpgt7U8fFyKqq9Sd1n wjiemQC5UBf7V5z+0WoZ6z0aFsyXPQVuqecyPWuLPEQtQYehwHPKtFCwuNpqfddiQEXC 0FzMxu1ZtPxTzySmYjf7sWBAhcE3aO0secQ0Yb/PwlktE0uewmqDgLb25i3IVucin3p6 85iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=QBgr9zaT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b5-20020a17090a550500b0020388f845e9si9922369pji.52.2022.11.06.01.55.19; Sun, 06 Nov 2022 01:55:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=QBgr9zaT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229850AbiKFJvf (ORCPT + 99 others); Sun, 6 Nov 2022 04:51:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229829AbiKFJva (ORCPT ); Sun, 6 Nov 2022 04:51:30 -0500 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 538E9E0B2 for ; Sun, 6 Nov 2022 01:51:29 -0800 (PST) Received: by mail-pf1-x42f.google.com with SMTP id q9so8152824pfg.5 for ; Sun, 06 Nov 2022 01:51:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=ypcKLldwiqaDsOlKIEGYcVxJUId8HFUjb6S7/qxsdjE=; b=QBgr9zaTfSggETOEfREZkmAC9yEJR6EOPOcX9uReV/JQPsZ/IU1xIcz14lf13MYlNJ PfwH/wZiLZ5HW/VSUBBAnIoxDaFNoBfpgHqP0OVPOwkagdtbatdoSACMxEfkwNSKPJJa +xD8mX/j+P4RRrKXr/QJ890acISwlsaHC5764DhakcUjQq7x97TjCgS78/quvz1p6KQo HgQj+z0vW2mini8Xj7P4O+0NAk9EQpcan1J+VKPTPWs/rNorAxtHQClBp01tB0ztr1+w ts2D7OIpGVyaIsjN11sEzKqqhHOl41ldAWwnqa2GlIlGUjMuTJJLiXpNwcW5VwhZhura Bg3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ypcKLldwiqaDsOlKIEGYcVxJUId8HFUjb6S7/qxsdjE=; b=1AcSIxkuym5G0ZfPnd6viV/bM5ZaKaCRh4Jufx8s/U6wXDY7OyusCXklLV8S/mRtAo XKaKaTeCp4lt0JJnARdRyp970aDZwMJ09P6Bpwz9sKq+4rYVNInFsUxp0l+oIWPR9OKg qsaimri9sAgTZpsTRZf/mYiWFuXLY5sQ0u/lq7apzE2QzeMoCOiuwm0M+sHi934pdWy4 JlhCaF/UtWODMhDfDhpYZ+htom8064JuA12JHGOd1iawOZbRxJxgF0RASMM80nneCkyr lmc84dDfIwkKNDzI47EBQk2CtcznXRigUZF9X7MNb/9dD02RVJCx+7y8QYmsRVA8gyjm UXlQ== X-Gm-Message-State: ACrzQf0M0em6zFr95qm2SthdvyL9/gDMM3MuBHz9QO0Nfn/pY/6dmmEk QYYjVLF9cSic04k5x3UV8oM= X-Received: by 2002:a63:6bc5:0:b0:460:bd9a:64b8 with SMTP id g188-20020a636bc5000000b00460bd9a64b8mr39144875pgc.257.1667728288714; Sun, 06 Nov 2022 01:51:28 -0800 (PST) Received: from localhost.localdomain (124x33x176x97.ap124.ftth.ucom.ne.jp. [124.33.176.97]) by smtp.gmail.com with ESMTPSA id b5-20020aa79505000000b0056bfd4a2702sm2368677pfp.45.2022.11.06.01.51.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Nov 2022 01:51:28 -0800 (PST) Sender: Vincent Mailhol From: Vincent Mailhol To: x86@kernel.org, Ingo Molnar , Borislav Petkov Cc: Nick Desaulniers , Thomas Gleixner , linux-kernel@vger.kernel.org, Yury Norov , llvm@lists.linux.dev, Vincent Mailhol , Borislav Petkov Subject: [PATCH v1 2/2] x86/asm/bitops: Use __builtin_clz*() to evaluate constant expressions Date: Sun, 6 Nov 2022 18:51:06 +0900 Message-Id: <20221106095106.849154-3-mailhol.vincent@wanadoo.fr> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> References: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748740112614044567?= X-GMAIL-MSGID: =?utf-8?q?1748740112614044567?= GCC and clang offers the __builtin_clz(x) and __builtin_clzll(x) functions which return the number of leading zero bits in x. c.f. [1]. By a simple subtraction, we can derive below equivalences: * For fls: Aside of the x = 0 special case, fls(x) is equivalent to BITS_PER_TYPE(x) - __builtin_clz(x). * For fls64: Aside of the x = 0 special case, fls64(x) is equivalent to BITS_PER_TYPE(x) - __builtin_clzll(x). __builtin_clzll() takes an unsigned long long as argument. We choose this version because BITS_PER_LONG_LONG is defined as 64 bits for all architecture making this flavor the most portable one. A BUILD_BUG_ON() safety net is added. When used on constant expressions, the compiler is only able to fold the builtin version (c.f. [2]). However, for non constant expressions, the kernel inline assembly results in better code for both GCC and clang. Use __builtin_constant_p() to select between the kernel's fls()/fls64() __builtin_clz()/__builtin_clzll() depending on whether the argument is constant or not. While renaming fls64() to variable_fls64(), change the argument type from __64 to u64 because we are not in an uapi header. [1] Built-in Functions Provided by GCC: https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html [2] commit 146034fed6ee ("x86/asm/bitops: Use __builtin_ffs() to evaluate constant expressions") CC: Borislav Petkov CC: Nick Desaulniers CC: Yury Norov Signed-off-by: Vincent Mailhol --- arch/x86/include/asm/bitops.h | 57 ++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index a31453d7686d..58fb2fc49760 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -333,18 +333,15 @@ static __always_inline int variable_ffs(int x) */ #define ffs(x) (__builtin_constant_p(x) ? __builtin_ffs(x) : variable_ffs(x)) -/** - * fls - find last set bit in word - * @x: the word to search - * - * This is defined in a similar way as the libc and compiler builtin - * ffs, but returns the position of the most significant set bit. - * - * fls(value) returns 0 if value is 0 or the position of the last - * set bit if value is nonzero. The last (most significant) bit is - * at position 32. - */ -static __always_inline int fls(unsigned int x) +static __always_inline int constant_fls(unsigned int x) +{ + if (!x) + return 0; + + return BITS_PER_TYPE(x) - __builtin_clz(x); +} + +static __always_inline int variable_fls(unsigned int x) { int r; @@ -375,18 +372,30 @@ static __always_inline int fls(unsigned int x) } /** - * fls64 - find last set bit in a 64-bit word + * fls - find last set bit in word * @x: the word to search * * This is defined in a similar way as the libc and compiler builtin - * ffsll, but returns the position of the most significant set bit. + * ffs, but returns the position of the most significant set bit. * - * fls64(value) returns 0 if value is 0 or the position of the last + * fls(value) returns 0 if value is 0 or the position of the last * set bit if value is nonzero. The last (most significant) bit is - * at position 64. + * at position 32. */ +#define fls(x) (__builtin_constant_p(x) ? constant_fls(x) : variable_fls(x)) + #ifdef CONFIG_X86_64 -static __always_inline int fls64(__u64 x) +static __always_inline int constant_fls64(u64 x) +{ + BUILD_BUG_ON(sizeof(unsigned long long) != sizeof(x)); + + if (!x) + return 0; + + return BITS_PER_TYPE(x) - __builtin_clzll(x); +} + +static __always_inline int variable_fls64(u64 x) { int bitpos = -1; /* @@ -399,6 +408,20 @@ static __always_inline int fls64(__u64 x) : "rm" (x)); return bitpos + 1; } + +/** + * fls64 - find last set bit in a 64-bit word + * @x: the word to search + * + * This is defined in a similar way as the libc and compiler builtin + * ffsll, but returns the position of the most significant set bit. + * + * fls64(value) returns 0 if value is 0 or the position of the last + * set bit if value is nonzero. The last (most significant) bit is + * at position 64. + */ +#define fls64(x) \ + (__builtin_constant_p(x) ? constant_fls64(x) : variable_fls64(x)) #else #include #endif