From patchwork Fri Nov 25 02:33:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Mailhol X-Patchwork-Id: 25788 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3729208wrr; Thu, 24 Nov 2022 18:42:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf7lKQffVV6od4yNZ40IBZWYzixmBFNGV60QYxv9d4CIY/WOT1sOippEMH80UoR73isCl1jq X-Received: by 2002:a17:90b:3d90:b0:212:de1c:a007 with SMTP id pq16-20020a17090b3d9000b00212de1ca007mr45670259pjb.30.1669344139450; Thu, 24 Nov 2022 18:42:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669344139; cv=none; d=google.com; s=arc-20160816; b=im0x/tppMoGbb87rNFyQA1AzQFbKUukoNpJ20+knOl3zjt7SwPF24QzlVTmPFKtLuv xRCIXrGzO6XDw7sKMmxMnblpXkdJiatFo4YeWgtsjKZRuB4aGOtV6Xn31LRTjumsuOU3 kdICUO9jPIRcDgyJQStF6ydbDjqGSVcTyf6dizVMWyn9GAjJDJglYdCRvxy3IH/p+n7W yLC3SyOSbnnd5Raw+OLqC0X8latuHpyUvG5MUTTW7ncAm1pgU/Ju8krVZBxP+B4GnEz+ sN1Y/5C1ObuTORSiacCY3Z/j6XSrAQUgCcpfjVWkwTVzPuVfk/08fj9BCAeJr9G7wTx7 g0Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=v3tX8c48SkCwp4614P1MZhKByQyd6IaaDpUY4QZpiCs=; b=Hl6XyKsqmqlV3uHmTnv3kVxxjAgoQB1rtYbjTWH6ltBGQ+3ASK8oyZYgjAucUBijRg KTMNXVcS5hsjiBYp4q1nwBVGfN2WTo35ici+wt+RoMXk0SbOgC0q7w6gAGGEAkK5USAY EW1ANhVFyWB1RtIZ9IShW+fgAs5JvGZWDx4iB10FqdB2GcM8QBZS+G/JeyWugqMy4u3C wyucj3k8SSy4xMqygS+m4jcyKnyhSev/VvCzbEYA25U6TuVpWjHCSx4yk84SNlJgAlRH B65xFVVpEcKb5OJjbnfaqNnNNYfT+Uj6tCVDDS67VOrknNUIq2TXCIeVr5TRdl0M4qbM 0Whw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=K8mWPNqL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 20-20020a631854000000b0047715c95650si783425pgy.534.2022.11.24.18.42.06; Thu, 24 Nov 2022 18:42:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=K8mWPNqL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229707AbiKYCeA (ORCPT + 99 others); Thu, 24 Nov 2022 21:34:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229701AbiKYCd5 (ORCPT ); Thu, 24 Nov 2022 21:33:57 -0500 Received: from mail-pj1-x102d.google.com (mail-pj1-x102d.google.com [IPv6:2607:f8b0:4864:20::102d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A2A1201A6 for ; Thu, 24 Nov 2022 18:33:56 -0800 (PST) Received: by mail-pj1-x102d.google.com with SMTP id b13-20020a17090a5a0d00b0021906102d05so1457364pjd.5 for ; Thu, 24 Nov 2022 18:33:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=v3tX8c48SkCwp4614P1MZhKByQyd6IaaDpUY4QZpiCs=; b=K8mWPNqL2s5XUfZHJTMpQCvAD2CWcuE/ue2pBfeMUMxUdsOS8Uvh/liwOaUFUoWlu+ JfeEmpmuzkjipWPfzWH9UOJkvl+ccxupfhUIw3+ucLZyu4T1Pd5YiuFvU94+SBuLfrUP /yZY0nj8RaY9lS24bnFN5RFJropy5/eHJ1Ur4fi0hytRRWJO+SfBMvBDYlR8ABU/ZSuh PqcKUjBX96UMIjvrKy/NfoZyHfgw+R5/LSko+Bn0SKn0EN0B0ccEyLvDmcou1LKXyVfT 4/FYkZM7DVtPWZ+IJojx+TvwB83xC4DpFsp8qOq42pl2eEJe0bCSg6F5dSfmcQdHTxML RbNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=v3tX8c48SkCwp4614P1MZhKByQyd6IaaDpUY4QZpiCs=; b=8H6yealCHJQTOovjHtbxKBXUW6SZZR4J2PEzm7wJ+hyHLI3iOSp22kGwQ19U+z2BwI 9RTkI/o64gTpmyaKdVPVaQi+5TqVwi9NU7wXi4/gGoMnRpuDwZGFDOSy6ze3RcFlpX3s AGijuMgOv6R2M85pACDdPHV+Z/lkoua6O+V11vyuXJGd0BavowxLrs9jMUXc8BRalzNg x8ZQinZ0n+Kanr5mlUMlRftxWVIUGWSxo6rCxcTb9UtJK5Gd0zmJLmj/iNl5sF1hJej2 c7R9W+hg7o+OktjAgue/Lz45SnT2SMPLmfs0pg5W9mOfQCFD77ZY15UNGXZ7XuoGW7Ug GXWg== X-Gm-Message-State: ANoB5pmNde/6aXRap8oFUaaZ+WWs1xqUB1tPw0GCx5hqhNU/ufVE6o9I NqHFAjQyiZxRJOZGwBsAUuA= X-Received: by 2002:a17:902:cec8:b0:189:2b93:3f9f with SMTP id d8-20020a170902cec800b001892b933f9fmr17825231plg.109.1669343635994; Thu, 24 Nov 2022 18:33:55 -0800 (PST) Received: from localhost.localdomain (124x33x176x97.ap124.ftth.ucom.ne.jp. [124.33.176.97]) by smtp.gmail.com with ESMTPSA id m127-20020a625885000000b0056bc5ad4862sm1893822pfb.28.2022.11.24.18.33.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Nov 2022 18:33:55 -0800 (PST) Sender: Vincent Mailhol From: Vincent Mailhol To: x86@kernel.org, Ingo Molnar , Borislav Petkov Cc: Nick Desaulniers , Thomas Gleixner , linux-kernel@vger.kernel.org, Yury Norov , llvm@lists.linux.dev, Peter Zijlstra , Vincent Mailhol , Borislav Petkov Subject: [PATCH v2 1/2] x86/asm/bitops: Replace __fls() by its generic builtin implementation Date: Fri, 25 Nov 2022 11:33:29 +0900 Message-Id: <20221125023330.195581-2-mailhol.vincent@wanadoo.fr> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221125023330.195581-1-mailhol.vincent@wanadoo.fr> References: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> <20221125023330.195581-1-mailhol.vincent@wanadoo.fr> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748740039693708765?= X-GMAIL-MSGID: =?utf-8?q?1750434200255788742?= Below snippet: #include unsigned int foo(unsigned long word) { return __fls(word); } produces this on GCC 12.1.0: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 48 89 fb mov %rdi,%rbx d: e8 00 00 00 00 call 12 12: 48 0f bd c3 bsr %rbx,%rax 16: 5b pop %rbx 17: 31 ff xor %edi,%edi 19: e9 00 00 00 00 jmp 1e and that on clang 14.0.6: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 50 push %rax b: 48 89 fb mov %rdi,%rbx e: e8 00 00 00 00 call 13 13: 48 89 1c 24 mov %rbx,(%rsp) 17: 48 0f bd 04 24 bsr (%rsp),%rax 1c: 48 83 c4 08 add $0x8,%rsp 20: 5b pop %rbx 21: c3 ret The implementation from [1] produces the exact same code on GCC and below code on clang: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 48 89 fb mov %rdi,%rbx d: e8 00 00 00 00 call 12 12: 48 0f bd c3 bsr %rbx,%rax 16: 5b pop %rbx 17: c3 ret The builtin implementation is better for two reasons: 1/ it saves two instructions on clang (a push and a stack pointer decrement). 2/ when used on constant expressions, the compiler is only able to fold the builtin version (c.f. [2]). For those two reasons, replace the assembly implementation by its builtin counterpart. [1] https://elixir.bootlin.com/linux/v6.0/source/include/asm-generic/bitops/builtin-__fls.h [2] commit 146034fed6ee ("x86/asm/bitops: Use __builtin_ffs() to evaluate constant expressions") CC: Borislav Petkov CC: Yury Norov Signed-off-by: Vincent Mailhol Reviewed-by: Nick Desaulniers --- arch/x86/include/asm/bitops.h | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index 2edf68475fec..a31453d7686d 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -285,19 +285,7 @@ static __always_inline unsigned long variable_ffz(unsigned long word) (unsigned long)__builtin_ctzl(~word) : \ variable_ffz(word)) -/* - * __fls: find last set bit in word - * @word: The word to search - * - * Undefined if no set bit exists, so code should check against 0 first. - */ -static __always_inline unsigned long __fls(unsigned long word) -{ - asm("bsr %1,%0" - : "=r" (word) - : "rm" (word)); - return word; -} +#include #undef ADDR From patchwork Fri Nov 25 02:33:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vincent Mailhol X-Patchwork-Id: 25789 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3729522wrr; Thu, 24 Nov 2022 18:43:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf5tMrpFcSK2RPMKvqyRiPo6U6Bgv3SFPXZfFgr5wsrgk5TB1AQQykWSveS+nzCAO2da55+u X-Received: by 2002:a17:902:a9c6:b0:188:52df:769e with SMTP id b6-20020a170902a9c600b0018852df769emr15811773plr.30.1669344199060; Thu, 24 Nov 2022 18:43:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669344199; cv=none; d=google.com; s=arc-20160816; b=ifkDGOpOYId+pAKf0YHak2N6kuctqraUnif1hikXnWCSg7W+L5dmMoVJHvCm7fpAYn Avonvtis/M6lR0gZ8UYtmK/Sv6e61JN506rV1KOCx+DL/XqeVj9Ssz0TamiUWW+Snz5c IuGvecpYOchwdG37m01UNiDHClN13Q3oedqbJzjxR4u0jWjL/sgqcKkzq75/aRejXKwE oCzPE6UnVyF94pqepRp90d5PYaJNax5OXR19lE3YgbWVCHleCl3gy4oWlsYADl3rzH7y rR0rQGkXXhGHnEH9e8XGggJzC94K6o9sAazabh4XlP9L9QD9Zx+NMQS1ee1Lf8IwT9Ex 6XFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from:sender :dkim-signature; bh=3yAoT1kLjr2RjOiYLwcVMKA0GhtRYKf2X90lTYA9Hn4=; b=DfbBHimYdFv7zvbYt+Fbke0NdfrdUlMKqHwFGQlA6MEjMVkYz1msMg9HkRuJSQpE24 6vbrWZe2qy5ecpNUS9gB++O9QN5iNbYQE1AXaAH/NRij34CQOgloBk75HnePjHOBhgVt in3zE9EmWtNL5TWcawZlb28Ut0nDEA9XMzawRRayDWTUlpH29sMRciQq9DaOpcxJS/3g VulCZn1ApiSMidzQnWeMm2Y1Psk3GCSSxipKcCv7Gym2ljUTvmKKNKJpgyGN0P/Ed+yY FDZdhAsB4OnTE6px4QyEojnwj84iJ/HQm7Kr4JAEqZnM96RATLWpZxy4ULL0V802rSjb nxVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="T/qdGX3i"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p1-20020a056a000a0100b00562331a3562si2920519pfh.130.2022.11.24.18.43.04; Thu, 24 Nov 2022 18:43:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="T/qdGX3i"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229718AbiKYCeC (ORCPT + 99 others); Thu, 24 Nov 2022 21:34:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229711AbiKYCeA (ORCPT ); Thu, 24 Nov 2022 21:34:00 -0500 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2ECCF23173 for ; Thu, 24 Nov 2022 18:33:59 -0800 (PST) Received: by mail-pg1-x52d.google.com with SMTP id f9so2827670pgf.7 for ; Thu, 24 Nov 2022 18:33:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=3yAoT1kLjr2RjOiYLwcVMKA0GhtRYKf2X90lTYA9Hn4=; b=T/qdGX3ipHc7MMkeq8ke6n/7Kx1Mu93QbEcxMleGD7kJGuP0VXCtHwETV4Z+K9pSOk KbAU/aTvuEZfcAhyhtq5/G7Xe6Egm+NBcRRXpHcaPOcCvERwy0p0TdVMT3b86T4M4qTY UbU2XcbRqpGpE+2omjBcyT7B8iONaRy365xNp++rW8bNWpr9Yb/t7PL6iDOlNyQ0cWz0 /mlyCSXYUhKEV37gn3VKtVq8oQDKdAxT7zZeVN2nPoSm1g6ICWSowekz0heRzPrOh1n5 HH6biCVATxCQHWVQ5VACH20U3Rn6S32lHun0tnsB//x/zdIP0ciqXAG+/Qd3p9OPd52o kMhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=3yAoT1kLjr2RjOiYLwcVMKA0GhtRYKf2X90lTYA9Hn4=; b=x0vucnPWeREPeenpsG5mxhFzLB6FXQh0d8WR2jEComplQCXkv/swS6qEAo0DZubVAB RqPkTD+5EsEgY8USpKL1ymeUF0syBDNexPsBFxzFKvpgzhp3SwaR03lzqOxg1l1vmT0F eIyXPSxZPB3OQdw56y1mGaB3YvUN6jCNCm+iOqXkOIvDeQlCgvaYDFloGc3LoDE/SstM BhhepHG2o03IQdM1sJGOYVXqprwuyBXY7fYQDtMZXKAHEaeQ6ZoN7v9BrSjkAnZFNLvt plxp4N3S4WZ7jRFAAXJBCkvRhiuMsCUnaTeExBLXdiUMysqf1FnP6izvI5SF/HPBjfV2 Pe6A== X-Gm-Message-State: ANoB5pl1GjOUB7wN1nprdNwqxeMs2BNrpYhJ30E+GqahRL9SC0Of6InE 9XyD9GgOXGGVFNDZuFjP5Yg= X-Received: by 2002:a05:6a00:d59:b0:56e:5bcd:d025 with SMTP id n25-20020a056a000d5900b0056e5bcdd025mr16409954pfv.65.1669343638652; Thu, 24 Nov 2022 18:33:58 -0800 (PST) Received: from localhost.localdomain (124x33x176x97.ap124.ftth.ucom.ne.jp. [124.33.176.97]) by smtp.gmail.com with ESMTPSA id m127-20020a625885000000b0056bc5ad4862sm1893822pfb.28.2022.11.24.18.33.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Nov 2022 18:33:58 -0800 (PST) Sender: Vincent Mailhol From: Vincent Mailhol To: x86@kernel.org, Ingo Molnar , Borislav Petkov Cc: Nick Desaulniers , Thomas Gleixner , linux-kernel@vger.kernel.org, Yury Norov , llvm@lists.linux.dev, Peter Zijlstra , Vincent Mailhol , Borislav Petkov Subject: [PATCH v2 2/2] x86/asm/bitops: Use __builtin_clz*() to evaluate constant expressions Date: Fri, 25 Nov 2022 11:33:30 +0900 Message-Id: <20221125023330.195581-3-mailhol.vincent@wanadoo.fr> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221125023330.195581-1-mailhol.vincent@wanadoo.fr> References: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> <20221125023330.195581-1-mailhol.vincent@wanadoo.fr> MIME-Version: 1.0 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748740112614044567?= X-GMAIL-MSGID: =?utf-8?q?1750434262937286036?= GCC and clang offers the __builtin_clz(x) and __builtin_clzll(x) functions which return the number of leading zero bits in x. c.f. [1]. By a simple subtraction, we can derive below equivalences: * For fls: Aside of the x = 0 special case, fls(x) is equivalent to BITS_PER_TYPE(x) - __builtin_clz(x). * For fls64: Aside of the x = 0 special case, fls64(x) is equivalent to BITS_PER_TYPE(x) - __builtin_clzll(x). __builtin_clzll() takes an unsigned long long as argument instead of a u64 which is fine because those two types are equivalent. Regardless, add a BUILD_BUG_ON() safety net is added to formally assert that u64 and unsigned long long are the same. When used on constant expressions, the compiler is only able to fold the builtin version (c.f. [2]). However, for non constant expressions, the kernel inline assembly results in better code for both GCC and clang. Use __builtin_constant_p() to select between the kernel's fls()/fls64() and __builtin_clz()/__builtin_clzll() depending on whether the argument is constant or not. While renaming fls64() to variable_fls64(), change the argument type from __64 to u64 because we are not in an uapi header. [1] Built-in Functions Provided by GCC: https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html [2] commit 146034fed6ee ("x86/asm/bitops: Use __builtin_ffs() to evaluate constant expressions") CC: Borislav Petkov CC: Nick Desaulniers CC: Yury Norov Signed-off-by: Vincent Mailhol --- arch/x86/include/asm/bitops.h | 56 ++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index a31453d7686d..d10e774af343 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -333,18 +333,15 @@ static __always_inline int variable_ffs(int x) */ #define ffs(x) (__builtin_constant_p(x) ? __builtin_ffs(x) : variable_ffs(x)) -/** - * fls - find last set bit in word - * @x: the word to search - * - * This is defined in a similar way as the libc and compiler builtin - * ffs, but returns the position of the most significant set bit. - * - * fls(value) returns 0 if value is 0 or the position of the last - * set bit if value is nonzero. The last (most significant) bit is - * at position 32. - */ -static __always_inline int fls(unsigned int x) +static __always_inline int constant_fls(unsigned int x) +{ + if (!x) + return 0; + + return BITS_PER_TYPE(x) - __builtin_clz(x); +} + +static __always_inline int variable_fls(unsigned int x) { int r; @@ -375,18 +372,29 @@ static __always_inline int fls(unsigned int x) } /** - * fls64 - find last set bit in a 64-bit word + * fls - find last set bit in word * @x: the word to search * * This is defined in a similar way as the libc and compiler builtin - * ffsll, but returns the position of the most significant set bit. + * ffs, but returns the position of the most significant set bit. * - * fls64(value) returns 0 if value is 0 or the position of the last + * fls(value) returns 0 if value is 0 or the position of the last * set bit if value is nonzero. The last (most significant) bit is - * at position 64. + * at position 32. */ +#define fls(x) (__builtin_constant_p(x) ? constant_fls(x) : variable_fls(x)) + #ifdef CONFIG_X86_64 -static __always_inline int fls64(__u64 x) +static __always_inline int constant_fls64(u64 x) +{ + if (!x) + return 0; + + BUILD_BUG_ON(!__same_type(x, unsigned long long)); + return BITS_PER_TYPE(x) - __builtin_clzll(x); +} + +static __always_inline int variable_fls64(u64 x) { int bitpos = -1; /* @@ -399,6 +407,20 @@ static __always_inline int fls64(__u64 x) : "rm" (x)); return bitpos + 1; } + +/** + * fls64 - find last set bit in a 64-bit word + * @x: the word to search + * + * This is defined in a similar way as the libc and compiler builtin + * ffsll, but returns the position of the most significant set bit. + * + * fls64(value) returns 0 if value is 0 or the position of the last + * set bit if value is nonzero. The last (most significant) bit is + * at position 64. + */ +#define fls64(x) \ + (__builtin_constant_p(x) ? constant_fls64(x) : variable_fls64(x)) #else #include #endif