Message ID | 20221020153540.8226-1-ubizjak@gmail.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4242:0:0:0:0:0 with SMTP id s2csp185220wrr; Thu, 20 Oct 2022 08:51:54 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6DCix0gtPJo0JqnPP2tbRsvqlfcpuMEaSdIXzCf7BFQcncFyOmcK3SknP0O4C0cw4nm6TM X-Received: by 2002:a05:6402:371b:b0:460:ff7d:f511 with SMTP id ek27-20020a056402371b00b00460ff7df511mr2251576edb.148.1666281114044; Thu, 20 Oct 2022 08:51:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666281114; cv=none; d=google.com; s=arc-20160816; b=IIo9nwoHYYcDjhppOosXU/M3frok7X5B4TDse6b+00Xj1qnFBBv7J91T7oEIcW/hY6 CV1v4duoJ/evp81NGww/HPJJFkNHk5u/rxd/cYjnSoxSILs/aF9thcrXvD2UgYtWdZic Uz8cBxnkXlajjoMEu028KiAAhpD858dEpvllJW1LnQqMjlemkCYyHkyn/Ps1sFaNexWi Q8828qtjWzQg6HCPly1TBSShgh5EBS9qPSi+5nIA2OSVfTEx+SdxE+Q2mcVm9HAED5ND 8MI9BHCpL6r/DA5Ma3riuWs+v8SYBN91RwOv4RSU/E7KMztfbubR+aE1RoB1PtrwiMBX NbMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=Xz6rkNFT+N3+z5GHliBzDaeRK81yOrfk7eIGZzmG81Q=; b=AdSrDW/ZhG1evnl34Epuq610HwhtlhnxsO8S+Zfx2it+ZP+PeLyRvseyzrPzaDjlbP wqrenJGI0jINBDqbZNguWWWAc5j+05rQnnFWHuJbRZ6r4f5dBb7GhwGSCp8OxPEFV51O z5z+Ua1rYwe1UCenZPDlyAxCBK9uM4MDq7RfMLEqsgeq+svjrKo2rX4bfB1u8F/sxJHC UG4GBVeoveDbndEwPtYW/YlpwqhQqJcLo2p0tIvo8nIHYjcCgbwRzApY+txr7WMbopuv nmC8ElMZK9xrGhsnT1RBkPwcelO9QnhJ4ILjsDln63zxFJqPE+DdXXPFVCG1TEd/FjtO 58Ag== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=LrPvFB8g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e14-20020a056402190e00b0045b965cf1dasi19879438edz.175.2022.10.20.08.51.27; Thu, 20 Oct 2022 08:51:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=LrPvFB8g; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230143AbiJTPf6 (ORCPT <rfc822;ruipengqi7@gmail.com> + 99 others); Thu, 20 Oct 2022 11:35:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229717AbiJTPf5 (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Thu, 20 Oct 2022 11:35:57 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE98A1805BF for <linux-kernel@vger.kernel.org>; Thu, 20 Oct 2022 08:35:53 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id bg9-20020a05600c3c8900b003bf249616b0so2493192wmb.3 for <linux-kernel@vger.kernel.org>; Thu, 20 Oct 2022 08:35:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=Xz6rkNFT+N3+z5GHliBzDaeRK81yOrfk7eIGZzmG81Q=; b=LrPvFB8gDO64B7UiPcOEb4r4czILVjpCtcxZINcfAFDa3JVnVixw8m/oGT/SX21zP/ Gr0oby3J6kWKup8VmdmabJ/RmMiUTF41qbixfM8VEDJItjOUfbI+i4eL5pPeXKNAC14E dFkAG4sTNCeJGTXP9FerhSiUuerkXXeqxmrzw4EjmqkkUs0TqQd5EofrRaek32k73Lrv hovubnAzUZ9PQje3IfBFZK8uo/EOI95RVgx3leGIGBT81cFNKVXmL2Sc4MI9FZ2Jo2o3 E4+/DFmy8GAK+x+xJAmWJ9xFNcSRq7lHXKBFpaqNZHYOCvETpZaQ5vW86279Aqavyawn TvCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Xz6rkNFT+N3+z5GHliBzDaeRK81yOrfk7eIGZzmG81Q=; b=HjTlqOl27t/sUXkJ9zTRd79JGMFY7fObKki/jhbBgB0kKN1vxOZ9ur/Z/104wOgPm+ w7JN186DP6ul6+u1Sm8cIcBezs5MQiYiMuNaLPyZqBMq6FKcOfM/Xc9lMBMuyznYhwlA dff3E0T8vmG0u0g/T34CuCyE+xyAXbU+H/ZmuWSDrnByaTulEWdVEPZimR+NamFRA/oF HqmCmHSspaQlrDXm/tNbv4H25TgK7r48wPZYOr7A6S9CJr+LeplmFW/CkSXQg2tN84Df jJC7J34lCpcbMzDmjdwKQOpXn6fOIOujijVyAgQVz6LUPhGD72SWPn7CRWlthPX2iCC0 4Xag== X-Gm-Message-State: ACrzQf1q2Aa6vjFOiHFvhuOrpi+oACZ94NE3TwgUqEvK5Ao8BdhY2DOO ottUlc5n1QM8hGNKas+oZr4= X-Received: by 2002:a05:600c:4f46:b0:3c6:fb4f:3e1b with SMTP id m6-20020a05600c4f4600b003c6fb4f3e1bmr9922108wmq.159.1666280151918; Thu, 20 Oct 2022 08:35:51 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id u11-20020adff88b000000b0022e2eaa2bdcsm16486515wrp.98.2022.10.20.08.35.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Oct 2022 08:35:51 -0700 (PDT) From: Uros Bizjak <ubizjak@gmail.com> To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak <ubizjak@gmail.com>, Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>, Chaitanya Kulkarni <kch@nvidia.com> Subject: [PATCH] nvmet: use try_cmpxchg in nvmet_update_sq_head Date: Thu, 20 Oct 2022 17:35:40 +0200 Message-Id: <20221020153540.8226-1-ubizjak@gmail.com> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747222385419150470?= X-GMAIL-MSGID: =?utf-8?q?1747222385419150470?= |
Series |
nvmet: use try_cmpxchg in nvmet_update_sq_head
|
|
Commit Message
Uros Bizjak
Oct. 20, 2022, 3:35 p.m. UTC
Use try_cmpxchg instead of cmpxchg (*ptr, old, new) == old in
nvmet_update_sq_head. x86 CMPXCHG instruction returns success in ZF flag, so
this change saves a compare after cmpxchg (and related move instruction in
front of cmpxchg).
Also, try_cmpxchg implicitly assigns old *ptr value to "old" when cmpxchg
fails. There is no need to re-read the value in the loop.
Note that the value from *ptr should be read using READ_ONCE to prevent
the compiler from merging, refetching or reordering the read.
No functional change intended.
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Uros Bizjak <ubizjak@gmail.com>
---
drivers/nvme/target/core.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
Comments
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
On 10/20/22 08:35, Uros Bizjak wrote: > Use try_cmpxchg instead of cmpxchg (*ptr, old, new) == old in > nvmet_update_sq_head. x86 CMPXCHG instruction returns success in ZF flag, so > this change saves a compare after cmpxchg (and related move instruction in > front of cmpxchg). > Is it worth a share delts of assembly instructions of the changes above? as developers on block mailing list are sharing the delta between before and after patch including the assembly. I also hope that you have tested this change with blktests nvme. Either way:- Reviewed-by: ChaItanya Kulkarni <kch@nvidia.com> -ck
On Tue, Oct 25, 2022 at 2:47 AM Chaitanya Kulkarni <chaitanyak@nvidia.com> wrote: > > On 10/20/22 08:35, Uros Bizjak wrote: > > Use try_cmpxchg instead of cmpxchg (*ptr, old, new) == old in > > nvmet_update_sq_head. x86 CMPXCHG instruction returns success in ZF flag, so > > this change saves a compare after cmpxchg (and related move instruction in > > front of cmpxchg). > > > > Is it worth a share delts of assembly instructions of the changes above? > as developers on block mailing list are sharing the delta between before > and after patch including the assembly. The difference in the assembly of nvmet_update_sq_head function is: before: 0000000000001d30 <nvmet_update_sq_head>: 1d30: 48 8b 4f 10 mov 0x10(%rdi),%rcx 1d34: 49 89 f8 mov %rdi,%r8 1d37: 0f b7 71 1a movzwl 0x1a(%rcx),%esi 1d3b: 66 85 f6 test %si,%si 1d3e: 75 14 jne 1d54 <nvmet_update_sq_head+0x24> 1d40: 49 8b 40 08 mov 0x8(%r8),%rax 1d44: 8b 51 1c mov 0x1c(%rcx),%edx 1d47: 66 89 50 08 mov %dx,0x8(%rax) 1d4b: e9 00 00 00 00 jmpq 1d50 <nvmet_update_sq_head+0x20> 1d4c: R_X86_64_PLT32 __x86_return_thunk-0x4 1d50: 0f b7 71 1a movzwl 0x1a(%rcx),%esi 1d54: 8b 79 1c mov 0x1c(%rcx),%edi 1d57: 31 d2 xor %edx,%edx 1d59: 8d 47 01 lea 0x1(%rdi),%eax 1d5c: f7 f6 div %esi 1d5e: 89 f8 mov %edi,%eax 1d60: f0 0f b1 51 1c lock cmpxchg %edx,0x1c(%rcx) 1d65: 49 8b 48 10 mov 0x10(%r8),%rcx 1d69: 39 c7 cmp %eax,%edi 1d6b: 75 e3 jne 1d50 <nvmet_update_sq_head+0x20> 1d6d: 49 8b 40 08 mov 0x8(%r8),%rax 1d71: 8b 51 1c mov 0x1c(%rcx),%edx 1d74: 66 89 50 08 mov %dx,0x8(%rax) 1d78: e9 00 00 00 00 jmpq 1d7d <nvmet_update_sq_head+0x4d> 1d79: R_X86_64_PLT32 __x86_return_thunk-0x4 1d7d: after: 0000000000001d30 <nvmet_update_sq_head>: 1d30: 48 8b 4f 10 mov 0x10(%rdi),%rcx 1d34: 0f b7 51 1a movzwl 0x1a(%rcx),%edx 1d38: 66 85 d2 test %dx,%dx 1d3b: 74 1e je 1d5b <nvmet_update_sq_head+0x2b> 1d3d: 8b 71 1c mov 0x1c(%rcx),%esi 1d40: 44 0f b7 c2 movzwl %dx,%r8d 1d44: 8d 46 01 lea 0x1(%rsi),%eax 1d47: 31 d2 xor %edx,%edx 1d49: 41 f7 f0 div %r8d 1d4c: 89 f0 mov %esi,%eax 1d4e: f0 0f b1 51 1c lock cmpxchg %edx,0x1c(%rcx) 1d53: 48 8b 4f 10 mov 0x10(%rdi),%rcx 1d57: 89 c6 mov %eax,%esi 1d59: 75 10 jne 1d6b <nvmet_update_sq_head+0x3b> 1d5b: 48 8b 47 08 mov 0x8(%rdi),%rax 1d5f: 8b 51 1c mov 0x1c(%rcx),%edx 1d62: 66 89 50 08 mov %dx,0x8(%rax) 1d66: e9 00 00 00 00 jmpq 1d6b <nvmet_update_sq_head+0x3b> 1d67: R_X86_64_PLT32 __x86_return_thunk-0x4 1d6b: 0f b7 51 1a movzwl 0x1a(%rcx),%edx 1d6f: eb cf jmp 1d40 <nvmet_update_sq_head+0x10> 1d71: You can see that in addition to the smaller size of the function, the load of req->sq->size at 1d6b got moved to a cold path. As the main benefit, the load at 1d3d is now out of the loop, and the value in %esi is now provided by cmpxchg insn itself at 1d4e (plus move at 1d57). Unfortunately, the division clobbers %eax, so some reg-reg moves are necessary. Note also that the compare at 1d69 is now gone. > I also hope that you have tested this change with blktests nvme. No, I didn't test the patch that thoroughly, but the change is the same as some similar recent changes in the generic code, so I confirmed the patch by inspecting asm code. OTOH, the kernel booted from a NVME SSD. > Either way:- > > Reviewed-by: ChaItanya Kulkarni <kch@nvidia.com> Thanks! Uros.
Thanks, applied to nvme-6.2.
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c index 14677145bbba..a384a0927aed 100644 --- a/drivers/nvme/target/core.c +++ b/drivers/nvme/target/core.c @@ -695,11 +695,10 @@ static void nvmet_update_sq_head(struct nvmet_req *req) if (req->sq->size) { u32 old_sqhd, new_sqhd; + old_sqhd = READ_ONCE(req->sq->sqhd); do { - old_sqhd = req->sq->sqhd; new_sqhd = (old_sqhd + 1) % req->sq->size; - } while (cmpxchg(&req->sq->sqhd, old_sqhd, new_sqhd) != - old_sqhd); + } while (!try_cmpxchg(&req->sq->sqhd, &old_sqhd, new_sqhd)); } req->cqe->sq_head = cpu_to_le16(req->sq->sqhd & 0x0000FFFF); }