From patchwork Fri Sep 15 18:36:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140880 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1437788vqi; Fri, 15 Sep 2023 19:21:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFoFEeZuFirVMDIJLgS86EeXcYB82AcRIr5Nm09NIBH62qu1JPwhwAhuB7dukstD6jHqcJc X-Received: by 2002:a05:6a00:1516:b0:68f:c6f8:1451 with SMTP id q22-20020a056a00151600b0068fc6f81451mr3955042pfu.16.1694830892843; Fri, 15 Sep 2023 19:21:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694830892; cv=none; d=google.com; s=arc-20160816; b=SbbgA9CuuE39zP998Mm66Gdx5ljtm8ZvSwVSF/O4h/C4MMD3ZUcnOICTVTg1DhdbPJ qsgvC5ifJEG2IsKGAUdM6a/rei0uYSxKNMwQi3mXTbU5+dnSfSN//qw8JTiP88WIp5NV YBvOT/moSfsQ8BfHeiecOUGGCD3zUkjALjTMKWwwNFNNeSgSc+DjBvuxfuH8kqEZeuaT ZVyFU78L6LX9E0iYOUEP+qTyD2b+GCx7vHtpg7PiPgRNnjSvu2vcJOnD/XVEdq6zRjKL APbvsYgKXbHCNWAK3CJNzm8BHluSfCrlFUqwCdjUb7M+aix5cTgSr+2JMBBq6JMflzVb hakA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5PWa8bY4dekYeG890jHuH5+xQm9Lpc2bhtlqgPaBIx8=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=0rVIAOmfdZdi8SIR7PrEkarEvYcnxq/Xdj5bis2+ON7TlN4GJXe0tWmCji095Pj5+H nqeW8/1N2+PEe7GZ0VauGECW+Q7Wobsyt8pBOzZfEElFXZSn0HPVuCWWZnhAgwe+ZKIR rkhhzkKjF3usivl7IKC2JipbxCqb4yZ+DMgZmmOxd6/pcH29iu2T+sdmfxqMe+X+CPJ7 5N+DC9eWxZMmvDlVJNm1EFbl2G8+MKJjO8T2tbUttoGM6gQT2//tAGr+LHmVwS2M//Tp Nqeaee1rlE4Wv0Z7ZuqumJIyD+14NLsShaVGwzbatjqE2B0+iR5za+DQdg+3YXv1cIpj miYg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=m1Ryp53i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id k7-20020a056a00134700b0068e3c235a5bsi4428349pfu.168.2023.09.15.19.21.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 19:21:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=m1Ryp53i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 9A7EC825A840; Fri, 15 Sep 2023 11:39:04 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236708AbjIOSiJ (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42158 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236509AbjIOSh3 (ORCPT ); Fri, 15 Sep 2023 14:37:29 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C487115; Fri, 15 Sep 2023 11:37:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=5PWa8bY4dekYeG890jHuH5+xQm9Lpc2bhtlqgPaBIx8=; b=m1Ryp53inYiFotTtBpCrbEJqhU mqeis08LHa2btJKQJ+ja28v2D+DIGo1PD25gPHrnN9TPG1PJSfbMS61hIUlhlstpyTQjIqdgbftjX PKnvMQthLiws4zgTBgwNPqwd44Jw2/kSpPLzxZVrlIQ8rgj/8mygtAqjlOPT02FIi3ho27qCUtDV/ sLH8hK+zALH4dBoBCo0Mzh3R1xo9jq3RggHQM6urHIgvQ7FipCt0e/C6ufa3pb1CRKC48NeCwtADq lbN8JXJvVC4a8Xir81JjqXJ8YoGXWIm1YAzIINL4KbLWtMac+XL7H+tLJLi1Ob4XZUQIgjS2ewfGY FaeKtW4w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIN-3p; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 01/17] iomap: Hold state_lock over call to ifs_set_range_uptodate() Date: Fri, 15 Sep 2023 19:36:51 +0100 Message-Id: <20230915183707.2707298-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:39:04 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777158998071297358 X-GMAIL-MSGID: 1777158998071297358 This is really preparation for the next patch, but it lets us call folio_mark_uptodate() in just one place instead of two. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index ae8673ce08b1..4c05fd457ee7 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -57,30 +57,32 @@ static inline bool ifs_block_is_uptodate(struct iomap_folio_state *ifs, return test_bit(block, ifs->state); } -static void ifs_set_range_uptodate(struct folio *folio, +static bool ifs_set_range_uptodate(struct folio *folio, struct iomap_folio_state *ifs, size_t off, size_t len) { struct inode *inode = folio->mapping->host; unsigned int first_blk = off >> inode->i_blkbits; unsigned int last_blk = (off + len - 1) >> inode->i_blkbits; unsigned int nr_blks = last_blk - first_blk + 1; - unsigned long flags; - spin_lock_irqsave(&ifs->state_lock, flags); bitmap_set(ifs->state, first_blk, nr_blks); - if (ifs_is_fully_uptodate(folio, ifs)) - folio_mark_uptodate(folio); - spin_unlock_irqrestore(&ifs->state_lock, flags); + return ifs_is_fully_uptodate(folio, ifs); } static void iomap_set_range_uptodate(struct folio *folio, size_t off, size_t len) { struct iomap_folio_state *ifs = folio->private; + unsigned long flags; + bool uptodate = true; - if (ifs) - ifs_set_range_uptodate(folio, ifs, off, len); - else + if (ifs) { + spin_lock_irqsave(&ifs->state_lock, flags); + uptodate = ifs_set_range_uptodate(folio, ifs, off, len); + spin_unlock_irqrestore(&ifs->state_lock, flags); + } + + if (uptodate) folio_mark_uptodate(folio); } From patchwork Fri Sep 15 18:36:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140846 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1411516vqi; Fri, 15 Sep 2023 18:01:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHCwrbVLjC+NtzeWuvYL9JBEXTlLYnx5nzLkxjm070OM+zGsegkkcrwgGGjniSWoyhL1Flj X-Received: by 2002:a05:6359:668d:b0:143:1063:d1f4 with SMTP id so13-20020a056359668d00b001431063d1f4mr1862243rwb.13.1694826068931; Fri, 15 Sep 2023 18:01:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694826068; cv=none; d=google.com; s=arc-20160816; b=GfSkOjiNtJc0IEJJoEHFDIPNa8ZUs6Ey+AdHpNxSBNpvj8zTJT3LetSis4AMdLWmhy iy9hHSWH0dDWqlZfiEDxL23vuFNtjXyQWeCFCDXWdPuVRc/pgs4Q/EWQNC8tQAPUxBuS EEPX+Xpb2kNyjNxcDyYKt0aLyQVkHyb8Z7S/ApYwxl52i7NhImabr1HUhQeIy3JXx6FC I41Kwz6Iieli5HystaOtlIbnn/oHtFMpBN9nc+daQNDnFAQ8dW1wIfRXW7rqAvTrc0eX cDrE6zO084OAGQzAN67BNYVPvnl8CgVnGIkOdzd4XaHmw+H1nx7bIE2EaHQpH8PwH89f /2iw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3RDv1UzFpsOkAGzAbd6eo8f3gCp7HajbbasO9DDS7OQ=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=0HARAX4CkohUBMA5/enqUAgY32MDlGW+fjQ4FVHfmDskDS59WCdmXoc5DsMI1ovb+E kQskBmfh2iHYM1Kdo6VECQMGPLvFeA+1GATCEXSZZOzdsnSUThzHBM9qtO9R7TXKhjqD hxj70pgs7ac+PD6m76DqS61Lb7SKX0d8HBGM6FV+swFrhSO6zkda4pwF5Fw22I6rmstn jLnmF8xPp/HVj5zuv28dZ0NpK49WcUZqBP8bSppiLwls+3PH1+uWnIvrKkr6KrmKaDVP 1+n8APT/g3ZJWa+KwKafps9cN8iqr/A1AWzvi/FYLNdxHEp2S5c5vv+3FHa2SSZJSY86 Iuzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b="l7/C+pIB"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id e24-20020a656798000000b0056963f38643si4011354pgr.280.2023.09.15.18.01.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 18:01:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b="l7/C+pIB"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id A590B80879C3; Fri, 15 Sep 2023 11:38:35 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236728AbjIOSiN (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236368AbjIOShf (ORCPT ); Fri, 15 Sep 2023 14:37:35 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 859E0115; Fri, 15 Sep 2023 11:37:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3RDv1UzFpsOkAGzAbd6eo8f3gCp7HajbbasO9DDS7OQ=; b=l7/C+pIBN/0+OfCBSo8I+LYeOy mF+TpEuZNJw8+JiNsmRfWrhjnNLgd6Txx79Xz2XVo7OVest1Qi+dglnG1T25vyb0Xpc8NLATE9/Au ybl6Gu/i/Td0hGN+ZdVvfD8LF/2mk/oF4PUmAsSRMpnyTXOei1O3inZEkAt6HEz7IaCtxJRYNOxM0 6GS70vkf9aU86BccdSHzK8DlUV7uejZVS8XHXtB4KyvwB8hapKImjlueOcr/dVW2WJbm4DdFvYPPw nbKZFLvA2rCr2J8VDHR9zw1FjnvHE8CpEr6SiUz+TbiJBEjFRsjX3erwBo5P1AshO50tSAoi8YhYN ECdR01vg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIQ-6i; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 02/17] iomap: Protect read_bytes_pending with the state_lock Date: Fri, 15 Sep 2023 19:36:52 +0100 Message-Id: <20230915183707.2707298-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:35 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777153939590572562 X-GMAIL-MSGID: 1777153939590572562 Perform one atomic operation (acquiring the spinlock) instead of two (spinlock & atomic_sub) per read completion. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 4c05fd457ee7..cade15b70627 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -29,9 +29,9 @@ typedef int (*iomap_punch_t)(struct inode *inode, loff_t offset, loff_t length); * and I/O completions. */ struct iomap_folio_state { - atomic_t read_bytes_pending; - atomic_t write_bytes_pending; spinlock_t state_lock; + unsigned int read_bytes_pending; + atomic_t write_bytes_pending; /* * Each block has two bits in this bitmap: @@ -183,7 +183,7 @@ static void ifs_free(struct folio *folio) if (!ifs) return; - WARN_ON_ONCE(atomic_read(&ifs->read_bytes_pending)); + WARN_ON_ONCE(ifs->read_bytes_pending != 0); WARN_ON_ONCE(atomic_read(&ifs->write_bytes_pending)); WARN_ON_ONCE(ifs_is_fully_uptodate(folio, ifs) != folio_test_uptodate(folio)); @@ -250,19 +250,33 @@ static void iomap_adjust_read_range(struct inode *inode, struct folio *folio, *lenp = plen; } -static void iomap_finish_folio_read(struct folio *folio, size_t offset, +static void iomap_finish_folio_read(struct folio *folio, size_t off, size_t len, int error) { struct iomap_folio_state *ifs = folio->private; + unsigned long flags; + bool uptodate; + bool finished = true; + + if (ifs) + spin_lock_irqsave(&ifs->state_lock, flags); if (unlikely(error)) { - folio_clear_uptodate(folio); + uptodate = false; folio_set_error(folio); } else { - iomap_set_range_uptodate(folio, offset, len); + uptodate = !ifs || ifs_set_range_uptodate(folio, ifs, off, len); } - if (!ifs || atomic_sub_and_test(len, &ifs->read_bytes_pending)) + if (ifs) { + ifs->read_bytes_pending -= len; + finished = !ifs->read_bytes_pending; + spin_unlock_irqrestore(&ifs->state_lock, flags); + } + + if (uptodate) + folio_mark_uptodate(folio); + if (finished) folio_unlock(folio); } @@ -360,8 +374,11 @@ static loff_t iomap_readpage_iter(const struct iomap_iter *iter, } ctx->cur_folio_in_bio = true; - if (ifs) - atomic_add(plen, &ifs->read_bytes_pending); + if (ifs) { + spin_lock_irq(&ifs->state_lock); + ifs->read_bytes_pending += plen; + spin_unlock_irq(&ifs->state_lock); + } sector = iomap_sector(iomap, pos); if (!ctx->bio || From patchwork Fri Sep 15 18:36:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140842 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1409873vqi; Fri, 15 Sep 2023 17:55:41 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGFclM+6cZQ0N+XJf5DTmjv80Yonlyy+PjShBJB3IDnRX/WUfEUk4eBT/qWV8R3CC1dFReN X-Received: by 2002:a17:903:1c1:b0:1c3:323f:f531 with SMTP id e1-20020a17090301c100b001c3323ff531mr3321272plh.20.1694825740908; Fri, 15 Sep 2023 17:55:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694825740; cv=none; d=google.com; s=arc-20160816; b=vAotIcTQCXr9TJd/VUnWgSCzo493fzLs7c3mcZLbGpbJ02AQx5VAl6YbrFLQ+D/F8V X+ZaZoDlYQ0F2OC+QejNE4IPf3pg7Tu4NJ9iTejQJd+hYiig/AHwwpCDPxjTwLZkyWq9 PDgIIwaYFTnmbVi6Wc1zpjK73YqrJml7X6Uh1F3qs7PxgmIhaRRoMRw0Sr+BUmwcgJIf 3GSHPrDpshgmJUg7gU/kNQwXKpgzWEdZvHShz1wpRCgqAxtCqTKSlDbvx9uwHUkSZxud xEIcLUdytMM42QOAsEq4o6bqMH0nJ4fJ+ScuJBgtWi3A2P2AOcp0wYRG6R9MsMccToSR XPGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sq3juwAyx/kGuC+9AEnQhgmCbLKhgaGhWILandlCIyQ=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=zReqa/fJfnR9zgmMvGlB0LP5ByjdPQSknuyYB0J2wMBDbOFgqVMCZWxGgGGUathmYc 8/H0l2PpYZcGtL9UXNKi+YOmVvy53sIcXveAjzJkvVLA4WvCBbQs/Ol6UpqQiOqXH+5r SdZI6/BVfPRbGYZoqmrr1C+MG7yF6e9Ye64bZVjafq1kjXRkqKm8t1pee2ykEflbsafT SVw9FUtqFIJ5c1mosmvMLCNKuPBbDZn7jVmO0aOuFRbI6Nj+IecbeEZgxNkzoAYmk9d+ RpBKFI6UZv6v1xQ3Jkit90mxU2w1be+xPHhT2I3c8XvEArIjCVhbPDBx4YRnxfuriMaL K7Ew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=lvm+93Zy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id le7-20020a170902fb0700b001bd949bcd75si3993069plb.575.2023.09.15.17.55.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 17:55:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=lvm+93Zy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id DB15E83A715D; Fri, 15 Sep 2023 11:38:27 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236715AbjIOSiL (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236515AbjIOShc (ORCPT ); Fri, 15 Sep 2023 14:37:32 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AD55115; Fri, 15 Sep 2023 11:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=sq3juwAyx/kGuC+9AEnQhgmCbLKhgaGhWILandlCIyQ=; b=lvm+93Zyyorgp8mPn2eE91Y0ck yXGQm7QiGVOUuyMWNgjTS4pNocHj7b8QyPTXC+m+yEougBY7HK7uxFHF4R9PMWFw5lNWvO5FMUXSH KIzHlaINvKtugaNxW5bmu6PFryvnw6AuavKx7Gy7vVWCxhZcRsnwoACLM/O8Wb9L980gJmemiBQa2 RYXD+V0C8B8y522E/iE9I3d+7qlSebZrsVo6lAVMgf7AHAKWHH9+etiPvc2BBDqeSvQw6x3P1E2Cn FxdZGaWZEZikCcoccFyIlsW8Hn04rypDvS97Z3PmKp904uOpHsAYHM9FqZxLoe6HM0XVRthvC8PXs PLyYU16Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIW-8u; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 03/17] mm: Add folio_end_read() Date: Fri, 15 Sep 2023 19:36:53 +0100 Message-Id: <20230915183707.2707298-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:27 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777153596357277711 X-GMAIL-MSGID: 1777153596357277711 Provide a function for filesystems to call when they have finished reading an entire folio. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 1 + mm/filemap.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 351c3b7f93a1..5bb2f5f802bc 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1129,6 +1129,7 @@ static inline void wait_on_page_locked(struct page *page) folio_wait_locked(page_folio(page)); } +void folio_end_read(struct folio *folio, bool success); void wait_on_page_writeback(struct page *page); void folio_wait_writeback(struct folio *folio); int folio_wait_writeback_killable(struct folio *folio); diff --git a/mm/filemap.c b/mm/filemap.c index 582f5317ff71..e3843cf1d8e1 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1527,6 +1527,28 @@ void folio_unlock(struct folio *folio) } EXPORT_SYMBOL(folio_unlock); +/** + * folio_end_read - End read on a folio. + * @folio: The folio. + * @success: True if all reads completed successfully. + * + * When all reads against a folio have completed, filesystems should + * call this function to let the pagecache know that no more reads + * are outstanding. This will unlock the folio and wake up any thread + * sleeping on the lock. The folio will also be marked uptodate if all + * reads succeeded. + * + * Context: May be called from interrupt or process context. May not be + * called from NMI context. + */ +void folio_end_read(struct folio *folio, bool success) +{ + if (success) + folio_mark_uptodate(folio); + folio_unlock(folio); +} +EXPORT_SYMBOL(folio_end_read); + /** * folio_end_private_2 - Clear PG_private_2 and wake any waiters. * @folio: The folio. From patchwork Fri Sep 15 18:36:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140656 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1287365vqi; Fri, 15 Sep 2023 13:00:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEga+zJBzcueq9lROdPZ+mvG4F5yRPnHA6C9kdS7AUBjf3Q+yDLlrFnF2HJb3a4xPruZKRg X-Received: by 2002:a05:6a21:338f:b0:152:1ce8:ce3a with SMTP id yy15-20020a056a21338f00b001521ce8ce3amr3292883pzb.18.1694808036862; Fri, 15 Sep 2023 13:00:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694808036; cv=none; d=google.com; s=arc-20160816; b=WYwjs9DwsKMCzHri1Z6BabZ1W3Cnp927rf0eqhWZVArNfLBYNSI08o9+eAxX8ksZOI mk+/y7Buu8oMRpWAY8EYdsTZy7tzHrO+lauKM+QTJ2eKz4ByC8swGI3aHD1yYQ6NfWFk mVx4ciDH3SgbZbOS6AjmlkO/st4ifv1u8T0XDnCN2BCna36h+kCb6Q6bR4QJvV4lTYIX jlwt646WR1sQyKXTJ3fIzpZrYybgYFv9sa2FHY4jmST2cbQtWnpMPOP5PbKTVy1y+aTD vMdnwJaEZ5V78is2Gsyijo5Q3+NXXyiCXTmnxqqV1loE2Fy1kxut5nDeE53qCOCEFX+V clTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZWM/c0Bk7RP+LkIKV16/78R+lD44jh4KeYNE9FiQZ2g=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=nL39R96abuVhbufUwPoI0HhVDHfpyfyZtsVfPINgdFIviYMF+aY26qSfT1OZMaJHSd aPibx61zXKykH/wXHoXqMW/3YJ4k40sYDWBUOTIzPSNjSKc6At5UqR0Ww0ej+x62M5Pv ppLw3qk2VB5YD81xB2peWfae1C8JJBRZ/j4y9uKwgyu0UTT2WeCpVLj0WEAiWesz2rQv uTjNVU4tlam+IKgAzTcTCqhUurCpvFHSl7iPfjljwcNGpayvRvWNtmmf/JsBnnJ2IEZc Y1iAjize89zhhP9ee9kHGvm6Nnv+/Ie/VFKOteOM+kX3J9a5CIQYFO3uM4NPlaa0b/WO IcUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=vfD+dPOF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id f20-20020a63f114000000b00565ee7952c4si3648883pgi.633.2023.09.15.13.00.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 13:00:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=vfD+dPOF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 2022183EDD9D; Fri, 15 Sep 2023 11:38:24 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236564AbjIOSht (ORCPT + 30 others); Fri, 15 Sep 2023 14:37:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236580AbjIOShV (ORCPT ); Fri, 15 Sep 2023 14:37:21 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 856801B8; Fri, 15 Sep 2023 11:37:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ZWM/c0Bk7RP+LkIKV16/78R+lD44jh4KeYNE9FiQZ2g=; b=vfD+dPOFEH/3fRQTfJ4YKiX6xf nN3FMpetxufZdOK9JVAoHYl43TAVYv16OvYJJ+p2BNrZKbtEDyS1FjbE5W+P2DrXrr4dQiy3DiMNh l1UzJlEIDljBct4aAOCJViqh2qq0eBg3eRFVUant/nrNbvn+9+dgQXAvITqHklzWqnjnI7TKy+1rl 5L76+77cjqW5RQLEO4z4mTnZgDKOhJTnmWu6WIim0WuwdNYiJuZ7b7/tqP1QsPtngqhnvYp/CMMVY 8ap7RlpBFzA7uw9m/78oZJMV07k9W2DugGnboHs+6XTsy2uiN/hUyGPsJRXFnJjuYENuCRc2DSNR1 JO6PJrYQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIY-D1; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 04/17] ext4: Use folio_end_read() Date: Fri, 15 Sep 2023 19:36:54 +0100 Message-Id: <20230915183707.2707298-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:24 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777135031705506341 X-GMAIL-MSGID: 1777135031705506341 folio_end_read() is the perfect fit for ext4. Signed-off-by: Matthew Wilcox (Oracle) --- fs/ext4/readpage.c | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index 3e7d160f543f..21e8f0aebb3c 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -70,15 +70,8 @@ static void __read_end_io(struct bio *bio) { struct folio_iter fi; - bio_for_each_folio_all(fi, bio) { - struct folio *folio = fi.folio; - - if (bio->bi_status) - folio_clear_uptodate(folio); - else - folio_mark_uptodate(folio); - folio_unlock(folio); - } + bio_for_each_folio_all(fi, bio) + folio_end_read(fi.folio, bio->bi_status == 0); if (bio->bi_private) mempool_free(bio->bi_private, bio_post_read_ctx_pool); bio_put(bio); @@ -336,8 +329,7 @@ int ext4_mpage_readpages(struct inode *inode, if (ext4_need_verity(inode, folio->index) && !fsverity_verify_folio(folio)) goto set_error_page; - folio_mark_uptodate(folio); - folio_unlock(folio); + folio_end_read(folio, true); continue; } } else if (fully_mapped) { From patchwork Fri Sep 15 18:36:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140932 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1472871vqi; Fri, 15 Sep 2023 21:31:27 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE2H722bbG9Oy4smLtv5ME84cmXens0pv7WwpLLKgjHavYtXe3HptuchayoMtZyCtPDoKVf X-Received: by 2002:a17:902:e5c2:b0:1bd:d911:2a85 with SMTP id u2-20020a170902e5c200b001bdd9112a85mr4552319plf.12.1694838686709; Fri, 15 Sep 2023 21:31:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694838686; cv=none; d=google.com; s=arc-20160816; b=Qfmq7J+E+25WT/X2DBwMBCRr9SDMNB460PPWH5Agm9jeNvXDlPIi3LmWEBJ4EJWu8k dwEduM9rJnoLEZqD7DOIpzBaKujIQ7AdUr0TcVc7M4vhp/wFpjrkGm0TsRa6iTfVkdWf Tf/hT5j7z/sbBqFcQorrrkT+rVrXHL78cCYRFCrksj9ujWIBB1Udro/hoCn1p4XCwo23 lTNEWCkVE2Afvd+ewLeQm1c2c++KhCYT9O6eXhSJV0OnvVFGuUcDX4ZLK1LS82yLIAxV GwV/z953YzgNCI6KM/sAYHWe3plnssK9wcCspyjF3gN0uCRcSt25Dh5cQWBj/s908f4k +SNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=wP0bf/c0k4lziBracMplPOL5jGzdkJZTznJybrN+G0o=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=fSb/QS5MtRravuWTDSL2n/7OzmWTCDsKws56reQLmTG5vzTQYzGQMKA/pbwuapa7zf 3FCL7BeeM51B0+M5N9h5oO6IpXBitPItAUYABjwlKCPt17+2O8wfk9smG+CbAUtUuBtd OxVKqEi71o07CS/uhNkufrvztPUxR9EAxqCyI2q08ILb1OSVpLd51KEUvku2h3vkojZM SUgKMWQRZdG/OF0FFChhbTS7FVKiK2WCdVr2yM+h3smEzFoeljInLp0+cNJAem86r1fg uKETzehK0mpBTRGMiRxJFrhy9Pl7c/fTtfTHNO5g84eZNOjIUc6Kr8RZN03sWTd32+8q TWxg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=nmNahHbU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [2620:137:e000::3:6]) by mx.google.com with ESMTPS id l20-20020a170903005400b001bf1d1d99besi4264852pla.423.2023.09.15.21.31.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 21:31:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=nmNahHbU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id A1FDB8414969; Fri, 15 Sep 2023 11:39:15 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236790AbjIOSiY (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236548AbjIOShs (ORCPT ); Fri, 15 Sep 2023 14:37:48 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 760DC1BDA; Fri, 15 Sep 2023 11:37:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wP0bf/c0k4lziBracMplPOL5jGzdkJZTznJybrN+G0o=; b=nmNahHbU8N6VQC3bwwkYcMi5SK 9z62LhVFB97uUetddKzSC5oxvrkgAa/37C9YLshZHa2WYGzbLW1TAnnZNykG8eoRg+AOQq8BqL20h TfICsXD5xZCp5JrkQ4Q1sz/lfXN+iKm2bI9bS0rzLlmR6H21VLEf8YRxZaKEZUmUCNVMKfXfEh5aO jns8hqlgjEOLgpJQUGWwTWmGwpgmYz8XJ09lPpNE96iue3gnjaEUr3FLrhGJIr5rUkDm6pkOnyjLu eKXEzJKBOlS5VYgsfvPQtgf/zPmG3M/q0F4HUD9j5wJj9j7ISu1xfFpZQYsQ8GjQvM65661Wvg30j F47ncT9Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMId-FC; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 05/17] buffer: Use folio_end_read() Date: Fri, 15 Sep 2023 19:36:55 +0100 Message-Id: <20230915183707.2707298-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:39:15 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777167170495353738 X-GMAIL-MSGID: 1777167170495353738 There are two places that we can use this new helper. Signed-off-by: Matthew Wilcox (Oracle) --- fs/buffer.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 2379564e5aea..345a9a42610f 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -282,13 +282,7 @@ static void end_buffer_async_read(struct buffer_head *bh, int uptodate) } while (tmp != bh); spin_unlock_irqrestore(&first->b_uptodate_lock, flags); - /* - * If all of the buffers are uptodate then we can set the page - * uptodate. - */ - if (folio_uptodate) - folio_mark_uptodate(folio); - folio_unlock(folio); + folio_end_read(folio, folio_uptodate); return; still_busy: @@ -2413,12 +2407,10 @@ int block_read_full_folio(struct folio *folio, get_block_t *get_block) if (!nr) { /* - * All buffers are uptodate - we can set the folio uptodate - * as well. But not if get_block() returned an error. + * All buffers are uptodate or get_block() returned an + * error when trying to map them - we can finish the read. */ - if (!page_error) - folio_mark_uptodate(folio); - folio_unlock(folio); + folio_end_read(folio, !page_error); return 0; } From patchwork Fri Sep 15 18:36:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140784 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1390807vqi; Fri, 15 Sep 2023 17:02:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFZiDgKiPvUWBQRvITDLd2aLYraYa5Frk2PVOQF7++8v3B217ITPqE3Vhrw+0IqBGBL2AdR X-Received: by 2002:a05:6a00:cd1:b0:68a:4261:ab7f with SMTP id b17-20020a056a000cd100b0068a4261ab7fmr3528303pfv.31.1694822520109; Fri, 15 Sep 2023 17:02:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694822520; cv=none; d=google.com; s=arc-20160816; b=WBJY9EPMETz3FzKgCfPg5/daCcHdtReSyBx6RRaDQnNQoZgqiPIsRDoUR5EyEcRcNj t4pIxwxYoGiRr2EWpqmwe3uGGwemrZhjA89E5mVXRl7cDS8OSo8i/lh2gyQUEiiD/EIz SEoz7iPCJXKsDhJ6qEbg/41MxsM9OOP8HuWQ0R8G5HEltjCQ6sfjpLp4oGP6qicHt8qc teVXUNo0Ugi3NrIofO7foFB6E4OBRJWIrDZcOjGKwCqt9eRD4fmYQBk9phlmOI6mVtay Yugbq8cdxfwogUxXS+VTXo5WT8H00uPW2L+M/ih8zcP5xZn/7GPZQYC6hB7G8QYzhs0p qVcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aanqWz7LXsKQhB1oKoWrq/hhDEToXtz4R83zwTN5ek4=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=WoEcIy1cCi2rXayFzOHJ1C8gcSg9RQnLVFyyURYxVIXwDgYQOvFuEIKQro2zdkeek8 dfzvpXYo57ItiO/R7YuJs3DIgggwbCb0PRNfbB12S2G0BdDzPpxIcHqqknTxj3T++5XR F8PGZxbLrjDqZAtpCCbRYYh0LmlnXokBQnVpfbYn2bi/Jmqffq+9OMrZMThyL3sHjDft XE9wqeWCyJ4IQleBuq+UVxNADFTkKYJqMcvyRAwviy8YqiJ4TXfgBoQcKcUW9VAlJhW3 9wtNYfrK7ZKH9G/xV2/wLlUYIWJ5sFP0xrPCOK0ownGG77wiclVgGI26pvemOnb3BFtn rjYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=eFAUDvC1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id l21-20020a656815000000b00563fc0d174csi3880070pgt.791.2023.09.15.17.01.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 17:02:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=eFAUDvC1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 823F9838C17C; Fri, 15 Sep 2023 11:38:24 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236625AbjIOShx (ORCPT + 30 others); Fri, 15 Sep 2023 14:37:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236464AbjIOShW (ORCPT ); Fri, 15 Sep 2023 14:37:22 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8288010D9; Fri, 15 Sep 2023 11:37:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=aanqWz7LXsKQhB1oKoWrq/hhDEToXtz4R83zwTN5ek4=; b=eFAUDvC1j+eUES+EbIT45gn1R1 ++XVlsh8I4oCrZdiCg6UnYc8aXWJEyHdUZQIC4cvEomnlBym8xe2PELLXDw/GqKgpT01e0JbE/VrW XHeSvudWZJ81qKFqooD32XIzGJsN00TKCoOlRmGvKrDKb31R3frDJY0JM5CS0AVOPhI/BwNftt6yQ xafCSe2PErcOIYuJ1W09lM5nqBETq5QoJRNHE9qNtIKOBvHrg/NwGWvndtPBaqHXMOiMCCAJqCC5P j3JNlJZnH0fLf7twNFL5cWWTzaY5KIF1HaGYRjeDBwPu8c3Q5cviTs9Sxn+VH6fP/S3xKFSNZD52s 8Is8ufIQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIf-HO; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 06/17] iomap: Use folio_end_read() Date: Fri, 15 Sep 2023 19:36:56 +0100 Message-Id: <20230915183707.2707298-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:24 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777150218879708842 X-GMAIL-MSGID: 1777150218879708842 Combine the setting of the uptodate flag with the clearing of the locked flag. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index cade15b70627..e59831bd9217 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -274,10 +274,8 @@ static void iomap_finish_folio_read(struct folio *folio, size_t off, spin_unlock_irqrestore(&ifs->state_lock, flags); } - if (uptodate) - folio_mark_uptodate(folio); if (finished) - folio_unlock(folio); + folio_end_read(folio, uptodate); } static void iomap_read_end_io(struct bio *bio) From patchwork Fri Sep 15 18:36:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140726 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1350137vqi; Fri, 15 Sep 2023 15:15:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEjWkizIy7RV/CoOguh/S38ib+EuqYiddRZMVDa1jFxXPkfzYK/QghYWnY6L49A5m7mdGG3 X-Received: by 2002:a17:90a:2ec3:b0:268:29cf:3231 with SMTP id h3-20020a17090a2ec300b0026829cf3231mr2727300pjs.3.1694816125988; Fri, 15 Sep 2023 15:15:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694816125; cv=none; d=google.com; s=arc-20160816; b=X40DMcYRPYEPlVsj1CGaFcY20ou+mn/axODhPLDBptYtIkJHIAioTh3RfndHV1EOkF DMmjQAXLfGPkAwQaSWC/Lf6Zj9l9ilUleh/IyxROvgfkIIYLJmIOlh93/6BRdobN2F2F /9CX9L5ak+DrHAf9SCnqv8l8MfvbFWAEWRfLUHh8T87OKD6j6uuoQMVme0DAUqnPU+jb hw2y/aF4u5Wlda6fQUuU+IBzDbJJ/9qfdAOYkROKkw9153cr+8Zlsns5iLY/EDStD9sQ ITi8t0Rrhn28P/rDuiUft2nRnOHlBtPUM87tMFV9Mnajv1KLrc7SM0kpig47J666cyu1 /d6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=0N1cmjHcEJag2AtHK/yNA0zjVT3A3o3O6S0qHTVMYWc=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=Y3VkdlFkuYmus98C8NA6RooYjwv/glTb7xsUodJZEfV2Ebu3I+V8gCNKayLkV8OZkz nhn4Je6zZlRNtXJ8h4DUvUoQgpVqApRtSt224TwW7sAMICo5lb06kXoQjFgaB9/srZVg JPm779GuiLsMU6Q+aYXkU1+CyiKBJUuW8f/aF6fg0AjqEyxkZR1jRY6eJgnFtdf0bq8C /db89NLlTuVIDC/tsAm53zUtrmAG1JnsaUA5GXVdLb2/Jw5mhsCL2SLZOkxTGOs4qXQ4 qoy9oteXVAAsOD2DBXOdaQ5fJR2Nn5gt/DcCFeh8qxFPaScP7u2vA4Iv5R4VYnBxkUEk 6qsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=LKxDO16w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from howler.vger.email (howler.vger.email. [23.128.96.34]) by mx.google.com with ESMTPS id x6-20020a17090ab00600b002634977e6e5si6067278pjq.142.2023.09.15.15.15.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 15:15:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=LKxDO16w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 7F68F8286E84; Fri, 15 Sep 2023 11:38:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236734AbjIOSiO (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236495AbjIOShj (ORCPT ); Fri, 15 Sep 2023 14:37:39 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97E87115; Fri, 15 Sep 2023 11:37:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=0N1cmjHcEJag2AtHK/yNA0zjVT3A3o3O6S0qHTVMYWc=; b=LKxDO16wvao6WY4341Zd7gqp8a 6tJ5yZe5JeqT3FGPaCbZo8YtdIVWyxnH8Ht3EXolZ98oXnKIy8ke2Jfebv75l2ca9l/SXyijqolLI GnypR8ReWzvNkFGWZEC1Y4LqTpv+aM6shDYXLlkz3SWyFjtbgxWn4D7ehYPC+Nf1p8sJycvegvvrp 6zNq6PsIlrqyya0zVORpXsfPamwV6SJNBOHUfreeJl7/dwEoZldZ5dEqNdvreqC3D19HwO2znGurS Te0aY/RPjQbzJatIzYB8JOZl88C4ZHTSQlT8KVb1JPC0gFMVItO7W8o6S8JlxCy182jcIHdLjFIl+ uDseTJ7w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIh-KO; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 07/17] bitops: Add xor_unlock_is_negative_byte() Date: Fri, 15 Sep 2023 19:36:57 +0100 Message-Id: <20230915183707.2707298-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:28 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777143514499181860 X-GMAIL-MSGID: 1777143514499181860 Replace clear_bit_and_unlock_is_negative_byte() with xor_unlock_is_negative_byte(). We have a few places that like to lock a folio, set a flag and unlock it again. Allow for the possibility of combining the latter two operations for efficiency. We are guaranteed that the caller holds the lock, so it is safe to unlock it with the xor. The caller must guarantee that nobody else will set the flag without holding the lock; it is not safe to do this with the PG_dirty flag, for example. Signed-off-by: Matthew Wilcox (Oracle) --- arch/powerpc/include/asm/bitops.h | 17 ++++-------- arch/x86/include/asm/bitops.h | 11 ++++---- .../asm-generic/bitops/instrumented-lock.h | 27 ++++++++++--------- include/asm-generic/bitops/lock.h | 21 ++++----------- kernel/kcsan/kcsan_test.c | 8 +++--- kernel/kcsan/selftest.c | 8 +++--- mm/filemap.c | 5 ++++ mm/kasan/kasan_test.c | 7 ++--- 8 files changed, 47 insertions(+), 57 deletions(-) diff --git a/arch/powerpc/include/asm/bitops.h b/arch/powerpc/include/asm/bitops.h index 7e0f0322912b..40cc3ded60cb 100644 --- a/arch/powerpc/include/asm/bitops.h +++ b/arch/powerpc/include/asm/bitops.h @@ -234,32 +234,25 @@ static inline int arch_test_and_change_bit(unsigned long nr, } #ifdef CONFIG_PPC64 -static inline unsigned long -clear_bit_unlock_return_word(int nr, volatile unsigned long *addr) +static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *p) { unsigned long old, t; - unsigned long *p = (unsigned long *)addr + BIT_WORD(nr); - unsigned long mask = BIT_MASK(nr); __asm__ __volatile__ ( PPC_RELEASE_BARRIER "1:" PPC_LLARX "%0,0,%3,0\n" - "andc %1,%0,%2\n" + "xor %1,%0,%2\n" PPC_STLCX "%1,0,%3\n" "bne- 1b\n" : "=&r" (old), "=&r" (t) : "r" (mask), "r" (p) : "cc", "memory"); - return old; + return (old & BIT_MASK(7)) != 0; } -/* - * This is a special function for mm/filemap.c - * Bit 7 corresponds to PG_waiters. - */ -#define arch_clear_bit_unlock_is_negative_byte(nr, addr) \ - (clear_bit_unlock_return_word(nr, addr) & BIT_MASK(7)) +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte #endif /* CONFIG_PPC64 */ diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index 2edf68475fec..f03c0a50ec3a 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -94,18 +94,17 @@ arch___clear_bit(unsigned long nr, volatile unsigned long *addr) asm volatile(__ASM_SIZE(btr) " %1,%0" : : ADDR, "Ir" (nr) : "memory"); } -static __always_inline bool -arch_clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) +static __always_inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *addr) { bool negative; - asm volatile(LOCK_PREFIX "andb %2,%1" + asm volatile(LOCK_PREFIX "xorb %2,%1" CC_SET(s) : CC_OUT(s) (negative), WBYTE_ADDR(addr) - : "ir" ((char) ~(1 << nr)) : "memory"); + : "iq" ((char)mask) : "memory"); return negative; } -#define arch_clear_bit_unlock_is_negative_byte \ - arch_clear_bit_unlock_is_negative_byte +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte static __always_inline void arch___clear_bit_unlock(long nr, volatile unsigned long *addr) diff --git a/include/asm-generic/bitops/instrumented-lock.h b/include/asm-generic/bitops/instrumented-lock.h index eb64bd4f11f3..e8ea3aeda9a9 100644 --- a/include/asm-generic/bitops/instrumented-lock.h +++ b/include/asm-generic/bitops/instrumented-lock.h @@ -58,27 +58,30 @@ static inline bool test_and_set_bit_lock(long nr, volatile unsigned long *addr) return arch_test_and_set_bit_lock(nr, addr); } -#if defined(arch_clear_bit_unlock_is_negative_byte) +#if defined(arch_xor_unlock_is_negative_byte) /** - * clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom - * byte is negative, for unlock. - * @nr: the bit to clear - * @addr: the address to start counting from + * xor_unlock_is_negative_byte - XOR a single byte in memory and test if + * it is negative, for unlock. + * @mask: Change the bits which are set in this mask. + * @addr: The address of the word containing the byte to change. * + * Changes some of bits 0-6 in the word pointed to by @addr. * This operation is atomic and provides release barrier semantics. + * Used to optimise some folio operations which are commonly paired + * with an unlock or end of writeback. Bit 7 is used as PG_waiters to + * indicate whether anybody is waiting for the unlock. * - * This is a bit of a one-trick-pony for the filemap code, which clears - * PG_locked and tests PG_waiters, + * Return: Whether the top bit of the byte is set. */ -static inline bool -clear_bit_unlock_is_negative_byte(long nr, volatile unsigned long *addr) +static inline bool xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *addr) { kcsan_release(); - instrument_atomic_write(addr + BIT_WORD(nr), sizeof(long)); - return arch_clear_bit_unlock_is_negative_byte(nr, addr); + instrument_atomic_write(addr, sizeof(long)); + return arch_xor_unlock_is_negative_byte(mask, addr); } /* Let everybody know we have it. */ -#define clear_bit_unlock_is_negative_byte clear_bit_unlock_is_negative_byte +#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte #endif #endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_LOCK_H */ diff --git a/include/asm-generic/bitops/lock.h b/include/asm-generic/bitops/lock.h index 40913516e654..6a638e89d130 100644 --- a/include/asm-generic/bitops/lock.h +++ b/include/asm-generic/bitops/lock.h @@ -66,27 +66,16 @@ arch___clear_bit_unlock(unsigned int nr, volatile unsigned long *p) raw_atomic_long_set_release((atomic_long_t *)p, old); } -/** - * arch_clear_bit_unlock_is_negative_byte - Clear a bit in memory and test if bottom - * byte is negative, for unlock. - * @nr: the bit to clear - * @addr: the address to start counting from - * - * This is a bit of a one-trick-pony for the filemap code, which clears - * PG_locked and tests PG_waiters, - */ -#ifndef arch_clear_bit_unlock_is_negative_byte -static inline bool arch_clear_bit_unlock_is_negative_byte(unsigned int nr, - volatile unsigned long *p) +#ifndef arch_xor_unlock_is_negative_byte +static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *p) { long old; - unsigned long mask = BIT_MASK(nr); - p += BIT_WORD(nr); - old = raw_atomic_long_fetch_andnot_release(mask, (atomic_long_t *)p); + old = raw_atomic_long_fetch_xor_release(mask, (atomic_long_t *)p); return !!(old & BIT(7)); } -#define arch_clear_bit_unlock_is_negative_byte arch_clear_bit_unlock_is_negative_byte +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte #endif #include diff --git a/kernel/kcsan/kcsan_test.c b/kernel/kcsan/kcsan_test.c index 0ddbdab5903d..1333d23ac4ef 100644 --- a/kernel/kcsan/kcsan_test.c +++ b/kernel/kcsan/kcsan_test.c @@ -700,10 +700,10 @@ static void test_barrier_nothreads(struct kunit *test) KCSAN_EXPECT_RW_BARRIER(mutex_lock(&test_mutex), false); KCSAN_EXPECT_RW_BARRIER(mutex_unlock(&test_mutex), true); -#ifdef clear_bit_unlock_is_negative_byte - KCSAN_EXPECT_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true); - KCSAN_EXPECT_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true); - KCSAN_EXPECT_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var), true); +#ifdef xor_unlock_is_negative_byte + KCSAN_EXPECT_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); + KCSAN_EXPECT_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); + KCSAN_EXPECT_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); #endif kcsan_nestable_atomic_end(); } diff --git a/kernel/kcsan/selftest.c b/kernel/kcsan/selftest.c index 8679322450f2..619be7417420 100644 --- a/kernel/kcsan/selftest.c +++ b/kernel/kcsan/selftest.c @@ -228,10 +228,10 @@ static bool __init test_barrier(void) spin_lock(&test_spinlock); KCSAN_CHECK_RW_BARRIER(spin_unlock(&test_spinlock)); -#ifdef clear_bit_unlock_is_negative_byte - KCSAN_CHECK_RW_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var)); - KCSAN_CHECK_READ_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var)); - KCSAN_CHECK_WRITE_BARRIER(clear_bit_unlock_is_negative_byte(0, &test_var)); +#ifdef xor_unlock_is_negative_byte + KCSAN_CHECK_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); + KCSAN_CHECK_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); + KCSAN_CHECK_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); #endif kcsan_nestable_atomic_end(); diff --git a/mm/filemap.c b/mm/filemap.c index e3843cf1d8e1..2db7c5da9e03 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1484,6 +1484,11 @@ void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter) } EXPORT_SYMBOL_GPL(folio_add_wait_queue); +#ifdef xor_unlock_is_negative_byte +#define clear_bit_unlock_is_negative_byte(nr, p) \ + xor_unlock_is_negative_byte(1 << nr, p) +#endif + #ifndef clear_bit_unlock_is_negative_byte /* diff --git a/mm/kasan/kasan_test.c b/mm/kasan/kasan_test.c index b61cc6a42541..c1de091dfc92 100644 --- a/mm/kasan/kasan_test.c +++ b/mm/kasan/kasan_test.c @@ -1098,9 +1098,10 @@ static void kasan_bitops_test_and_modify(struct kunit *test, int nr, void *addr) KUNIT_EXPECT_KASAN_FAIL(test, __test_and_change_bit(nr, addr)); KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = test_bit(nr, addr)); -#if defined(clear_bit_unlock_is_negative_byte) - KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = - clear_bit_unlock_is_negative_byte(nr, addr)); +#if defined(xor_unlock_is_negative_byte) + if (nr < 7) + KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = + xor_unlock_is_negative_byte(1 << nr, addr)); #endif } From patchwork Fri Sep 15 18:36:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140847 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1411769vqi; Fri, 15 Sep 2023 18:01:39 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGAslPxnEwINEQLHohzj6mJkdwtAP2rKThMLEGw9nZEM9xEqDlZJkbRZy49zlTbJ2auYc4M X-Received: by 2002:a05:6a20:7f94:b0:138:60e:9c4 with SMTP id d20-20020a056a207f9400b00138060e09c4mr4304341pzj.23.1694826098987; Fri, 15 Sep 2023 18:01:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694826098; cv=none; d=google.com; s=arc-20160816; b=lWQIjXQDUrR/AoGz5S/i8Xsku6kNVxEJazJ79Ik2J3Wp709gn1BvcU7bA6wBEbHu3H b190Hk0N9+shww2us1HgPIZ3/bFaXuU6lJt8G4lm2wn68izYVf3Ft6eV66Fb6iSEB/XM vrlrdagOt2dWWvyoRlHu/4tjKa5/mC6OxR7PsIPcF+Kpt/XymnykCYML9m9jSb6V0wg5 gZIhbTENFhhrU+X843UDohzgQ7fY7HpjQECNZRty3wBqSkUmNVzxFhLr7eFHpaUfrgpl O05Kpkq3Ls2GcHkcISaYsC0cH67t0BnmnzjrLn5f0fqcWO1guqwfCwdG0kvklWe2SNPz rMjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6EBV41RIEaRQJ/bvvEj83J2s787qIb+bZO0o/tTL46o=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=MDMqxmYnFMwT6NVVNF3tUvYIA8XZGJdDUweBv5cJ5sBHPdmNQ+elGu/OzII5bfW0Lu vIf7HDMCqUMEJU4lONkIg5cOFdpZbKJEOjsvFivYkHTAFi7O8sG5InK033pyrOUmAhmT ec2AHxewxePH1aVwCa0/695UBSKxBU4bAbXnhn1v6aq2n1O4WEKanLzt6QfV3Ddn7GMz 2JT/OzsGfd9uJ8vO55k6SF4PKuyoo9JDZAEwbNmKqtKSIEn1H2GOmMUxOpnp/H0a77hj NHSganJky8v9huKrwoBT+KXA2wEwEY57vszUcLy72I1N/dptcDMF+TYA4QIEj0oHmN9O dfyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=PGQxstVh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id z12-20020a63190c000000b00566022a3894si4061646pgl.415.2023.09.15.18.01.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 18:01:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=PGQxstVh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 8E7E981D82DF; Fri, 15 Sep 2023 11:38:55 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236758AbjIOSiQ (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236511AbjIOSho (ORCPT ); Fri, 15 Sep 2023 14:37:44 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83E7D196; Fri, 15 Sep 2023 11:37:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=6EBV41RIEaRQJ/bvvEj83J2s787qIb+bZO0o/tTL46o=; b=PGQxstVhyCK0bbn5NHcYRNn1ww TxkyLFF82AJXaUuPCP6DmEIuZbEa0EI2ja1MyNLl3TdgTTfLGsIjYXP4J5tsPhGSDcF4Lowtd/gxr WmuuHVRLsydmaX8bgQiQF/TghtX7r/DGahRcDs+Z8x33x9u9jN+r0263r9IdacDiIbu5vI+zvQue/ CUN2QU8Lx2GRWbdVVFYgK0NfRCOq1DP4DgN1U15kTUtq3xrZm+drFqyseUENixVGv3Jf5rprOB25c 1fBweM05s0kmBDCS73000UmQyQJG9ZJmwSibr6lcT6hX4gnU+xO7zy4dEck0fA2C6m5EnJs/hCw5q ZJpNOApA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIj-Ml; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 08/17] alpha: Implement xor_unlock_is_negative_byte Date: Fri, 15 Sep 2023 19:36:58 +0100 Message-Id: <20230915183707.2707298-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:55 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777153971694091827 X-GMAIL-MSGID: 1777153971694091827 Inspired by the alpha clear_bit() and arch_atomic_add_return(), this will surely be more efficient than the generic one defined in filemap.c. Signed-off-by: Matthew Wilcox (Oracle) --- arch/alpha/include/asm/bitops.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/alpha/include/asm/bitops.h b/arch/alpha/include/asm/bitops.h index bafb1c1f0fdc..138930e8a504 100644 --- a/arch/alpha/include/asm/bitops.h +++ b/arch/alpha/include/asm/bitops.h @@ -286,6 +286,27 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) #define arch_test_bit generic_test_bit #define arch_test_bit_acquire generic_test_bit_acquire +static inline bool xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *p) +{ + unsigned long temp, old; + + __asm__ __volatile__( + "1: ldl_l %0,%4\n" + " xor %0,%3,%0\n" + " xor %0,%3,%2\n" + " stl_c %0,%1\n" + " beq %0,2f\n" + ".subsection 2\n" + "2: br 1b\n" + ".previous" + :"=&r" (temp), "=m" (*p), "=&r" (old) + :"Ir" (mask), "m" (*p)); + + return (old & BIT(7)) != 0; +} +#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte + /* * ffz = Find First Zero in word. Undefined if no zero exists, * so code should check against ~0UL first.. From patchwork Fri Sep 15 18:36:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140688 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1313239vqi; Fri, 15 Sep 2023 13:54:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFNAOh2tf4fBty0p0bGFlwLxgwh6M8HZ7ADHcib5SHU1hXiV/0sXUiePgrsmL1OZkMNjftj X-Received: by 2002:a05:6a20:2451:b0:158:1ade:8661 with SMTP id t17-20020a056a20245100b001581ade8661mr3243322pzc.18.1694811263091; Fri, 15 Sep 2023 13:54:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694811263; cv=none; d=google.com; s=arc-20160816; b=KGfb3lUke0TdQtQHCRIukhQ8xI1ONWorI7/T3j1SmwaykdfcZXj/UGpbMpOOCi+Ldn KLP2VAdKfs0MmIWewxfke8Sk/cgu6vTZh2ShtmpULs9WrkLtjKu97zJZzUtkbMsMw35C BRhPgMZRr3Sn38TK09N717zJDjAD7TU2BYImw2sNr7FZ9D6BbJFWY1T2Bhb/hSUSVe5H 8Rf9UOKxL0cH9mXmJZxCt97Nm9FlWj8NAC51tC+xr4Ka1FEXWwErMv37C6WkifwPae75 VXBRaZOrSF91rsE87tlYBWPOBsMucwIDt5U0MEal1p6HbGg8HlnolPmtIg0QlkYHyBKU 4TxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=97eXyyMqb/hrrtcwsA0mCTzMLmtahF6NIf6z9CRLAho=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=tTa8YRH9vzAh+Zr++hpNtoTfvtU9W3RKjzPfTez1USjtuOk97g9eVQ5dP6OAU6q9tR vlztSWbagisnIJ7bl4wmL9GUejr6cFEFZjjABUwETV6ZgcGMzS0zgTpYTqjQaFq8e3Cr rFaLF9lwFmdjDiSLh+MRGlznm0aiPYTIBxrGJ7NyeJltPG6aH9brmP4ANjL0L4s+AZD1 nk4fsdP7Y1qBfVR/TFNgSNouiUzVtcnoNGqYmeCvDbnbbVOgw+DhtlmdhF+ovjG/sSPI 5U7aZkaAeIMg+pqyEn5Hgv3mOijvBqIo6ILoZiKDkOtleFmCAVUbg8UWuVqyiSDFxIz8 QR6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=o0ZS3c1w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id u190-20020a6385c7000000b0053fb354c194si3860306pgd.659.2023.09.15.13.54.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 13:54:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=o0ZS3c1w; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8E44B83A7163; Fri, 15 Sep 2023 11:38:28 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236748AbjIOSiP (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236503AbjIOShl (ORCPT ); Fri, 15 Sep 2023 14:37:41 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 848F7115; Fri, 15 Sep 2023 11:37:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=97eXyyMqb/hrrtcwsA0mCTzMLmtahF6NIf6z9CRLAho=; b=o0ZS3c1w/qdRz22hiED4PYe5tH miH5vzoyL9HJMWsg73IzM2sQlI9eRgqxlhle23BnFEWfuUvVKxVfb0juP2UbHFtU3lsXCF32GZiqy U075r64rz/3yomqiObJyF7Q/kNyEc8ElTmfdYe4hPTNIuG0B9KaT2nqblIOGdEdd03nY0QTzNCYnO gCDiQmlntbQH3bdHuIKfGPTjxuiYZoJ40KIoWEePsK+yRkAaclF2alCfkSKQJ4ZrSOzSAMwo1B+lm yqjMxnAAH0YnL2QLrEeCqDEbQnq46m14kA/+lwymyAd9aX/hRu69HWQAQMPoqo0SWQJMdqFFWzS2M epROG8Hw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIl-PE; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 09/17] m68k: Implement xor_unlock_is_negative_byte Date: Fri, 15 Sep 2023 19:36:59 +0100 Message-Id: <20230915183707.2707298-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:28 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777138415374460966 X-GMAIL-MSGID: 1777138415374460966 Using EOR to clear the guaranteed-to-be-set lock bit will test the negative flag just like the x86 implementation. This should be more efficient than the generic implementation in filemap.c. It would be better if m68k had __GCC_ASM_FLAG_OUTPUTS__. Signed-off-by: Matthew Wilcox (Oracle) --- arch/m68k/include/asm/bitops.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/arch/m68k/include/asm/bitops.h b/arch/m68k/include/asm/bitops.h index e984af71df6b..909ebe7cab5d 100644 --- a/arch/m68k/include/asm/bitops.h +++ b/arch/m68k/include/asm/bitops.h @@ -319,6 +319,20 @@ arch___test_and_change_bit(unsigned long nr, volatile unsigned long *addr) return test_and_change_bit(nr, addr); } +static inline bool xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *p) +{ + char result; + char *cp = (char *)p + 3; /* m68k is big-endian */ + + __asm__ __volatile__ ("eor.b %1, %2; smi %0" + : "=d" (result) + : "di" (mask), "o" (*cp) + : "memory"); + return result; +} +#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte + /* * The true 68020 and more advanced processors support the "bfffo" * instruction for finding bits. ColdFire and simple 68000 parts From patchwork Fri Sep 15 18:37:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140854 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1418499vqi; Fri, 15 Sep 2023 18:19:07 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGXxflod//ZwSSB+gmesl51OFaHMrcvHb6OcAw8WCSMpYfYALU8/YpZPKSzPo5ixC6NfZ+Z X-Received: by 2002:a05:6870:4252:b0:1d6:617d:62eb with SMTP id v18-20020a056870425200b001d6617d62ebmr3914341oac.39.1694827146985; Fri, 15 Sep 2023 18:19:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694827146; cv=none; d=google.com; s=arc-20160816; b=Bz2oSV0PzE93wj2rHttc9px99GonpSXKPu+B3/53OOsUQHVY5mH1gctVJG06WjoOvh DzQbvO4reUqFslR/DXhbmU5JFUSUrWtg+urqikBD+BKctVpnrPnMm99OFXnLA3iWmzx8 /M2vi7Hy/eUEhho+Vy/5tnJ6g0QuSGsuDTHm/UFcTVjEwfeVgGx6j1nfeLV9Tb9b8xH2 lFQ8csEomeqabr9cHiIo2B17ceXP+P0DMen2+nX9zE9su197VlHmCoLR/kHgt0OMCHZO Elz0zJblZEnF5CIBImW928sSTfDWDV/UXcalrvuWHPWZyCvmChNdc26mBd4F+YthjLPy I8fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vWwsy+BgXOEcfKuqkIEkXjDw+riS2szWYiR2hDnP9b4=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=CzMLseClqnIwK4JnqiP0e9ms1iKoioobvDCh3n7lQeI5vbb8fBeEgvwQzeLSIS9wMM FZv8TZllKqeoJQA9YA/xTPI5qyQChzkiOQOhjAmY6iDbSUmI1x17WGGieMZhUS8IFGQP pX4bbu56XHFlc3JQ2OngTok5Rp2XiuCILC+e6KEi2oKB6Lse6s96qtdVl89jGYdJW8lY LXj2iWEYvSf6SwaVQ/UmmrM1yVR7219ye6BIOUjl3AgM+CuU+iqr7yzUKpQsfOd5NpNd DP0Nx+QsXlFf4beZr5Fs7GcEq9MJs8j1bZFfaMIM3hQ4hln62/Mo4rNReVphhbikAZnO VO8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=iaE2slop; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id q30-20020a63f95e000000b00569362d6cebsi4052487pgk.2.2023.09.15.18.19.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 18:19:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=iaE2slop; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 0C457809B08A; Fri, 15 Sep 2023 11:39:25 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236694AbjIOSiG (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236596AbjIOShY (ORCPT ); Fri, 15 Sep 2023 14:37:24 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A2C6D196; Fri, 15 Sep 2023 11:37:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=vWwsy+BgXOEcfKuqkIEkXjDw+riS2szWYiR2hDnP9b4=; b=iaE2slop4N4xLJSD+OgbNbmYDB gvFtf3rpiE7MHfx1VcIaaTK6mqcWPzGB45tY/We9XP2f0KJYV6gGKzvmGnaGt2e4dMZjvKt6U+qoZ WsFrvPotolFPbFSAus8TnTLqvsp79DzGWxNd6OPiUC0zBFJ8QqtYxPBpgfsmB7lLCeWKcELGjqU3R SI2HI870LgmTnj0NvcJMVY/Ff+qZeSlET7hIisVQBmwZejpdiwpNCHb2wohJKyFAUerGvW1W2rtzt QQhq6L1YGWcKYaWuTaIaXH7nHZkN9TozL0NTGQj9pBBGGCg5ZFFmlcZipxMpNs2sZyXo59jSYV1fQ U35De4qQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIn-Rw; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 10/17] mips: Implement xor_unlock_is_negative_byte Date: Fri, 15 Sep 2023 19:37:00 +0100 Message-Id: <20230915183707.2707298-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:39:25 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777155070765077269 X-GMAIL-MSGID: 1777155070765077269 Inspired by the mips test_and_change_bit(), this will surely be more efficient than the generic one defined in filemap.c Signed-off-by: Matthew Wilcox (Oracle) --- arch/mips/include/asm/bitops.h | 26 +++++++++++++++++++++++++- arch/mips/lib/bitops.c | 14 ++++++++++++++ 2 files changed, 39 insertions(+), 1 deletion(-) diff --git a/arch/mips/include/asm/bitops.h b/arch/mips/include/asm/bitops.h index b4bf754f7db3..d98a05c478f4 100644 --- a/arch/mips/include/asm/bitops.h +++ b/arch/mips/include/asm/bitops.h @@ -73,7 +73,8 @@ int __mips_test_and_clear_bit(unsigned long nr, volatile unsigned long *addr); int __mips_test_and_change_bit(unsigned long nr, volatile unsigned long *addr); - +bool __mips_xor_is_negative_byte(unsigned long mask, + volatile unsigned long *addr); /* * set_bit - Atomically set a bit in memory @@ -279,6 +280,29 @@ static inline int test_and_change_bit(unsigned long nr, return res; } +static inline bool xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *p) +{ + unsigned long orig; + bool res; + + smp_mb__before_atomic(); + + if (!kernel_uses_llsc) { + res = __mips_xor_is_negative_byte(mask, p); + } else { + orig = __test_bit_op(*p, "%0", + "xor\t%1, %0, %3", + "ir"(mask)); + res = (orig & BIT(7)) != 0; + } + + smp_llsc_mb(); + + return res; +} +#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte + #undef __bit_op #undef __test_bit_op diff --git a/arch/mips/lib/bitops.c b/arch/mips/lib/bitops.c index 116d0bd8b2ae..00aee98e9d54 100644 --- a/arch/mips/lib/bitops.c +++ b/arch/mips/lib/bitops.c @@ -146,3 +146,17 @@ int __mips_test_and_change_bit(unsigned long nr, volatile unsigned long *addr) return res; } EXPORT_SYMBOL(__mips_test_and_change_bit); + +bool __mips_xor_is_negative_byte(unsigned long mask, + volatile unsigned long *addr) +{ + unsigned long flags; + unsigned long data; + + raw_local_irq_save(flags); + data = *addr; + *addr = data ^ mask; + raw_local_irq_restore(flags); + + return (data & BIT(7)) != 0; +} From patchwork Fri Sep 15 18:37:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140787 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1392078vqi; Fri, 15 Sep 2023 17:04:22 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG/v1vZMmOSYOu+J+UoeMZ/3gCDu0cVXnOi1YrTys2+hx6qaAmaOm68R2AgrZ98rnCja8Kk X-Received: by 2002:a17:902:c945:b0:1c1:ee26:2337 with SMTP id i5-20020a170902c94500b001c1ee262337mr3310697pla.59.1694822662084; Fri, 15 Sep 2023 17:04:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694822662; cv=none; d=google.com; s=arc-20160816; b=tdA7yqoXWeOkLZWhCinjg0kkzBJBrZ9rYgMAntUr+6f0iEnzrCNy69uopn43XbBxuP CsK++NaJzR3V/gwdawZ7xbwMheMTR6cqoAwvxMiHeZeH69Bbj/H1jOB0gRj1244es7De GUNu+7lFj4ZesCYCeqQ11YwQj2qz+fiRNeCSijEpC7xq41d54EgtFAOjWPiHmRs13085 O7v6rh0j9tz/p5VLFKyVhy0Pw+N42YGpKo3tUQWUsJKX07bzzbBCJ+uAEIbFdG23rjz0 gXR7W3RX6ljLpdNNOcjCpVypJKQcdRflmGyYd0swaLeysWzVJiyA7Vtw8r24JmL6NiOP rkgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1WZX9lC/NFReBBEGgdt7ENtP8Fsb2zBQZdLGumoTT54=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=ngQu7C4Miz4WK4CVNXtdVaHPZme45DKBaOWHnpQxRhvyEvSn1Rv2HxXw/vCjx0BjY9 4a3g/wC37iAX2YdzIZkiacexjAfZH2KGSZvtJcnj/IUnRppt/lm+Sk2dKNgdHQkBLqp1 y1mEgfp95CmIU8G0A3L/NKLweQqhNPWI32/V6nco3BSbPhYLsdd6TlcDTHdZad4iCB7c 6UETvqB+zF/XG6t6RXRs3+YaviKqzADVYhK6OYVhSVS9w1IQYb7MjYhLSbyIhdQbs9Tb yqRhopD02LaHWKzE+hDl4H3aSSPwb7OO/wJdZIriA8nKVujsPstscnNrEedyq9LiDDgB mGww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b="fajBqc/P"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id h14-20020a170902680e00b001c3f5db54acsi3975999plk.635.2023.09.15.17.04.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 17:04:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b="fajBqc/P"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8438C83A7164; Fri, 15 Sep 2023 11:38:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236702AbjIOSiI (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236603AbjIOSh0 (ORCPT ); Fri, 15 Sep 2023 14:37:26 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 924161A5; Fri, 15 Sep 2023 11:37:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=1WZX9lC/NFReBBEGgdt7ENtP8Fsb2zBQZdLGumoTT54=; b=fajBqc/PVmpLWssoMHeJr28vDZ ilkXDBbt56Rb0ualgExvyRQqSpVYXiMZZHCeDXpikIfzxBLmEIzlDYNLfXAZ1Cn/lojNZCh2Wrexh yqIjb+XAVcAH5C3+JpeakQ9TG52GjgtRYiCRD43iBzgHvEj450no85QLc/jqO9Mj7jluQT+wNuylE PIM6rNYr1NdwNDhRENIxbeRz5E6r62lwMe8gNo0dLbIMTIIU62YKKqGRW+VOAL9eY28yseH3ZtoWa vOUnKWwDJStVsL5+9mH1Pc8jGNkzmd4IPnjll/EekaNgUC4o3hB8j4zXSCQ9/Xgey9HuMmuysh7Dm xG1N10FA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgj-00BMIq-U5; Fri, 15 Sep 2023 18:37:09 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 11/17] powerpc: Implement arch_xor_unlock_is_negative_byte on 32-bit Date: Fri, 15 Sep 2023 19:37:01 +0100 Message-Id: <20230915183707.2707298-12-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777150367739919968 X-GMAIL-MSGID: 1777150367739919968 Simply remove the ifdef. The assembly is identical to that in the non-optimised case of test_and_clear_bits() on PPC32, and it's not clear to me how the PPC32 optimisation works, nor whether it would work for arch_xor_unlock_is_negative_byte(). If that optimisation would work, someone can implement it later, but this is more efficient than the implementation in filemap.c. Signed-off-by: Matthew Wilcox (Oracle) --- arch/powerpc/include/asm/bitops.h | 4 ---- 1 file changed, 4 deletions(-) diff --git a/arch/powerpc/include/asm/bitops.h b/arch/powerpc/include/asm/bitops.h index 40cc3ded60cb..671ecc6711e3 100644 --- a/arch/powerpc/include/asm/bitops.h +++ b/arch/powerpc/include/asm/bitops.h @@ -233,7 +233,6 @@ static inline int arch_test_and_change_bit(unsigned long nr, return test_and_change_bits(BIT_MASK(nr), addr + BIT_WORD(nr)) != 0; } -#ifdef CONFIG_PPC64 static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, volatile unsigned long *p) { @@ -251,11 +250,8 @@ static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, return (old & BIT_MASK(7)) != 0; } - #define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte -#endif /* CONFIG_PPC64 */ - #include static inline void arch___clear_bit_unlock(int nr, volatile unsigned long *addr) From patchwork Fri Sep 15 18:37:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140844 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1410344vqi; Fri, 15 Sep 2023 17:57:26 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHXGUhW4B5pPK6UwcldVMAFSsatburowM7LojlIpvf11TahMSCYEJHmTBmqYQDA3iboQrtO X-Received: by 2002:aca:2302:0:b0:3a8:48e7:a6ab with SMTP id e2-20020aca2302000000b003a848e7a6abmr3286247oie.20.1694825846771; Fri, 15 Sep 2023 17:57:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694825846; cv=none; d=google.com; s=arc-20160816; b=OEZoYtrU5+nCBrxygx9hRX/m/05OFyh9WJ9B4PFa/037KJ3OCdDL9zU3pO4UTSqqgn DwRmSaYJLDuy5HFTI0pcnOTcxoWn7SBd3jxLc1raRI2e7ohRsUt+YY1c7dEccQZaGx+j 4aAT9CzThOHdXehCXcm9h7QUpEMjVMvdqDBtZVTHorrqXBqe5a5OIKgYjuE/KpADOFWQ ca6KA4ljD4kfuMlSBs3tSpggP4PUH83K85OO+W6yAwwHU3J8j2YwED6SJfIdqYsmSg8m 59dBjxE1eRFusvULwZzoM3ajDi2e8iA+ThXUOxtYK/VYK2V7cKoRYkU7/O7FG+mgfgON k9FQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aiOK8Ht5TYsSZN2/F06YkBbUJBHTCETngx448aYGRJo=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=U77o9zSv2a+ZtKz3A0tlVKT9y6zZwlz9+ve62eRQI8vF0/+wxo0+I2lhrqL/nAFAIW LAxQ57oRkF18IxXDzDr9faptLYl8z8M3zt+PIfB5siAVSYFKnSQfVqXE+Ic7wchMcGun osqMLLtNpojMpBOWQw8xx3zv/aQp/j9XriLRRBdlTK9C6IQtNE3yTqTA6tQwPOTtjaEK 3Sw99FI+4DJKddlaHAlalgp1MXRktte4jO1JQkY+5nOzSVP5wT9ap4NfJZiDU8gQF8X8 oKpCWNLd+AP0aKbBUYs56z7JfbcxZlqH5dgi5ojFBzIAjz8NegJviW9MGRHcdR6SD2kY W1EA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=lZMQHaT7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id m11-20020a056a00080b00b006901a8f5dadsi4253213pfk.289.2023.09.15.17.57.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 17:57:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=lZMQHaT7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 70CAC838C176; Fri, 15 Sep 2023 11:38:23 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236598AbjIOShv (ORCPT + 30 others); Fri, 15 Sep 2023 14:37:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236584AbjIOShX (ORCPT ); Fri, 15 Sep 2023 14:37:23 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 989CE115; Fri, 15 Sep 2023 11:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=aiOK8Ht5TYsSZN2/F06YkBbUJBHTCETngx448aYGRJo=; b=lZMQHaT7hrHggJzlXzltWr+GSJ qxn67jKJoABNtVciXoBauvhbi6w3aXeLZ0ycZ3yb9R6L4faLx91CivKqEPdiry2EIJjszDL8doCjW 1UfpDGS7b8//AUC/Wz3x0Xe2ldF0BmllLcpXTk3b8tv9FJoO4eqRjV5sm4mQKyejpk/kFrrJX8Lb8 a0wwwl6e0Wh4jKeS43OdT65t/QTdLpmFX8WmtjN8ta2O6GwcTFPfaiyWj/f04D58AtSZ4/Xsvjs8/ jK4zI1ygqWAio3mfxoew53STS7AXfgxZBwVMSeFUh39RWCIxD7DFd5zw9+ZLhR/jlNdpWm1gvozav dtKXH3oA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgk-00BMIw-0T; Fri, 15 Sep 2023 18:37:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 12/17] riscv: Implement xor_unlock_is_negative_byte Date: Fri, 15 Sep 2023 19:37:02 +0100 Message-Id: <20230915183707.2707298-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:23 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777153707077384229 X-GMAIL-MSGID: 1777153707077384229 Inspired by the riscv clear_bit_unlock(), this will surely be more efficient than the generic one defined in filemap.c. Signed-off-by: Matthew Wilcox (Oracle) --- arch/riscv/include/asm/bitops.h | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/arch/riscv/include/asm/bitops.h b/arch/riscv/include/asm/bitops.h index 3540b690944b..15e3044298a2 100644 --- a/arch/riscv/include/asm/bitops.h +++ b/arch/riscv/include/asm/bitops.h @@ -191,6 +191,19 @@ static inline void __clear_bit_unlock( clear_bit_unlock(nr, addr); } +static inline bool xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *addr) +{ + unsigned long res; + __asm__ __volatile__ ( + __AMO(xor) ".rl %0, %2, %1" + : "=r" (res), "+A" (*addr) + : "r" (__NOP(mask)) + : "memory"); + return (res & BIT(7)) != 0; +} +#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte + #undef __test_and_op_bit #undef __op_bit #undef __NOP From patchwork Fri Sep 15 18:37:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140912 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1458267vqi; Fri, 15 Sep 2023 20:36:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEFlBzO93wP6J7XneD3E0dW+3HxL+3Q7H7ouKYCTBjhe38fLeVTS0+jXEtXwmm37cseUbQF X-Received: by 2002:a05:6a00:1c92:b0:68b:fc76:7dea with SMTP id y18-20020a056a001c9200b0068bfc767deamr8894790pfw.12.1694835369024; Fri, 15 Sep 2023 20:36:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694835369; cv=none; d=google.com; s=arc-20160816; b=uh4Dfx1b/t3pe6Io60LP0uKX7BTtzhPTPbzJUe4D7vuGZAk6n/qlnNNE7gG29lqD41 Xr9HMCbYRER5YBqK/YccwUFmVH5jk2z7O2Byh6BAz/podGDglCoCTkqFXBUHlmoH1Vxb Yl/+me9sILMrW+2EU/hp/ZBtycYAqPeDwJbv1XQ7Fx2ADTaK/2t4yXuM4sWq8QN+qAqy tzi2C+CT/Olg0jzSOuVdCoCLIc5dzlQSWsW5KbcM4HWkLjMCB0Sylqfq1XV5p6/Gh5Dt CjTHcsKQYzunkKHNGTcM08aIlqcYUXSFpgYXMBEx25dWHwaRh4Fqp4iOlIZW8XscVHlH yM4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NoYAPE+UO7xNrTZ3UFo3zibiwdFupzttBoXxylMj4Qw=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=aZlapefOA9WGa+cVBwvReNnxxPVZCjV8E8kJwnDItyQCX5uUXA+rb5+lD3+6J3B9f1 a4h9ulog4a7/WgnwtTdY5GtDRonedm0vhc2OKfeNFz88E+xahmoB0nPyyGM4JXZvdNTz lq/QCc34iD7wun8yHMF64jxSRPZD4c5TOSRh3AaOarxXBmERclluTd8cXL4trSHH1Wq9 gWtAmZxAt5peSopiE+zXgpzIs7LJCegEKRDqLhJO7ympAF7ZRuGLu7xHpu9VqGQ6pTXh kASyjE79ZX13MtdiDrnZRkmJ9UHdMUUaqAbZD892TyyLw524qy4s296EO+Tu1z3O/2au ZPNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=H9LvAtog; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id g7-20020a056a0023c700b006902507d409si4502484pfc.174.2023.09.15.20.36.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 20:36:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=H9LvAtog; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 0827980A222D; Fri, 15 Sep 2023 11:39:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236685AbjIOSiD (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236587AbjIOShX (ORCPT ); Fri, 15 Sep 2023 14:37:23 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B165210E6; Fri, 15 Sep 2023 11:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=NoYAPE+UO7xNrTZ3UFo3zibiwdFupzttBoXxylMj4Qw=; b=H9LvAtogpvwX8ITigBgif20y3d Ksa3CrrVI3uaFqgH46vcGpzQGIbqqcgmmazDLvlAPd3WywSfgjryFPPXaF/5s/4Ksm1o5DXsq6PHT d118mcjKIzIS1RtLFAt/4I2kRM0niTEAbsc1MW/RuESCl87qFg73uDcHMTGKkeUONBGbDyBj5jrsx b7tUYeKMv4Ax9ha9W5IXgD9GUjlHlk+FnXw2zD9DZkL6avroCIGOqicNVMtEulxTb6/tsGEDX+11x OQIhNrD9uxgZFfv2F5U8xcBzdADdQizHw/96XJotDdOexdkjT2PlWFuT773ZhaL83QnsJwAtKgnsI hEgaBaCw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgk-00BMIy-3W; Fri, 15 Sep 2023 18:37:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 13/17] s390: Implement arch_xor_unlock_is_negative_byte Date: Fri, 15 Sep 2023 19:37:03 +0100 Message-Id: <20230915183707.2707298-14-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:39:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777163691915351783 X-GMAIL-MSGID: 1777163691915351783 Inspired by the s390 arch_test_and_clear_bit(), this will surely be more efficient than the generic one defined in filemap.c. Signed-off-by: Matthew Wilcox (Oracle) --- arch/s390/include/asm/bitops.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/s390/include/asm/bitops.h b/arch/s390/include/asm/bitops.h index 2de74fcd0578..c467dffa8c12 100644 --- a/arch/s390/include/asm/bitops.h +++ b/arch/s390/include/asm/bitops.h @@ -201,6 +201,16 @@ static inline void arch___clear_bit_unlock(unsigned long nr, arch___clear_bit(nr, ptr); } +static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, + volatile unsigned long *ptr) +{ + unsigned long old; + + old = __atomic64_xor_barrier(mask, (long *)ptr); + return old & BIT(7); +} +#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte + #include #include #include From patchwork Fri Sep 15 18:37:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140745 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1364214vqi; Fri, 15 Sep 2023 15:52:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGQOup4KhtB7a94T/IeywtqRRIdsm9mUG2fHS40VfiLUw1i189M4Il0DQqTo7LiA78FyPtx X-Received: by 2002:a05:6a00:24ca:b0:690:463a:fa9d with SMTP id d10-20020a056a0024ca00b00690463afa9dmr3585660pfv.22.1694818352648; Fri, 15 Sep 2023 15:52:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694818352; cv=none; d=google.com; s=arc-20160816; b=JChHjGYf8utRSX/csTf+yJg/hOzoyL+wJNiyxpPVMBLsPkDIF0RDrianRnO7yu3XJc WgxkO3Lk8G2pcwjAsXi5xUUfBVwomo0Xtd5OCOcUEsLgtow1VQpA69hYoZnOZrJeRGry coAjEtd+2D/paPz4DfcQUtkaoVFk/4S147fYkoua6+z8Jz/sIibntZ0LF54Ha7N40hEi d08BZNfTHd1RQbtK2Pe7EFdam90gA5Tq8s7zNZwnJLu6+KyuscSAk3wP9ipIe59xegaJ qNe1MPUl76wngN7U7ZbNEzrCnqPxQ0vd+DgAj+eqkg0g/AWfjbamqYE1E61Q8LvPXPfv rPuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+IWZ1pVUhVN3z1l/dMaghxmkHDofRoTXIwAAX7e84tQ=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=AdiDS9mE+xkoQoT2sSHIEmS1zyjPd9ZCZT4AZEx7o8JpN/k5D5y/dPHJDM3mLMCG7B zxJ021qMmhi8s12CLz+aOIMzLrZrQUfU+X7Rx6/w43Q2MZWOg4SGfMc5mm+iIEaC1WBe jNRQ9A1RVulcG0szDDr3MoprWzl+uiAHaEooOBQ63IrtaXaGMGOnmD15VSKre59rXsBO ZKJ2bTEFp+hZwSTCVVdGwTpMImXQB0smBJw1VV1rNOJ1dqSlOGIjx/v0xFFpYQwtUJlY 1GmAhS7z0s3as3zGkLLtiuinKVwLAhiTvI7+YReIjPNgFskia0kbO+vDL7Bu12Dh4HMo bodg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Awtg9B4S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id bx15-20020a056a00428f00b0068c7033a5f5si3888687pfb.74.2023.09.15.15.52.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 15:52:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Awtg9B4S; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id B1A3180218AC; Fri, 15 Sep 2023 11:38:51 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236679AbjIOSiA (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236589AbjIOShY (ORCPT ); Fri, 15 Sep 2023 14:37:24 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0D3110F7; Fri, 15 Sep 2023 11:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+IWZ1pVUhVN3z1l/dMaghxmkHDofRoTXIwAAX7e84tQ=; b=Awtg9B4S8FOkho5cI1bYeYXr32 A/lwQ5v9+ST7XpFne3FVrkJIjX06WHJGZy2Y7Oj1GLuEmQ9UeWtFKxUjWWZMnYt2ag8KVz7no2bwE W8QZNKYkP7R60whmbRDnpmstn62QkRsWjNboQ1TW8GqF8DM3ky4xq4AoUb74B4yjrgNeaG9kjVj4e HcoQleM1nqwQ0SgcP3HXZPwj0KrtX281KtSA0luMyEXxgD0YlMW8z+npKLjXGRSqX4ihLSPW+jkFf khB6bdSTKNthkTudWwDaZhahi2HRBEeoeY7zOAzBUDFqQN6DZul7mOHVmhY2asnTyjCqGiTxC9kHE 4K55kgBA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgk-00BMJ0-6E; Fri, 15 Sep 2023 18:37:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 14/17] mm: Delete checks for xor_unlock_is_negative_byte() Date: Fri, 15 Sep 2023 19:37:04 +0100 Message-Id: <20230915183707.2707298-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:51 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777145848783929115 X-GMAIL-MSGID: 1777145848783929115 Architectures which don't define their own use the one in asm-generic/bitops/lock.h. Get rid of all the ifdefs around "maybe we don't have it". Signed-off-by: Matthew Wilcox (Oracle) --- arch/alpha/include/asm/bitops.h | 1 - arch/m68k/include/asm/bitops.h | 1 - arch/mips/include/asm/bitops.h | 1 - arch/riscv/include/asm/bitops.h | 1 - .../asm-generic/bitops/instrumented-lock.h | 5 ---- include/asm-generic/bitops/lock.h | 1 - kernel/kcsan/kcsan_test.c | 3 -- kernel/kcsan/selftest.c | 3 -- mm/filemap.c | 30 +------------------ mm/kasan/kasan_test.c | 3 -- 10 files changed, 1 insertion(+), 48 deletions(-) diff --git a/arch/alpha/include/asm/bitops.h b/arch/alpha/include/asm/bitops.h index 138930e8a504..29de3c623f65 100644 --- a/arch/alpha/include/asm/bitops.h +++ b/arch/alpha/include/asm/bitops.h @@ -305,7 +305,6 @@ static inline bool xor_unlock_is_negative_byte(unsigned long mask, return (old & BIT(7)) != 0; } -#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte /* * ffz = Find First Zero in word. Undefined if no zero exists, diff --git a/arch/m68k/include/asm/bitops.h b/arch/m68k/include/asm/bitops.h index 909ebe7cab5d..ec82bb734aef 100644 --- a/arch/m68k/include/asm/bitops.h +++ b/arch/m68k/include/asm/bitops.h @@ -331,7 +331,6 @@ static inline bool xor_unlock_is_negative_byte(unsigned long mask, : "memory"); return result; } -#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte /* * The true 68020 and more advanced processors support the "bfffo" diff --git a/arch/mips/include/asm/bitops.h b/arch/mips/include/asm/bitops.h index d98a05c478f4..89f73d1a4ea4 100644 --- a/arch/mips/include/asm/bitops.h +++ b/arch/mips/include/asm/bitops.h @@ -301,7 +301,6 @@ static inline bool xor_unlock_is_negative_byte(unsigned long mask, return res; } -#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte #undef __bit_op #undef __test_bit_op diff --git a/arch/riscv/include/asm/bitops.h b/arch/riscv/include/asm/bitops.h index 15e3044298a2..65f6eee4ab8d 100644 --- a/arch/riscv/include/asm/bitops.h +++ b/arch/riscv/include/asm/bitops.h @@ -202,7 +202,6 @@ static inline bool xor_unlock_is_negative_byte(unsigned long mask, : "memory"); return (res & BIT(7)) != 0; } -#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte #undef __test_and_op_bit #undef __op_bit diff --git a/include/asm-generic/bitops/instrumented-lock.h b/include/asm-generic/bitops/instrumented-lock.h index e8ea3aeda9a9..542d3727ee4e 100644 --- a/include/asm-generic/bitops/instrumented-lock.h +++ b/include/asm-generic/bitops/instrumented-lock.h @@ -58,7 +58,6 @@ static inline bool test_and_set_bit_lock(long nr, volatile unsigned long *addr) return arch_test_and_set_bit_lock(nr, addr); } -#if defined(arch_xor_unlock_is_negative_byte) /** * xor_unlock_is_negative_byte - XOR a single byte in memory and test if * it is negative, for unlock. @@ -80,8 +79,4 @@ static inline bool xor_unlock_is_negative_byte(unsigned long mask, instrument_atomic_write(addr, sizeof(long)); return arch_xor_unlock_is_negative_byte(mask, addr); } -/* Let everybody know we have it. */ -#define xor_unlock_is_negative_byte xor_unlock_is_negative_byte -#endif - #endif /* _ASM_GENERIC_BITOPS_INSTRUMENTED_LOCK_H */ diff --git a/include/asm-generic/bitops/lock.h b/include/asm-generic/bitops/lock.h index 6a638e89d130..14d4ec8c5152 100644 --- a/include/asm-generic/bitops/lock.h +++ b/include/asm-generic/bitops/lock.h @@ -75,7 +75,6 @@ static inline bool arch_xor_unlock_is_negative_byte(unsigned long mask, old = raw_atomic_long_fetch_xor_release(mask, (atomic_long_t *)p); return !!(old & BIT(7)); } -#define arch_xor_unlock_is_negative_byte arch_xor_unlock_is_negative_byte #endif #include diff --git a/kernel/kcsan/kcsan_test.c b/kernel/kcsan/kcsan_test.c index 1333d23ac4ef..015586217875 100644 --- a/kernel/kcsan/kcsan_test.c +++ b/kernel/kcsan/kcsan_test.c @@ -699,12 +699,9 @@ static void test_barrier_nothreads(struct kunit *test) KCSAN_EXPECT_RW_BARRIER(spin_unlock(&test_spinlock), true); KCSAN_EXPECT_RW_BARRIER(mutex_lock(&test_mutex), false); KCSAN_EXPECT_RW_BARRIER(mutex_unlock(&test_mutex), true); - -#ifdef xor_unlock_is_negative_byte KCSAN_EXPECT_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); KCSAN_EXPECT_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); KCSAN_EXPECT_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var), true); -#endif kcsan_nestable_atomic_end(); } diff --git a/kernel/kcsan/selftest.c b/kernel/kcsan/selftest.c index 619be7417420..84a1200271af 100644 --- a/kernel/kcsan/selftest.c +++ b/kernel/kcsan/selftest.c @@ -227,12 +227,9 @@ static bool __init test_barrier(void) KCSAN_CHECK_RW_BARRIER(arch_spin_unlock(&arch_spinlock)); spin_lock(&test_spinlock); KCSAN_CHECK_RW_BARRIER(spin_unlock(&test_spinlock)); - -#ifdef xor_unlock_is_negative_byte KCSAN_CHECK_RW_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); KCSAN_CHECK_READ_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); KCSAN_CHECK_WRITE_BARRIER(xor_unlock_is_negative_byte(1, &test_var)); -#endif kcsan_nestable_atomic_end(); return ret; diff --git a/mm/filemap.c b/mm/filemap.c index 2db7c5da9e03..330e21da6863 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1484,34 +1484,6 @@ void folio_add_wait_queue(struct folio *folio, wait_queue_entry_t *waiter) } EXPORT_SYMBOL_GPL(folio_add_wait_queue); -#ifdef xor_unlock_is_negative_byte -#define clear_bit_unlock_is_negative_byte(nr, p) \ - xor_unlock_is_negative_byte(1 << nr, p) -#endif - -#ifndef clear_bit_unlock_is_negative_byte - -/* - * PG_waiters is the high bit in the same byte as PG_lock. - * - * On x86 (and on many other architectures), we can clear PG_lock and - * test the sign bit at the same time. But if the architecture does - * not support that special operation, we just do this all by hand - * instead. - * - * The read of PG_waiters has to be after (or concurrently with) PG_locked - * being cleared, but a memory barrier should be unnecessary since it is - * in the same byte as PG_locked. - */ -static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem) -{ - clear_bit_unlock(nr, mem); - /* smp_mb__after_atomic(); */ - return test_bit(PG_waiters, mem); -} - -#endif - /** * folio_unlock - Unlock a locked folio. * @folio: The folio. @@ -1527,7 +1499,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0))) + if (xor_unlock_is_negative_byte(1 << PG_locked, folio_flags(folio, 0))) folio_wake_bit(folio, PG_locked); } EXPORT_SYMBOL(folio_unlock); diff --git a/mm/kasan/kasan_test.c b/mm/kasan/kasan_test.c index c1de091dfc92..b241b57427b1 100644 --- a/mm/kasan/kasan_test.c +++ b/mm/kasan/kasan_test.c @@ -1097,12 +1097,9 @@ static void kasan_bitops_test_and_modify(struct kunit *test, int nr, void *addr) KUNIT_EXPECT_KASAN_FAIL(test, test_and_change_bit(nr, addr)); KUNIT_EXPECT_KASAN_FAIL(test, __test_and_change_bit(nr, addr)); KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = test_bit(nr, addr)); - -#if defined(xor_unlock_is_negative_byte) if (nr < 7) KUNIT_EXPECT_KASAN_FAIL(test, kasan_int_result = xor_unlock_is_negative_byte(1 << nr, addr)); -#endif } static void kasan_bitops_generic(struct kunit *test) From patchwork Fri Sep 15 18:37:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140695 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1321946vqi; Fri, 15 Sep 2023 14:11:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFwFVe19vKppCeq7WWZ0qfcK3mFXTAXO4xH1qUX3QIMNGEhM6V/ic2HogLwU3jhcKi7BlsQ X-Received: by 2002:a05:6a00:808e:b0:690:3956:6ec7 with SMTP id eh14-20020a056a00808e00b0069039566ec7mr2959331pfb.2.1694812297997; Fri, 15 Sep 2023 14:11:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694812297; cv=none; d=google.com; s=arc-20160816; b=v2shdIwNkMCfqT6HgtigjMMuoiu6o5FsdfU6DY9UPA4DkbYCCAP2gvQkHoPpB07KAP Je+Wbg4tSEN8r9zGAHYSGPCCa4z5M0um42LKT/umDcLllA5D1rsL35EoGzf4iHGpL45E H8GydAsiaqrz+y2EiU5lHEYO3wSFasX0RS9TXIO6bs5l0649wwcqu6rODj6UL1fLzT+w 62xNGfXBLjoVe96rOj5E8f2m/8A8YIoc6az+wXZr5baJFX36FVPJBmrLu6MkTHKgBHVX LA/LWcwYiAhl6Y2jhE5j+StPbCHkBlrEorRKE6J/neyPjf9LRIq59mIhxKLifehgyGUV 3G/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3Pka9W0lBex+AQwPauK3FVj6595pxpOZ2H5fxz/mNAA=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=w6hO6eiul6IDPHtyLAIDCvSq4WQLMjRH/AMJy5fmizfSxrd/XFeZ10vYmweI1IJhBF q8lIyUEAsnyj1Q5PAg+ERKyWSR89A47fn2+CDVnWa+KjtGtH2StEoelBcCpikMCzCWdv jbyvwK7DnzD2kSH4TZC1b4vFU9Fi0jVPPyAiE9XnTB02clbK1OF/m5C5XcBhWtmDqYHb 9GUHWKUp1yF7J2N3IBf49rYwqniAnhknC4DpgLOfeb7RUo4cJrYkzA1EIgpLNFICq6e4 eiFRzq0Er9yeGAQbkfuRs/wIEIrbpgunTdlegZHjTwEXcUs6kQK66kmiBf8GaHTgPGLV gA4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b="dFE/BuzQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id y3-20020a636403000000b005694f4b9e85si3845684pgb.229.2023.09.15.14.11.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 14:11:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b="dFE/BuzQ"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 1537D84116DB; Fri, 15 Sep 2023 11:38:37 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236657AbjIOSh6 (ORCPT + 30 others); Fri, 15 Sep 2023 14:37:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236593AbjIOShY (ORCPT ); Fri, 15 Sep 2023 14:37:24 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D42151A5; Fri, 15 Sep 2023 11:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=3Pka9W0lBex+AQwPauK3FVj6595pxpOZ2H5fxz/mNAA=; b=dFE/BuzQ/4dk84AB1Iba3+XdrC JXMu/iEd34lZucm5IbEtzAh+uiXBzC6JWIWXJ6L7G0viTTqwusion1sIbrNWTucYQR93SKyF/Zplk 9ahMcug8JjkZUPz4oWGKulmYYR/++Y385/j2XFDUIPdup9V4Z15DB41Yg2t3NuIfP7KsQttats7Uf BTf4Hpf7CqATNEDHv1XhExUH5g3e0/knqGKLDjxfObp04SXaNypzCk5zeSr3WHLluqDpuJylaIadD O7R+3Y3zcxD3M5re5XX7GAOJy7py560s5Kow44PupHQ/cHTyaIw+bxxmbsRClEkRGDt8tHkwSaLr/ Nmqa0TOw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgk-00BMJ2-9B; Fri, 15 Sep 2023 18:37:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 15/17] mm: Add folio_xor_flags_has_waiters() Date: Fri, 15 Sep 2023 19:37:05 +0100 Message-Id: <20230915183707.2707298-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:38:37 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777139500264800800 X-GMAIL-MSGID: 1777139500264800800 Optimise folio_end_read() by setting the uptodate bit at the same time we clear the unlock bit. This saves at least one memory barrier and one write-after-write hazard. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/page-flags.h | 19 +++++++++++++++++++ mm/filemap.c | 14 +++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 5c02720c53a5..a88e64acebfe 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -692,6 +692,25 @@ TESTPAGEFLAG_FALSE(Ksm, ksm) u64 stable_page_flags(struct page *page); +/** + * folio_xor_flags_has_waiters - Change some folio flags. + * @folio: The folio. + * @mask: Bits set in this word will be changed. + * + * This must only be used for flags which are changed with the folio + * lock held. For example, it is unsafe to use for PG_dirty as that + * can be set without the folio lock held. It can also only be used + * on flags which are in the range 0-6 as some of the implementations + * only affect those bits. + * + * Return: Whether there are tasks waiting on the folio. + */ +static inline bool folio_xor_flags_has_waiters(struct folio *folio, + unsigned long mask) +{ + return xor_unlock_is_negative_byte(mask, folio_flags(folio, 0)); +} + /** * folio_test_uptodate - Is this folio up to date? * @folio: The folio. diff --git a/mm/filemap.c b/mm/filemap.c index 330e21da6863..8262b85593be 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1499,7 +1499,7 @@ void folio_unlock(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); BUILD_BUG_ON(PG_locked > 7); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); - if (xor_unlock_is_negative_byte(1 << PG_locked, folio_flags(folio, 0))) + if (folio_xor_flags_has_waiters(folio, 1 << PG_locked)) folio_wake_bit(folio, PG_locked); } EXPORT_SYMBOL(folio_unlock); @@ -1520,9 +1520,17 @@ EXPORT_SYMBOL(folio_unlock); */ void folio_end_read(struct folio *folio, bool success) { + unsigned long mask = 1 << PG_locked; + + /* Must be in bottom byte for x86 to work */ + BUILD_BUG_ON(PG_uptodate > 7); + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio); + if (success) - folio_mark_uptodate(folio); - folio_unlock(folio); + mask |= 1 << PG_uptodate; + if (folio_xor_flags_has_waiters(folio, mask)) + folio_wake_bit(folio, PG_locked); } EXPORT_SYMBOL(folio_end_read); From patchwork Fri Sep 15 18:37:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140619 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1249679vqi; Fri, 15 Sep 2023 11:44:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IExAqh9MiUacLlh7w/eJA3Rc5XuZTvNniwKmp8yadcH3Risg0m8hb6d/K+ZEhEsAGhv2yt6 X-Received: by 2002:a17:903:110e:b0:1b8:8682:62fb with SMTP id n14-20020a170903110e00b001b8868262fbmr7752317plh.4.1694803443609; Fri, 15 Sep 2023 11:44:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694803443; cv=none; d=google.com; s=arc-20160816; b=MMZWTTYifoWcsOYx+6YujOEdL4d7ttDR3h5cp3UlAk6df/MM/cO2Ee8aWK8YqMWUcO s95i4qseDIz9gH/zP0xmUPD+N5WVK9ZVoZ0bKIPK3jQd4EyN4KwJdR8U7URN4lgvZQxz xOKalvZYNhh2CsvmIwhmze/WW7pIDCHsO2rJdJ1UA3FtFDEGYkGnwnhBVMzOha9orjnd nkEImGlo//7fHYLuGxpgknM2L8GAyXYgYaYtnO0M/KIEdKierorO6SXlhGNtIFSauppQ x09Jk4OHcKB++K9oY0aaexLU1BujvsTQJFAVYk0kgncx+X0Zt3qOyNpKUABOM7+N50gF okSg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=I/mcxM9CcPo8xWIB8wMPmyQcFOb+eL3EOkmPufsZvNw=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=Gu3zwHsMoVPzm55TQCH5N3c6D2dFnhjFdywO6ghNhxkEEZkTwXUrl7RSK/FUww866t Qh7fhtc+gEqmQG96azPNynoFvWwIHK3z5uQUVfm5u8yyVcsEcDdy++LERjTSuCFTVfmr njJ7HcFSZbMhBBlIfv5G/N/yJQh15HxCV5xqH43bPfJ7Pp+4fkb3eX6A8GmTfcgLhyKX TvWyyrHInBQ7NTEqL+SOP6DwRf/PHte+F2f9gJz/GntPT+8wawO+YhooJKy0gmkU73dG 7ennssEgpkYXKrLX3Q5hM3iTsJsYSPUlomIR/PvUDw73mMYVFwk6WUzaVHSYK9kalyqm Rh5A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=algmUIsk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id u11-20020a170902e80b00b001bdc9c55295si4068141plg.593.2023.09.15.11.44.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Sep 2023 11:44:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=algmUIsk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 983208260AFA; Fri, 15 Sep 2023 11:39:03 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236641AbjIOSh4 (ORCPT + 30 others); Fri, 15 Sep 2023 14:37:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236592AbjIOShY (ORCPT ); Fri, 15 Sep 2023 14:37:24 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F1B701A8; Fri, 15 Sep 2023 11:37:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=I/mcxM9CcPo8xWIB8wMPmyQcFOb+eL3EOkmPufsZvNw=; b=algmUIskdAyoKr4qTO12fn4nRP fWLPKU3eyRF4p6Wl8lL1WBSLrQnHyYX2LVbQ31kpdJwdCDcjOUlPKUaKt5ioI+pkE+HwdXlzdg7zO lZ3FlU/oq2TQ7lGaqRCMeQg62X7pIhANT4NOvAuIfVm3/PmOzSbaTsPT2AXa39N2sxzwaecBCw6Gz Vlcid+5xzcrq03Ytu2JU2xoHSwAg5Zy6EcHen3aVwRFhMHGfhzM7GIyZQ69/80pCOjI9w/aW0QYIw oH6FkDmIYQ8sHKn/4STyq1ufRrC9MrZsk89jtRUNrePxRD43t9kJK1s8MAFRelCd8ti9WHeBfluuo DPIGfbzg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgk-00BMJ4-Bx; Fri, 15 Sep 2023 18:37:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 16/17] mm: Make __end_folio_writeback() return void Date: Fri, 15 Sep 2023 19:37:06 +0100 Message-Id: <20230915183707.2707298-17-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:39:03 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777130215757562919 X-GMAIL-MSGID: 1777130215757562919 Rather than check the result of test-and-clear, just check that we have the writeback bit set at the start. This wouldn't catch every case, but it's good enough (and enables the next patch). Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 9 +++++++-- mm/internal.h | 2 +- mm/page-writeback.c | 38 ++++++++++++++++---------------------- 3 files changed, 24 insertions(+), 25 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 8262b85593be..53c0d71aae8e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1595,9 +1595,15 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); /** * folio_end_writeback - End writeback against a folio. * @folio: The folio. + * + * The folio must actually be under writeback. + * + * Context: May be called from process or interrupt context. */ void folio_end_writeback(struct folio *folio) { + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); + /* * folio_test_clear_reclaim() could be used here but it is an * atomic operation and overkill in this particular case. Failing @@ -1617,8 +1623,7 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake(). */ folio_get(folio); - if (!__folio_end_writeback(folio)) - BUG(); + __folio_end_writeback(folio); smp_mb__after_atomic(); folio_wake(folio, PG_writeback); diff --git a/mm/internal.h b/mm/internal.h index 30cf724ddbce..ccb08dd9b5ec 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ static inline void wake_throttle_isolated(pg_data_t *pgdat) vm_fault_t do_swap_page(struct vm_fault *vmf); void folio_rotate_reclaimable(struct folio *folio); -bool __folio_end_writeback(struct folio *folio); +void __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); void folio_activate(struct folio *folio); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index b8d3d7040a50..410b53e888e3 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2940,11 +2940,10 @@ static void wb_inode_writeback_end(struct bdi_writeback *wb) spin_unlock_irqrestore(&wb->work_lock, flags); } -bool __folio_end_writeback(struct folio *folio) +void __folio_end_writeback(struct folio *folio) { long nr = folio_nr_pages(folio); struct address_space *mapping = folio_mapping(folio); - bool ret; folio_memcg_lock(folio); if (mapping && mapping_use_writeback_tags(mapping)) { @@ -2953,19 +2952,16 @@ bool __folio_end_writeback(struct folio *folio) unsigned long flags; xa_lock_irqsave(&mapping->i_pages, flags); - ret = folio_test_clear_writeback(folio); - if (ret) { - __xa_clear_mark(&mapping->i_pages, folio_index(folio), - PAGECACHE_TAG_WRITEBACK); - if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) { - struct bdi_writeback *wb = inode_to_wb(inode); - - wb_stat_mod(wb, WB_WRITEBACK, -nr); - __wb_writeout_add(wb, nr); - if (!mapping_tagged(mapping, - PAGECACHE_TAG_WRITEBACK)) - wb_inode_writeback_end(wb); - } + folio_test_clear_writeback(folio); + __xa_clear_mark(&mapping->i_pages, folio_index(folio), + PAGECACHE_TAG_WRITEBACK); + if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) { + struct bdi_writeback *wb = inode_to_wb(inode); + + wb_stat_mod(wb, WB_WRITEBACK, -nr); + __wb_writeout_add(wb, nr); + if (!mapping_tagged(mapping, PAGECACHE_TAG_WRITEBACK)) + wb_inode_writeback_end(wb); } if (mapping->host && !mapping_tagged(mapping, @@ -2974,15 +2970,13 @@ bool __folio_end_writeback(struct folio *folio) xa_unlock_irqrestore(&mapping->i_pages, flags); } else { - ret = folio_test_clear_writeback(folio); - } - if (ret) { - lruvec_stat_mod_folio(folio, NR_WRITEBACK, -nr); - zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr); - node_stat_mod_folio(folio, NR_WRITTEN, nr); + folio_test_clear_writeback(folio); } + + lruvec_stat_mod_folio(folio, NR_WRITEBACK, -nr); + zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr); + node_stat_mod_folio(folio, NR_WRITTEN, nr); folio_memcg_unlock(folio); - return ret; } bool __folio_start_writeback(struct folio *folio, bool keep_write) From patchwork Fri Sep 15 18:37:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 140995 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp1544448vqi; Sat, 16 Sep 2023 01:17:09 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEI3JLNBDRiZQoICl5PLqPlkQ7kDWlimCl4Tj0ODxujK2Exd+mrfI1B2XW+MNLXDbgU/eJR X-Received: by 2002:a17:902:c1cc:b0:1bb:25bd:d09c with SMTP id c12-20020a170902c1cc00b001bb25bdd09cmr3385546plc.1.1694852228795; Sat, 16 Sep 2023 01:17:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694852228; cv=none; d=google.com; s=arc-20160816; b=Tn0n6sgbzjI1vK+Kz0ZpHXq5CfO/I6HnICKMluEKIIJvcuydP6gXstOTW0uIGr5Z0/ Xsp1HuYdv2FclhlqXpZ1s2DT0yOMbcUZZC2Sj+Gc60TiN/8VtI1LNTa1FADFqxe5OLt2 78t4y2nR/bwVqbNn7xgOBYOxrSzaDY21eNSbnJmf+5wxnO3ukKdmKzMNEAr8JMCD3UiK bT7nCD0PO2slk1uJzAWq+BTURKEuac93X/Lz6oMq1r6EvXbuy1dUXB9DPaQYcrU7gQgz krOzj8gUsnK0YGq4A01sHzqBGgy7ZUq6g1VsUAC5H7P3zqu8MeLqx84N4/JHp8EJt5r6 rN9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=n7u0d+abmaVaMLzO4YFLO6CA1M26m1t19aSjcVFsvyQ=; fh=Exu8p2J0bt/MoYkEByA/Woj5EWTdUoRsvJYmK88z1hY=; b=p4WMZ+2NTUApBV1+6AYYT7Kr/tZGW2YZFPVfvS3F61OX+xcgogieO1weRd3O5Elr5S eHlHcMYVofCygletcQapqfVQeQ9dTOep+6TwRk0UZme4KP0UO8r4u7da55SSb6f7AAUF Nq6yUqEMgk95kCRbVxbzOUTX8Yx7swAJDnV9qYYHAqUKuYTHev5KXPH3iOS8G11khQWF THf+1Ghnji/Zvvpwns3gL2xIsu0IpRZOp36KFYUckntOACnO7clIJmoL2RK0Li3J/cwc zVOX2CeRAjs5pRCBW1vyAmiWjB1y7d+KbcycKeW52Cl2apsrN6cJQB0GFfUOcqKwW23E AuNw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Ki3hIdnh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id x15-20020a170902a38f00b001bc67c761edsi865648pla.93.2023.09.16.01.17.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 16 Sep 2023 01:17:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Ki3hIdnh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id DF86D8414958; Fri, 15 Sep 2023 11:39:03 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236771AbjIOSiS (ORCPT + 30 others); Fri, 15 Sep 2023 14:38:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236547AbjIOShr (ORCPT ); Fri, 15 Sep 2023 14:37:47 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75DBA1BD4; Fri, 15 Sep 2023 11:37:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=n7u0d+abmaVaMLzO4YFLO6CA1M26m1t19aSjcVFsvyQ=; b=Ki3hIdnhdMv60FfdfEOljvltYs O2p6WhI14WeOIkOkpuGNThSphzEWgamOZ2tl4TadV+sLFcS3OGaX5vUz6YydTrxv0+xPb9J15TdfB Wkj52lf53qWnjsP77RGlXpFUIqCMw36qV8f7ZsEpS7+qGq+phMBML/R7NY85CrpdbzXbEfabQ3J0F 9hTbkW4hBz4ZwVrUc91m2AUqR+Gj771pbofj+7lhGgqrTcCND5JHMjPbpzJGYkCrsq1qgJ12msS9o ohnFu3M2vVceno8fThSIIaDRW2ceuNofx1ZW3XjdFwp9oJR6+liR9lEN879iiI+VogQUS8uSWTNBo reN4LkIA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qhDgk-00BMJ6-Ef; Fri, 15 Sep 2023 18:37:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-kernel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, torvalds@linux-foundation.org, Nicholas Piggin Subject: [PATCH 17/17] mm: Use folio_xor_flags_has_waiters() in folio_end_writeback() Date: Fri, 15 Sep 2023 19:37:07 +0100 Message-Id: <20230915183707.2707298-18-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230915183707.2707298-1-willy@infradead.org> References: <20230915183707.2707298-1-willy@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Fri, 15 Sep 2023 11:39:04 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777181370580571303 X-GMAIL-MSGID: 1777181370580571303 Match how folio_unlock() works by combining the test for PG_waiters with the clearing of PG_writeback. This should have a small performance win, and removes the last user of folio_wake(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 15 +++------------ mm/internal.h | 2 +- mm/page-writeback.c | 9 ++++++--- 3 files changed, 10 insertions(+), 16 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 53c0d71aae8e..f57b62d65001 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1177,13 +1177,6 @@ static void folio_wake_bit(struct folio *folio, int bit_nr) spin_unlock_irqrestore(&q->lock, flags); } -static void folio_wake(struct folio *folio, int bit) -{ - if (!folio_test_waiters(folio)) - return; - folio_wake_bit(folio, bit); -} - /* * A choice of three behaviors for folio_wait_bit_common(): */ @@ -1620,13 +1613,11 @@ void folio_end_writeback(struct folio *folio) * Writeback does not hold a folio reference of its own, relying * on truncation to wait for the clearing of PG_writeback. * But here we must make sure that the folio is not freed and - * reused before the folio_wake(). + * reused before the folio_wake_bit(). */ folio_get(folio); - __folio_end_writeback(folio); - - smp_mb__after_atomic(); - folio_wake(folio, PG_writeback); + if (__folio_end_writeback(folio)) + folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); folio_put(folio); } diff --git a/mm/internal.h b/mm/internal.h index ccb08dd9b5ec..30cf724ddbce 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -105,7 +105,7 @@ static inline void wake_throttle_isolated(pg_data_t *pgdat) vm_fault_t do_swap_page(struct vm_fault *vmf); void folio_rotate_reclaimable(struct folio *folio); -void __folio_end_writeback(struct folio *folio); +bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); void folio_activate(struct folio *folio); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 410b53e888e3..c52514fe1d23 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2940,10 +2940,11 @@ static void wb_inode_writeback_end(struct bdi_writeback *wb) spin_unlock_irqrestore(&wb->work_lock, flags); } -void __folio_end_writeback(struct folio *folio) +bool __folio_end_writeback(struct folio *folio) { long nr = folio_nr_pages(folio); struct address_space *mapping = folio_mapping(folio); + bool ret; folio_memcg_lock(folio); if (mapping && mapping_use_writeback_tags(mapping)) { @@ -2952,7 +2953,7 @@ void __folio_end_writeback(struct folio *folio) unsigned long flags; xa_lock_irqsave(&mapping->i_pages, flags); - folio_test_clear_writeback(folio); + ret = folio_xor_flags_has_waiters(folio, 1 << PG_writeback); __xa_clear_mark(&mapping->i_pages, folio_index(folio), PAGECACHE_TAG_WRITEBACK); if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) { @@ -2970,13 +2971,15 @@ void __folio_end_writeback(struct folio *folio) xa_unlock_irqrestore(&mapping->i_pages, flags); } else { - folio_test_clear_writeback(folio); + ret = folio_xor_flags_has_waiters(folio, 1 << PG_writeback); } lruvec_stat_mod_folio(folio, NR_WRITEBACK, -nr); zone_stat_mod_folio(folio, NR_ZONE_WRITE_PENDING, -nr); node_stat_mod_folio(folio, NR_WRITTEN, nr); folio_memcg_unlock(folio); + + return ret; } bool __folio_start_writeback(struct folio *folio, bool keep_write)