Message ID | 20230120195215.42243-1-jlayton@kernel.org |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp395788wrn; Fri, 20 Jan 2023 11:55:25 -0800 (PST) X-Google-Smtp-Source: AMrXdXu+Le3zNKf4WEA3NPPp20qHnumAEIlX3hY2LILJ0ZP/C+JZrSNNZ3Ib71jU8flOXf6+OGn8 X-Received: by 2002:a17:903:cd:b0:194:721e:611d with SMTP id x13-20020a17090300cd00b00194721e611dmr13437707plc.14.1674244525067; Fri, 20 Jan 2023 11:55:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674244525; cv=none; d=google.com; s=arc-20160816; b=DGm/E46orDJcJvYClTyNx9x/QuUHcE2NiA9SYpVEVN8hXwZUqhNFbVgUh/bCeZyl+8 hFDOuFOYygSnuo82F3kox1Xbj6fCAkaMSyL/xvOvYRLwmloCojv8vZnmsq4ttUmjfnIZ 3tmxARCHi6PRCEYQsHby8lAFcTFbtZ7DWghsSZpaqAa2UkcsLlo4jpzWN4RI3kCVdfeY SkxEFdGstrmY3i65S5In50h+WpblfIJp6PUe6yQIyO4lWccEqG3KwnKO3ZDPbAvJ7yTY YriHuudjib+zHFeypqEwCvA8ZQzFiUSwwl5dFAJ1zflTagw/I0K1D4dPE0WjtNpq0IZs mtJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=PjF0q+RzMHEXkyeoRAETD9SRq9N92UZpv3OIqZU5vxM=; b=lTh+IdKO7lNk1Y4ZvhO7iaxKMxPKe0omRp2gkyuwDzQEiBhYLU2LEeyoIANAvmKDNh 3X1YuN1tJ/lOhnxqIKwmw7TUEXlec4+KfBrmBwCJs5KMlZxYz7nC5immN4fo3IaODAcp 6zscZkykju8b1xw3AMA7hSaaWd+KsTu/6dsSjRyGUdxNU65fL09kSQNwfVmUXxKf4ia1 S8DVFAzqThNRvCqgQuU5TbFbhkgc9cvI+m4ALdOGkuCIFqdluqEMDjMFNqmttMWRRlR0 7NL5g/ism5ejaQaose4ccdh6JchNxgZEvQJ77tVxvcu8Orn0F/mkUDDRUD1gnFvvTUVN QrsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jMBjJPTK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o13-20020a170902d4cd00b00178b95aa01fsi26683858plg.614.2023.01.20.11.55.10; Fri, 20 Jan 2023 11:55:25 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jMBjJPTK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229893AbjATTwU (ORCPT <rfc822;forouhar.linux@gmail.com> + 99 others); Fri, 20 Jan 2023 14:52:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229645AbjATTwT (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Fri, 20 Jan 2023 14:52:19 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 543E8BB96; Fri, 20 Jan 2023 11:52:18 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E006D6206D; Fri, 20 Jan 2023 19:52:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A029CC433EF; Fri, 20 Jan 2023 19:52:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1674244337; bh=6jPk4/xopHMScMSF0b0LFfVr32gVbdEpBA+awgkZELo=; h=From:To:Cc:Subject:Date:From; b=jMBjJPTKclEPTTQKAp4VCZr3bgTnKC4OuMDd0uIzEIiB82PH2+ossDtRZRHOGTmVr L0+/iouxOH07R9zP7rBrptUhD82E0Yqn1EwrKjjcrmqWk3lIw+HQzvtQiUBPwNuleN icBgcQRsggoh0eqket2S67OQcpA672u0mEwOzr69Xc2vSv6zA+U7+cMPzG7kN78ALy dgzVEALHVFz34A1VdkHWuR71/SnXGu5Pn9wily6zYwh7v5PtX2MwqdJ+M7hNFghX2S 7hqunNs9hEV/rKaU33c6sPAGcpSKreZkDx2alyMN34/2urwFOavcqXaT23d8jpYzda BXJ7WXFUl+MUQ== From: Jeff Layton <jlayton@kernel.org> To: Chuck Lever <chuck.lever@oracle.com> Cc: Ruben Vestergaard <rubenv@drcmr.dk>, Torkil Svensgaard <torkil@drcmr.dk>, Shachar Kagan <skagan@nvidia.com>, linux-nfs@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH] nfsd: don't free files unconditionally in __nfsd_file_cache_purge Date: Fri, 20 Jan 2023 14:52:14 -0500 Message-Id: <20230120195215.42243-1-jlayton@kernel.org> X-Mailer: git-send-email 2.39.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755480544545383187?= X-GMAIL-MSGID: =?utf-8?q?1755572626928198647?= |
Series |
nfsd: don't free files unconditionally in __nfsd_file_cache_purge
|
|
Commit Message
Jeff Layton
Jan. 20, 2023, 7:52 p.m. UTC
nfsd_file_cache_purge is called when the server is shutting down, in
which case, tearing things down is generally fine, but it also gets
called when the exports cache is flushed.
Instead of walking the cache and freeing everything unconditionally,
handle it the same as when we have a notification of conflicting access.
Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache")
Reported-by: Ruben Vestergaard <rubenv@drcmr.dk>
Reported-by: Torkil Svensgaard <torkil@drcmr.dk>
Reported-by: Shachar Kagan <skagan@nvidia.com>
Signed-off-by: Jeff Layton <jlayton@kernel.org>
---
fs/nfsd/filecache.c | 61 +++++++++++++++++++++++++++------------------
1 file changed, 37 insertions(+), 24 deletions(-)
v2: use the same method to purge entries from the cache as we do when
there is a notification of conflicting access.
Comments
> On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: > > nfsd_file_cache_purge is called when the server is shutting down, in > which case, tearing things down is generally fine, but it also gets > called when the exports cache is flushed. Yeah... cache flush is probably the case we've been missing. > Instead of walking the cache and freeing everything unconditionally, > handle it the same as when we have a notification of conflicting access. > > Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") > Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> > Reported-by: Torkil Svensgaard <torkil@drcmr.dk> > Reported-by: Shachar Kagan <skagan@nvidia.com> > Signed-off-by: Jeff Layton <jlayton@kernel.org> > --- > fs/nfsd/filecache.c | 61 +++++++++++++++++++++++++++------------------ > 1 file changed, 37 insertions(+), 24 deletions(-) > > v2: use the same method to purge entries from the cache as we do when > there is a notification of conflicting access. > > diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c > index 58ac93e7e680..397ae212b98d 100644 > --- a/fs/nfsd/filecache.c > +++ b/fs/nfsd/filecache.c > @@ -661,6 +661,40 @@ static struct shrinker nfsd_file_shrinker = { > .seeks = 1, > }; > > +/** > + * maybe_queue_nfsd_file - try to unhash and queue a nfsd_file to be freed > + * @nf: nfsd_file to attempt to queue > + * @dispose: private list to queue successfully-put objects > + * > + * Unhash an nfsd_file, try to get a reference to it, and then put that > + * reference. If it's the last reference, queue it to the dispose list. > + * > + * The caller MUST hold the rcu_read_lock() ! __nfsd_file_cache_purge() isn't holding rcu_read_lock(), it's holding the nfsd_mutex. Is this comment incorrect, or is it just too specific? Or did I miss something obvious? > + */ > +static void > +maybe_queue_nfsd_file(struct nfsd_file *nf, struct list_head *dispose) I prefer the name nfsd_file_try_to_queue() or nfsd_file_try_to_dispose(). nfsd_file_ should be the prefix where possible. Unless you're redriving, I can fix that. > +{ > + int decrement = 1; > + > + /* If we raced with someone else unhashing, ignore it */ > + if (!nfsd_file_unhash(nf)) > + return; > + > + /* If we can't get a reference, ignore it */ > + if (!nfsd_file_get(nf)) > + return; > + > + /* Extra decrement if we remove from the LRU */ > + if (nfsd_file_lru_remove(nf)) > + ++decrement; > + > + /* If refcount goes to 0, then put on the dispose list */ > + if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > + list_add(&nf->nf_lru, dispose); > + trace_nfsd_file_closing(nf); > + } > +} > + > /** > * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode > * @inode: inode on which to close out nfsd_files > @@ -688,30 +722,12 @@ nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose) > > rcu_read_lock(); > do { > - int decrement = 1; > - > nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key, > nfsd_file_rhash_params); > if (!nf) > break; > > - /* If we raced with someone else unhashing, ignore it */ > - if (!nfsd_file_unhash(nf)) > - continue; > - > - /* If we can't get a reference, ignore it */ > - if (!nfsd_file_get(nf)) > - continue; > - > - /* Extra decrement if we remove from the LRU */ > - if (nfsd_file_lru_remove(nf)) > - ++decrement; > - > - /* If refcount goes to 0, then put on the dispose list */ > - if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > - list_add(&nf->nf_lru, dispose); > - trace_nfsd_file_closing(nf); > - } > + maybe_queue_nfsd_file(nf, dispose); > } while (1); > rcu_read_unlock(); > } > @@ -928,11 +944,8 @@ __nfsd_file_cache_purge(struct net *net) > > nf = rhashtable_walk_next(&iter); > while (!IS_ERR_OR_NULL(nf)) { > - if (!net || nf->nf_net == net) { > - nfsd_file_unhash(nf); > - nfsd_file_lru_remove(nf); > - list_add(&nf->nf_lru, &dispose); > - } > + if (!net || nf->nf_net == net) > + maybe_queue_nfsd_file(nf, &dispose); > nf = rhashtable_walk_next(&iter); > } > > -- > 2.39.0 > -- Chuck Lever
On Fri, 2023-01-20 at 20:21 +0000, Chuck Lever III wrote: > > > On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: > > > > nfsd_file_cache_purge is called when the server is shutting down, in > > which case, tearing things down is generally fine, but it also gets > > called when the exports cache is flushed. > > Yeah... cache flush is probably the case we've been missing. > > > > Instead of walking the cache and freeing everything unconditionally, > > handle it the same as when we have a notification of conflicting access. > > > > Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") > > Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> > > Reported-by: Torkil Svensgaard <torkil@drcmr.dk> > > Reported-by: Shachar Kagan <skagan@nvidia.com> > > Signed-off-by: Jeff Layton <jlayton@kernel.org> > > --- > > fs/nfsd/filecache.c | 61 +++++++++++++++++++++++++++------------------ > > 1 file changed, 37 insertions(+), 24 deletions(-) > > > > v2: use the same method to purge entries from the cache as we do when > > there is a notification of conflicting access. > > > > diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c > > index 58ac93e7e680..397ae212b98d 100644 > > --- a/fs/nfsd/filecache.c > > +++ b/fs/nfsd/filecache.c > > @@ -661,6 +661,40 @@ static struct shrinker nfsd_file_shrinker = { > > .seeks = 1, > > }; > > > > +/** > > + * maybe_queue_nfsd_file - try to unhash and queue a nfsd_file to be freed > > + * @nf: nfsd_file to attempt to queue > > + * @dispose: private list to queue successfully-put objects > > + * > > + * Unhash an nfsd_file, try to get a reference to it, and then put that > > + * reference. If it's the last reference, queue it to the dispose list. > > + * > > + * The caller MUST hold the rcu_read_lock() ! > > __nfsd_file_cache_purge() isn't holding rcu_read_lock(), it's > holding the nfsd_mutex. Is this comment incorrect, or is it just > too specific? Or did I miss something obvious? > It's implicitly taken by rhashtable_walk_start and released by rhashtable_walk_stop. FWIW, it'd be nice if there were a lockdep_assert_held equivalent for the rcu_read_lock() here, but I didn't see one of those. There is a rcu_read_lock_held(), but I didn't see a good way to get that to compile out when lockdep was disabled. > > > + */ > > +static void > > +maybe_queue_nfsd_file(struct nfsd_file *nf, struct list_head *dispose) > > I prefer the name nfsd_file_try_to_queue() or nfsd_file_try_to_dispose(). > nfsd_file_ should be the prefix where possible. Unless you're > redriving, I can fix that. > > Rename at will. > > +{ > > + int decrement = 1; > > + > > + /* If we raced with someone else unhashing, ignore it */ > > + if (!nfsd_file_unhash(nf)) > > + return; > > + > > + /* If we can't get a reference, ignore it */ > > + if (!nfsd_file_get(nf)) > > + return; > > + > > + /* Extra decrement if we remove from the LRU */ > > + if (nfsd_file_lru_remove(nf)) > > + ++decrement; > > + > > + /* If refcount goes to 0, then put on the dispose list */ > > + if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > > + list_add(&nf->nf_lru, dispose); > > + trace_nfsd_file_closing(nf); > > + } > > +} > > + > > /** > > * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode > > * @inode: inode on which to close out nfsd_files > > @@ -688,30 +722,12 @@ nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose) > > > > rcu_read_lock(); > > do { > > - int decrement = 1; > > - > > nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key, > > nfsd_file_rhash_params); > > if (!nf) > > break; > > > > - /* If we raced with someone else unhashing, ignore it */ > > - if (!nfsd_file_unhash(nf)) > > - continue; > > - > > - /* If we can't get a reference, ignore it */ > > - if (!nfsd_file_get(nf)) > > - continue; > > - > > - /* Extra decrement if we remove from the LRU */ > > - if (nfsd_file_lru_remove(nf)) > > - ++decrement; > > - > > - /* If refcount goes to 0, then put on the dispose list */ > > - if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > > - list_add(&nf->nf_lru, dispose); > > - trace_nfsd_file_closing(nf); > > - } > > + maybe_queue_nfsd_file(nf, dispose); > > } while (1); > > rcu_read_unlock(); > > } > > @@ -928,11 +944,8 @@ __nfsd_file_cache_purge(struct net *net) > > > > nf = rhashtable_walk_next(&iter); > > while (!IS_ERR_OR_NULL(nf)) { > > - if (!net || nf->nf_net == net) { > > - nfsd_file_unhash(nf); > > - nfsd_file_lru_remove(nf); > > - list_add(&nf->nf_lru, &dispose); > > - } > > + if (!net || nf->nf_net == net) > > + maybe_queue_nfsd_file(nf, &dispose); > > nf = rhashtable_walk_next(&iter); > > } > > > > -- > > 2.39.0 > > > > -- > Chuck Lever > > >
> On Jan 20, 2023, at 3:54 PM, Jeff Layton <jlayton@kernel.org> wrote: > > On Fri, 2023-01-20 at 20:21 +0000, Chuck Lever III wrote: >> >>> On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: >>> >>> nfsd_file_cache_purge is called when the server is shutting down, in >>> which case, tearing things down is generally fine, but it also gets >>> called when the exports cache is flushed. >> >> Yeah... cache flush is probably the case we've been missing. >> >> >>> Instead of walking the cache and freeing everything unconditionally, >>> handle it the same as when we have a notification of conflicting access. >>> >>> Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") >>> Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> >>> Reported-by: Torkil Svensgaard <torkil@drcmr.dk> >>> Reported-by: Shachar Kagan <skagan@nvidia.com> >>> Signed-off-by: Jeff Layton <jlayton@kernel.org> >>> --- >>> fs/nfsd/filecache.c | 61 +++++++++++++++++++++++++++------------------ >>> 1 file changed, 37 insertions(+), 24 deletions(-) >>> >>> v2: use the same method to purge entries from the cache as we do when >>> there is a notification of conflicting access. >>> >>> diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c >>> index 58ac93e7e680..397ae212b98d 100644 >>> --- a/fs/nfsd/filecache.c >>> +++ b/fs/nfsd/filecache.c >>> @@ -661,6 +661,40 @@ static struct shrinker nfsd_file_shrinker = { >>> .seeks = 1, >>> }; >>> >>> +/** >>> + * maybe_queue_nfsd_file - try to unhash and queue a nfsd_file to be freed >>> + * @nf: nfsd_file to attempt to queue >>> + * @dispose: private list to queue successfully-put objects >>> + * >>> + * Unhash an nfsd_file, try to get a reference to it, and then put that >>> + * reference. If it's the last reference, queue it to the dispose list. >>> + * >>> + * The caller MUST hold the rcu_read_lock() ! >> >> __nfsd_file_cache_purge() isn't holding rcu_read_lock(), it's >> holding the nfsd_mutex. Is this comment incorrect, or is it just >> too specific? Or did I miss something obvious? >> > > It's implicitly taken by rhashtable_walk_start and released by > rhashtable_walk_stop. > > FWIW, it'd be nice if there were a lockdep_assert_held equivalent for > the rcu_read_lock() here, but I didn't see one of those. Would static void nfsd_file_cond_queue(struct nfsd_file *nf, struct list_head *dispose) __must_hold(RCU) { do the trick? > There is a > rcu_read_lock_held(), but I didn't see a good way to get that to compile > out when lockdep was disabled. > >>> + */ >>> +static void >>> +maybe_queue_nfsd_file(struct nfsd_file *nf, struct list_head *dispose) >> >> I prefer the name nfsd_file_try_to_queue() or nfsd_file_try_to_dispose(). >> nfsd_file_ should be the prefix where possible. Unless you're >> redriving, I can fix that. >> >> > > Rename at will. Done and applied to nfsd-fixes, replacing v1 of this patch. >>> +{ >>> + int decrement = 1; >>> + >>> + /* If we raced with someone else unhashing, ignore it */ >>> + if (!nfsd_file_unhash(nf)) >>> + return; >>> + >>> + /* If we can't get a reference, ignore it */ >>> + if (!nfsd_file_get(nf)) >>> + return; >>> + >>> + /* Extra decrement if we remove from the LRU */ >>> + if (nfsd_file_lru_remove(nf)) >>> + ++decrement; >>> + >>> + /* If refcount goes to 0, then put on the dispose list */ >>> + if (refcount_sub_and_test(decrement, &nf->nf_ref)) { >>> + list_add(&nf->nf_lru, dispose); >>> + trace_nfsd_file_closing(nf); >>> + } >>> +} >>> + >>> /** >>> * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode >>> * @inode: inode on which to close out nfsd_files >>> @@ -688,30 +722,12 @@ nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose) >>> >>> rcu_read_lock(); >>> do { >>> - int decrement = 1; >>> - >>> nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key, >>> nfsd_file_rhash_params); >>> if (!nf) >>> break; >>> >>> - /* If we raced with someone else unhashing, ignore it */ >>> - if (!nfsd_file_unhash(nf)) >>> - continue; >>> - >>> - /* If we can't get a reference, ignore it */ >>> - if (!nfsd_file_get(nf)) >>> - continue; >>> - >>> - /* Extra decrement if we remove from the LRU */ >>> - if (nfsd_file_lru_remove(nf)) >>> - ++decrement; >>> - >>> - /* If refcount goes to 0, then put on the dispose list */ >>> - if (refcount_sub_and_test(decrement, &nf->nf_ref)) { >>> - list_add(&nf->nf_lru, dispose); >>> - trace_nfsd_file_closing(nf); >>> - } >>> + maybe_queue_nfsd_file(nf, dispose); >>> } while (1); >>> rcu_read_unlock(); >>> } >>> @@ -928,11 +944,8 @@ __nfsd_file_cache_purge(struct net *net) >>> >>> nf = rhashtable_walk_next(&iter); >>> while (!IS_ERR_OR_NULL(nf)) { >>> - if (!net || nf->nf_net == net) { >>> - nfsd_file_unhash(nf); >>> - nfsd_file_lru_remove(nf); >>> - list_add(&nf->nf_lru, &dispose); >>> - } >>> + if (!net || nf->nf_net == net) >>> + maybe_queue_nfsd_file(nf, &dispose); >>> nf = rhashtable_walk_next(&iter); >>> } >>> >>> -- >>> 2.39.0 >>> >> >> -- >> Chuck Lever >> >> >> > > -- > Jeff Layton <jlayton@kernel.org> -- Chuck Lever
On Sat, 2023-01-21 at 17:04 +0000, Chuck Lever III wrote: > > > On Jan 20, 2023, at 3:54 PM, Jeff Layton <jlayton@kernel.org> wrote: > > > > On Fri, 2023-01-20 at 20:21 +0000, Chuck Lever III wrote: > > > > > > > On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > nfsd_file_cache_purge is called when the server is shutting down, in > > > > which case, tearing things down is generally fine, but it also gets > > > > called when the exports cache is flushed. > > > > > > Yeah... cache flush is probably the case we've been missing. > > > > > > > > > > Instead of walking the cache and freeing everything unconditionally, > > > > handle it the same as when we have a notification of conflicting access. > > > > > > > > Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") > > > > Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> > > > > Reported-by: Torkil Svensgaard <torkil@drcmr.dk> > > > > Reported-by: Shachar Kagan <skagan@nvidia.com> > > > > Signed-off-by: Jeff Layton <jlayton@kernel.org> > > > > --- > > > > fs/nfsd/filecache.c | 61 +++++++++++++++++++++++++++------------------ > > > > 1 file changed, 37 insertions(+), 24 deletions(-) > > > > > > > > v2: use the same method to purge entries from the cache as we do when > > > > there is a notification of conflicting access. > > > > > > > > diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c > > > > index 58ac93e7e680..397ae212b98d 100644 > > > > --- a/fs/nfsd/filecache.c > > > > +++ b/fs/nfsd/filecache.c > > > > @@ -661,6 +661,40 @@ static struct shrinker nfsd_file_shrinker = { > > > > .seeks = 1, > > > > }; > > > > > > > > +/** > > > > + * maybe_queue_nfsd_file - try to unhash and queue a nfsd_file to be freed > > > > + * @nf: nfsd_file to attempt to queue > > > > + * @dispose: private list to queue successfully-put objects > > > > + * > > > > + * Unhash an nfsd_file, try to get a reference to it, and then put that > > > > + * reference. If it's the last reference, queue it to the dispose list. > > > > + * > > > > + * The caller MUST hold the rcu_read_lock() ! > > > > > > __nfsd_file_cache_purge() isn't holding rcu_read_lock(), it's > > > holding the nfsd_mutex. Is this comment incorrect, or is it just > > > too specific? Or did I miss something obvious? > > > > > > > It's implicitly taken by rhashtable_walk_start and released by > > rhashtable_walk_stop. > > > > FWIW, it'd be nice if there were a lockdep_assert_held equivalent for > > the rcu_read_lock() here, but I didn't see one of those. > > Would > > static void > nfsd_file_cond_queue(struct nfsd_file *nf, struct list_head *dispose) > __must_hold(RCU) > { > > do the trick? > Seems like a reasonable thing to add. > > > There is a > > rcu_read_lock_held(), but I didn't see a good way to get that to compile > > out when lockdep was disabled. > > > > > > + */ > > > > +static void > > > > +maybe_queue_nfsd_file(struct nfsd_file *nf, struct list_head *dispose) > > > > > > I prefer the name nfsd_file_try_to_queue() or nfsd_file_try_to_dispose(). > > > nfsd_file_ should be the prefix where possible. Unless you're > > > redriving, I can fix that. > > > > > > > > > > Rename at will. > > Done and applied to nfsd-fixes, replacing v1 of this patch. > LGTM. Thanks! > > > > > +{ > > > > + int decrement = 1; > > > > + > > > > + /* If we raced with someone else unhashing, ignore it */ > > > > + if (!nfsd_file_unhash(nf)) > > > > + return; > > > > + > > > > + /* If we can't get a reference, ignore it */ > > > > + if (!nfsd_file_get(nf)) > > > > + return; > > > > + > > > > + /* Extra decrement if we remove from the LRU */ > > > > + if (nfsd_file_lru_remove(nf)) > > > > + ++decrement; > > > > + > > > > + /* If refcount goes to 0, then put on the dispose list */ > > > > + if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > > > > + list_add(&nf->nf_lru, dispose); > > > > + trace_nfsd_file_closing(nf); > > > > + } > > > > +} > > > > + > > > > /** > > > > * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode > > > > * @inode: inode on which to close out nfsd_files > > > > @@ -688,30 +722,12 @@ nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose) > > > > > > > > rcu_read_lock(); > > > > do { > > > > - int decrement = 1; > > > > - > > > > nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key, > > > > nfsd_file_rhash_params); > > > > if (!nf) > > > > break; > > > > > > > > - /* If we raced with someone else unhashing, ignore it */ > > > > - if (!nfsd_file_unhash(nf)) > > > > - continue; > > > > - > > > > - /* If we can't get a reference, ignore it */ > > > > - if (!nfsd_file_get(nf)) > > > > - continue; > > > > - > > > > - /* Extra decrement if we remove from the LRU */ > > > > - if (nfsd_file_lru_remove(nf)) > > > > - ++decrement; > > > > - > > > > - /* If refcount goes to 0, then put on the dispose list */ > > > > - if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > > > > - list_add(&nf->nf_lru, dispose); > > > > - trace_nfsd_file_closing(nf); > > > > - } > > > > + maybe_queue_nfsd_file(nf, dispose); > > > > } while (1); > > > > rcu_read_unlock(); > > > > } > > > > @@ -928,11 +944,8 @@ __nfsd_file_cache_purge(struct net *net) > > > > > > > > nf = rhashtable_walk_next(&iter); > > > > while (!IS_ERR_OR_NULL(nf)) { > > > > - if (!net || nf->nf_net == net) { > > > > - nfsd_file_unhash(nf); > > > > - nfsd_file_lru_remove(nf); > > > > - list_add(&nf->nf_lru, &dispose); > > > > - } > > > > + if (!net || nf->nf_net == net) > > > > + maybe_queue_nfsd_file(nf, &dispose); > > > > nf = rhashtable_walk_next(&iter); > > > > } > > > > > > > > -- > > > > 2.39.0 > > > > > > > > > > -- > > > Chuck Lever > > > > > > > > > > > > > -- > > Jeff Layton <jlayton@kernel.org> > > -- > Chuck Lever > > >
On Sun, 2023-01-22 at 16:58 +0000, Shachar Kagan wrote: > On Sat, 2023-01-21 at 17:04 +0000, Chuck Lever III wrote: > > > > > On Jan 20, 2023, at 3:54 PM, Jeff Layton <jlayton@kernel.org> wrote: > > > > > > On Fri, 2023-01-20 at 20:21 +0000, Chuck Lever III wrote: > > > > > > > > > On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: > > > > > > > > > > nfsd_file_cache_purge is called when the server is shutting > > > > > down, in which case, tearing things down is generally fine, but > > > > > it also gets called when the exports cache is flushed. > > > > > > > > Yeah... cache flush is probably the case we've been missing. > > > > > > > > > > > > > Instead of walking the cache and freeing everything > > > > > unconditionally, handle it the same as when we have a notification of conflicting access. > > > > > > > > > > Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") > > > > > Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> > > > > > Reported-by: Torkil Svensgaard <torkil@drcmr.dk> > > > > > Reported-by: Shachar Kagan <skagan@nvidia.com> > > > > > Signed-off-by: Jeff Layton <jlayton@kernel.org> I ran our tests with the commit, the issue didn't occur. Thanks for the quick response. > > > > > --- > > > > > fs/nfsd/filecache.c | 61 > > > > > +++++++++++++++++++++++++++------------------ > > > > > 1 file changed, 37 insertions(+), 24 deletions(-) > > > > > > > > > > v2: use the same method to purge entries from the cache as we do when > > > > > there is a notification of conflicting access. > > > > > > > > > > diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index > > > > > 58ac93e7e680..397ae212b98d 100644 > > > > > --- a/fs/nfsd/filecache.c > > > > > +++ b/fs/nfsd/filecache.c > > > > > @@ -661,6 +661,40 @@ static struct shrinker nfsd_file_shrinker = { > > > > > .seeks = 1, > > > > > }; > > > > > > > > > > +/** > > > > > + * maybe_queue_nfsd_file - try to unhash and queue a nfsd_file > > > > > +to be freed > > > > > + * @nf: nfsd_file to attempt to queue > > > > > + * @dispose: private list to queue successfully-put objects > > > > > + * > > > > > + * Unhash an nfsd_file, try to get a reference to it, and then > > > > > +put that > > > > > + * reference. If it's the last reference, queue it to the dispose list. > > > > > + * > > > > > + * The caller MUST hold the rcu_read_lock() ! > > > > > > > > __nfsd_file_cache_purge() isn't holding rcu_read_lock(), it's > > > > holding the nfsd_mutex. Is this comment incorrect, or is it just > > > > too specific? Or did I miss something obvious? > > > > > > > > > > It's implicitly taken by rhashtable_walk_start and released by > > > rhashtable_walk_stop. > > > > > > FWIW, it'd be nice if there were a lockdep_assert_held equivalent > > > for the rcu_read_lock() here, but I didn't see one of those. > > > > Would > > > > static void > > nfsd_file_cond_queue(struct nfsd_file *nf, struct list_head *dispose) > > __must_hold(RCU) > > { > > > > do the trick? > > > > Seems like a reasonable thing to add. > > > > > > There is a > > > rcu_read_lock_held(), but I didn't see a good way to get that to > > > compile out when lockdep was disabled. > > > > > > > > + */ > > > > > +static void > > > > > +maybe_queue_nfsd_file(struct nfsd_file *nf, struct list_head > > > > > +*dispose) > > > > > > > > I prefer the name nfsd_file_try_to_queue() or nfsd_file_try_to_dispose(). > > > > nfsd_file_ should be the prefix where possible. Unless you're > > > > redriving, I can fix that. > > > > > > > > > > > > > > Rename at will. > > > > Done and applied to nfsd-fixes, replacing v1 of this patch. > > > > LGTM. Thanks! > > > > > > > > +{ > > > > > + int decrement = 1; > > > > > + > > > > > + /* If we raced with someone else unhashing, ignore it */ > > > > > + if (!nfsd_file_unhash(nf)) > > > > > + return; > > > > > + > > > > > + /* If we can't get a reference, ignore it */ > > > > > + if (!nfsd_file_get(nf)) > > > > > + return; > > > > > + > > > > > + /* Extra decrement if we remove from the LRU */ > > > > > + if (nfsd_file_lru_remove(nf)) > > > > > + ++decrement; > > > > > + > > > > > + /* If refcount goes to 0, then put on the dispose list */ > > > > > + if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > > > > > + list_add(&nf->nf_lru, dispose); > > > > > + trace_nfsd_file_closing(nf); > > > > > + } > > > > > +} > > > > > + > > > > > /** > > > > > * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode > > > > > * @inode: inode on which to close out nfsd_files > > > > > @@ -688,30 +722,12 @@ nfsd_file_queue_for_close(struct inode > > > > > *inode, struct list_head *dispose) > > > > > > > > > > rcu_read_lock(); > > > > > do { > > > > > - int decrement = 1; > > > > > - > > > > > nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key, > > > > > nfsd_file_rhash_params); > > > > > if (!nf) > > > > > break; > > > > > > > > > > - /* If we raced with someone else unhashing, ignore it */ > > > > > - if (!nfsd_file_unhash(nf)) > > > > > - continue; > > > > > - > > > > > - /* If we can't get a reference, ignore it */ > > > > > - if (!nfsd_file_get(nf)) > > > > > - continue; > > > > > - > > > > > - /* Extra decrement if we remove from the LRU */ > > > > > - if (nfsd_file_lru_remove(nf)) > > > > > - ++decrement; > > > > > - > > > > > - /* If refcount goes to 0, then put on the dispose list */ > > > > > - if (refcount_sub_and_test(decrement, &nf->nf_ref)) { > > > > > - list_add(&nf->nf_lru, dispose); > > > > > - trace_nfsd_file_closing(nf); > > > > > - } > > > > > + maybe_queue_nfsd_file(nf, dispose); > > > > > } while (1); > > > > > rcu_read_unlock(); > > > > > } > > > > > @@ -928,11 +944,8 @@ __nfsd_file_cache_purge(struct net *net) > > > > > > > > > > nf = rhashtable_walk_next(&iter); > > > > > while (!IS_ERR_OR_NULL(nf)) { > > > > > - if (!net || nf->nf_net == net) { > > > > > - nfsd_file_unhash(nf); > > > > > - nfsd_file_lru_remove(nf); > > > > > - list_add(&nf->nf_lru, &dispose); > > > > > - } > > > > > + if (!net || nf->nf_net == net) > > > > > + maybe_queue_nfsd_file(nf, > > > > > + &dispose); > > > > > nf = rhashtable_walk_next(&iter); > > > > > } > > > > > > > > > > -- > > > > > 2.39.0 > > > > > > > > > > > > > -- > > > > Chuck Lever > > > > > > > > > > > > > > > > > > -- > > > Jeff Layton <jlayton@kernel.org> > > > > -- > > Chuck Lever > > > > > > > > -- > Jeff Layton <jlayton@kernel.org>
> On Jan 22, 2023, at 9:58 AM, Shachar Kagan <skagan@nvidia.com> wrote: > > > On Sun, 2023-01-22 at 16:58 +0000, Shachar Kagan wrote: > >> On Sat, 2023-01-21 at 17:04 +0000, Chuck Lever III wrote: >>> >>>> On Jan 20, 2023, at 3:54 PM, Jeff Layton <jlayton@kernel.org> wrote: >>>> >>>> On Fri, 2023-01-20 at 20:21 +0000, Chuck Lever III wrote: >>>>> >>>>>> On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: >>>>>> >>>>>> nfsd_file_cache_purge is called when the server is shutting >>>>>> down, in which case, tearing things down is generally fine, but >>>>>> it also gets called when the exports cache is flushed. >>>>> >>>>> Yeah... cache flush is probably the case we've been missing. >>>>> >>>>> >>>>>> Instead of walking the cache and freeing everything >>>>>> unconditionally, handle it the same as when we have a notification of conflicting access. >>>>>> >>>>>> Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") >>>>>> Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> >>>>>> Reported-by: Torkil Svensgaard <torkil@drcmr.dk> >>>>>> Reported-by: Shachar Kagan <skagan@nvidia.com> >>>>>> Signed-off-by: Jeff Layton <jlayton@kernel.org> > > I ran our tests with the commit, the issue didn't occur. > Thanks for the quick response. May I add Tested-by: Shachar Kagan <skagan@nvidia.com> ? -- Chuck Lever
>> On Jan 22, 2023, at 9:58 AM, Shachar Kagan <skagan@nvidia.com> wrote: >> >>> On Sat, 2023-01-21 at 17:04 +0000, Chuck Lever III wrote: >>>> >>>>> On Jan 20, 2023, at 3:54 PM, Jeff Layton <jlayton@kernel.org> wrote: >>>>> >>>>> On Fri, 2023-01-20 at 20:21 +0000, Chuck Lever III wrote: >>>>>> >>>>>>> On Jan 20, 2023, at 2:52 PM, Jeff Layton <jlayton@kernel.org> wrote: >>>>>>> >>>>>>> nfsd_file_cache_purge is called when the server is shutting down, >>>>>>> in which case, tearing things down is generally fine, but it also >>>>>>> gets called when the exports cache is flushed. >>>>>> >>>>>> Yeah... cache flush is probably the case we've been missing. >>>>>> >>>>>> >>>>>>> Instead of walking the cache and freeing everything >>>>>>> unconditionally, handle it the same as when we have a notification of conflicting access. >>>>>>> >>>>>>> Fixes: ac3a2585f018 ("nfsd: rework refcounting in filecache") >>>>>>> Reported-by: Ruben Vestergaard <rubenv@drcmr.dk> >>>>>>> Reported-by: Torkil Svensgaard <torkil@drcmr.dk> >>>>>>> Reported-by: Shachar Kagan <skagan@nvidia.com> >>>>>>> Signed-off-by: Jeff Layton <jlayton@kernel.org> >> >> I ran our tests with the commit, the issue didn't occur. >> Thanks for the quick response. > > May I add Tested-by: Shachar Kagan <skagan@nvidia.com> ? > Yes > -- > Chuck Lever -- Shachar Kagan
diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 58ac93e7e680..397ae212b98d 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -661,6 +661,40 @@ static struct shrinker nfsd_file_shrinker = { .seeks = 1, }; +/** + * maybe_queue_nfsd_file - try to unhash and queue a nfsd_file to be freed + * @nf: nfsd_file to attempt to queue + * @dispose: private list to queue successfully-put objects + * + * Unhash an nfsd_file, try to get a reference to it, and then put that + * reference. If it's the last reference, queue it to the dispose list. + * + * The caller MUST hold the rcu_read_lock() ! + */ +static void +maybe_queue_nfsd_file(struct nfsd_file *nf, struct list_head *dispose) +{ + int decrement = 1; + + /* If we raced with someone else unhashing, ignore it */ + if (!nfsd_file_unhash(nf)) + return; + + /* If we can't get a reference, ignore it */ + if (!nfsd_file_get(nf)) + return; + + /* Extra decrement if we remove from the LRU */ + if (nfsd_file_lru_remove(nf)) + ++decrement; + + /* If refcount goes to 0, then put on the dispose list */ + if (refcount_sub_and_test(decrement, &nf->nf_ref)) { + list_add(&nf->nf_lru, dispose); + trace_nfsd_file_closing(nf); + } +} + /** * nfsd_file_queue_for_close: try to close out any open nfsd_files for an inode * @inode: inode on which to close out nfsd_files @@ -688,30 +722,12 @@ nfsd_file_queue_for_close(struct inode *inode, struct list_head *dispose) rcu_read_lock(); do { - int decrement = 1; - nf = rhashtable_lookup(&nfsd_file_rhash_tbl, &key, nfsd_file_rhash_params); if (!nf) break; - /* If we raced with someone else unhashing, ignore it */ - if (!nfsd_file_unhash(nf)) - continue; - - /* If we can't get a reference, ignore it */ - if (!nfsd_file_get(nf)) - continue; - - /* Extra decrement if we remove from the LRU */ - if (nfsd_file_lru_remove(nf)) - ++decrement; - - /* If refcount goes to 0, then put on the dispose list */ - if (refcount_sub_and_test(decrement, &nf->nf_ref)) { - list_add(&nf->nf_lru, dispose); - trace_nfsd_file_closing(nf); - } + maybe_queue_nfsd_file(nf, dispose); } while (1); rcu_read_unlock(); } @@ -928,11 +944,8 @@ __nfsd_file_cache_purge(struct net *net) nf = rhashtable_walk_next(&iter); while (!IS_ERR_OR_NULL(nf)) { - if (!net || nf->nf_net == net) { - nfsd_file_unhash(nf); - nfsd_file_lru_remove(nf); - list_add(&nf->nf_lru, &dispose); - } + if (!net || nf->nf_net == net) + maybe_queue_nfsd_file(nf, &dispose); nf = rhashtable_walk_next(&iter); }