From patchwork Fri Dec 8 03:27:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 175557 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5222860vqy; Thu, 7 Dec 2023 19:39:58 -0800 (PST) X-Google-Smtp-Source: AGHT+IE+2pogBkM/pmrdMO1bpeQliC6+AISr3rsAjSL2+fvP34rMO4/UXfJJNRVCTKXEyuEL2D3U X-Received: by 2002:a05:6a00:10cb:b0:6ce:379f:5ea0 with SMTP id d11-20020a056a0010cb00b006ce379f5ea0mr315030pfu.1.1702006798596; Thu, 07 Dec 2023 19:39:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702006798; cv=none; d=google.com; s=arc-20160816; b=AeXLrTsBLEUFn4TRe0Gdr6DNABNgnonxm2cKWqKQRTeHzQQ01XSswqmLoBPTPkmnkt U2IfjafTxddRP8O08arNAS/ym+1cAFEs5sZqSCkzvm06IN0/hYJmiJaXuEUdH3NufDZm bJS4F0Fa1ijyieLO0eKpyKWVBhVg6Fy9U+EuQFk28FsKYBOCBi0YDkTCG5ms0MAe8VHI heAEsN7WC6T7a2CeGq+4mYgzNCmwOnz4aVcQ/w1QYVg2wvOGXOlWVGVr4J+MAyKpOGIZ 9GY8njnXnh3mJcX+nQaN1JtmarXciztjX2XH/h19LV3XDCmpL+C+Qx6PRIFVU37Mj1ha J0jQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=Vu248dzuKJlGI9r2nLRgAmjct9Ps79FDgE6mGoaCwWU=; fh=I45k8M33HW2Wqj1OlgncPyvUt41clxPTDAFs+9Wisus=; b=AVVX87Oi9D988TXV61m8DoOIMT2wVZqf8l+RbdnygjRUJ/CGjCFPJ4R+RQWlP7Bi6i UBZeRCLcnWFy26COFK9AU82ZA1A4Qv/0RUrBXjUcVKodbJZRfj9F5oz3BfRg+MRUd9Ih QPp1jAgp0rwTB/mGQgzVQzdcE0q3b4efbTCHiGdEr7ZFnv9tADMSmDMamlCTHIgIUQBm rQLnoSoEQMFg+FPGVhc+DmOHn60gJ4LXlAkRW7RM0qBhaT8/SYKvwJ+yfDJBJEew3+Cr zV7AEkiUiPhKnDw9M2CFqBmJ9+WCmbvFepQcLOrLPTR6OgWZXw39580gHMRyHW/4+MuE B+3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=anmxB+fk; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=RVlBKXdB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id q12-20020a056a00088c00b006cddc343ceesi808507pfj.181.2023.12.07.19.39.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 19:39:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=anmxB+fk; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=RVlBKXdB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 9AEF38087E0E; Thu, 7 Dec 2023 19:39:52 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231892AbjLHDjn (ORCPT + 99 others); Thu, 7 Dec 2023 22:39:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229531AbjLHDjn (ORCPT ); Thu, 7 Dec 2023 22:39:43 -0500 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21E8D10E6; Thu, 7 Dec 2023 19:39:49 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id AD09022102; Fri, 8 Dec 2023 03:39:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1702006787; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vu248dzuKJlGI9r2nLRgAmjct9Ps79FDgE6mGoaCwWU=; b=anmxB+fkH2fqhNie3MRW+XjNo0iXCABg60BSQw5Gsayu+iRy7OkPLqu3TWa1mTc05JFt4b PB64CHduam7aN+CxqIAk22QBDh8iM9NqskeG90Tsq/T3SdNhJ7iBncCsTGKBzfB+NjIlQb YV+pMG2fSEtB7Ejcs7hIWeF6ddVnbHY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1702006787; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vu248dzuKJlGI9r2nLRgAmjct9Ps79FDgE6mGoaCwWU=; b=RVlBKXdBjGA2gqTyBfMjuQ3JTo5Idgr/yQkZVdm/OFHxFJVi1kDjDn+oWJNh1MzyNl0vx4 h92N1tS4/CTf9XCQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 1DB3C13725; Fri, 8 Dec 2023 03:30:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id h/XQKNmNcmX+MAAAD6G6ig (envelope-from ); Fri, 08 Dec 2023 03:30:33 +0000 From: NeilBrown To: Al Viro , Christian Brauner , Jens Axboe , Oleg Nesterov , Chuck Lever , Jeff Layton Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH 1/3] nfsd: use __fput_sync() to avoid delayed closing of files. Date: Fri, 8 Dec 2023 14:27:26 +1100 Message-ID: <20231208033006.5546-2-neilb@suse.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231208033006.5546-1-neilb@suse.de> References: <20231208033006.5546-1-neilb@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [10.00 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; BAYES_SPAM(5.10)[100.00%]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_SEVEN(0.00)[9]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[] Authentication-Results: smtp-out1.suse.de; none X-Spam-Score: 10.00 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 07 Dec 2023 19:39:53 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784683480706825206 X-GMAIL-MSGID: 1784683480706825206 Calling fput() directly or though filp_close() from a kernel thread like nfsd causes the final __fput() (if necessary) to be called from a workqueue. This means that nfsd is not forced to wait for any work to complete. If the ->release of ->destroy_inode function is slow for any reason, this can result in nfsd closing files more quickly than the workqueue can complete the close and the queue of pending closes can grow without bounces (30 million has been seen at one customer site, though this was in part due to a slowness in xfs which has since been fixed). nfsd does not need this. This quite appropriate and safe for nfsd to do its own close work. There is now reason that close should ever wait for nfsd, so no deadlock can occur. So change all fput() calls to __fput_sync(), and convert filp_close() to the sequence get_file();filp_close();__fput_sync(). This ensure that no fput work is queued to the workqueue. Note that this removes the only in-module use of flush_fput_queue(). Signed-off-by: NeilBrown --- fs/nfsd/filecache.c | 3 ++- fs/nfsd/lockd.c | 2 +- fs/nfsd/nfs4proc.c | 4 ++-- fs/nfsd/nfs4recover.c | 2 +- fs/nfsd/vfs.c | 12 ++++++------ 5 files changed, 12 insertions(+), 11 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index ef063f93fde9..e9734c7451b5 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -283,7 +283,9 @@ nfsd_file_free(struct nfsd_file *nf) nfsd_file_mark_put(nf->nf_mark); if (nf->nf_file) { nfsd_file_check_write_error(nf); + get_file(nf->nf_file); filp_close(nf->nf_file, NULL); + __fput_sync(nf->nf_file); } /* @@ -631,7 +633,6 @@ nfsd_file_close_inode_sync(struct inode *inode) list_del_init(&nf->nf_lru); nfsd_file_free(nf); } - flush_delayed_fput(); } /** diff --git a/fs/nfsd/lockd.c b/fs/nfsd/lockd.c index 46a7f9b813e5..f9d1059096a4 100644 --- a/fs/nfsd/lockd.c +++ b/fs/nfsd/lockd.c @@ -60,7 +60,7 @@ nlm_fopen(struct svc_rqst *rqstp, struct nfs_fh *f, struct file **filp, static void nlm_fclose(struct file *filp) { - fput(filp); + __fput_sync(filp); } static const struct nlmsvc_binding nfsd_nlm_ops = { diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c index 6f2d4aa4970d..20d60823d530 100644 --- a/fs/nfsd/nfs4proc.c +++ b/fs/nfsd/nfs4proc.c @@ -629,7 +629,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, nn->somebody_reclaimed = true; out: if (open->op_filp) { - fput(open->op_filp); + __fput_sync(open->op_filp); open->op_filp = NULL; } if (resfh && resfh != &cstate->current_fh) { @@ -1546,7 +1546,7 @@ nfsd4_cleanup_inter_ssc(struct nfsd4_ssc_umount_item *nsui, struct file *filp, long timeout = msecs_to_jiffies(nfsd4_ssc_umount_timeout); nfs42_ssc_close(filp); - fput(filp); + __fput_sync(filp); spin_lock(&nn->nfsd_ssc_lock); list_del(&nsui->nsui_list); diff --git a/fs/nfsd/nfs4recover.c b/fs/nfsd/nfs4recover.c index 3509e73abe1f..f8f0112fd9f5 100644 --- a/fs/nfsd/nfs4recover.c +++ b/fs/nfsd/nfs4recover.c @@ -561,7 +561,7 @@ nfsd4_shutdown_recdir(struct net *net) if (!nn->rec_file) return; - fput(nn->rec_file); + __fput_sync(nn->rec_file); nn->rec_file = NULL; } diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c index fbbea7498f02..15a811229211 100644 --- a/fs/nfsd/vfs.c +++ b/fs/nfsd/vfs.c @@ -879,7 +879,7 @@ __nfsd_open(struct svc_rqst *rqstp, struct svc_fh *fhp, umode_t type, host_err = ima_file_check(file, may_flags); if (host_err) { - fput(file); + __fput_sync(file); goto out; } @@ -1884,10 +1884,10 @@ nfsd_rename(struct svc_rqst *rqstp, struct svc_fh *ffhp, char *fname, int flen, fh_drop_write(ffhp); /* - * If the target dentry has cached open files, then we need to try to - * close them prior to doing the rename. Flushing delayed fput - * shouldn't be done with locks held however, so we delay it until this - * point and then reattempt the whole shebang. + * If the target dentry has cached open files, then we need to + * try to close them prior to doing the rename. Final fput + * shouldn't be done with locks held however, so we delay it + * until this point and then reattempt the whole shebang. */ if (close_cached) { close_cached = false; @@ -2141,7 +2141,7 @@ nfsd_readdir(struct svc_rqst *rqstp, struct svc_fh *fhp, loff_t *offsetp, if (err == nfserr_eof || err == nfserr_toosmall) err = nfs_ok; /* can still be found in ->err */ out_close: - fput(file); + __fput_sync(file); out: return err; } From patchwork Fri Dec 8 03:27:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 175551 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5220126vqy; Thu, 7 Dec 2023 19:31:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IEH/KV7Zwnv3D/8ZKyssg7UsWaS9nHyXVUmVVtZ5QNWf0e+GEuIJr4daIg5duTdACUkrbQ8 X-Received: by 2002:a05:6a20:42a3:b0:18f:97c:614c with SMTP id o35-20020a056a2042a300b0018f097c614cmr4155963pzj.73.1702006273300; Thu, 07 Dec 2023 19:31:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702006273; cv=none; d=google.com; s=arc-20160816; b=eUEx7xOKkNA3fNy9EIxapEaFRun+SHoJaifHEFiCfrsk9vsDyHQ2EysQBubtgHjURJ BeXIu2dMcyx9Oko/AyLV9Qsg1Zmun7BaoXAyAMfVi6irQCE1ev0pnrkUOfyd2/iB7Ag6 qlRNOhPVWABY0PDSBlNd7U5mVf5JLfRhg6nM03M+LiXM/aqAvTyIaE9AL/5R0bzV+mgU 5ENWF3eEo/SC2iKP2wtKNawwLJiikWYYkPGOgNTgfiTjwPEt60DO9iufhwmJ+7p5H/aZ rA8BcfazDD2ky9VXy27ccYkii0yJwxztKHkZQhuOLYbDedrZK42uF/AWonwy/Iopxm4A ToSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature:dkim-signature:dkim-signature; bh=FPbjaalRYKm9QivlQuryrahlBhRTkM5VSSKDElwEPLI=; fh=I45k8M33HW2Wqj1OlgncPyvUt41clxPTDAFs+9Wisus=; b=ESHWhelOw3MDTckacDOGOccdzY3k10WXuup9PImBGAH/Z4BH+cm1xvCyTxw4heuEXo 6PplM9pz3hH+yyPTqhBUzZo2MVR78EzHIrnvafKLGqb/2Zf/psDMhuvzwLp3jwy5LdoZ nFXHhHmd28WGiqeAjCB4i7qyF7yrGyccZQUG9Jv0IHVymE4EksQrwEyMbQEsLXJ64iDV dKP8xPhpDqSapCO3XjOIN51s0k7N4uQmc/9Og67Yv9tqcVFXTmP/viZUhvTQjLEsW/1w sUYfcopmeWbBo2uxePrNbvnU5r6HJxGwuENKVVjnIHiNKOV7XxPCZnwKWe+yqm6aGNGG Xfng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Zg5Ds5JA; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Zg5Ds5JA; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=OxZEHxqI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id m12-20020a056a00080c00b006cea5b0c565si787970pfk.282.2023.12.07.19.31.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 19:31:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Zg5Ds5JA; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Zg5Ds5JA; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=OxZEHxqI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 421D2803BEC1; Thu, 7 Dec 2023 19:31:10 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232752AbjLHDao (ORCPT + 99 others); Thu, 7 Dec 2023 22:30:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229531AbjLHDam (ORCPT ); Thu, 7 Dec 2023 22:30:42 -0500 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2a07:de40:b251:101:10:150:64:2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3161B1712; Thu, 7 Dec 2023 19:30:48 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BD3ED1FD95; Fri, 8 Dec 2023 03:30:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1702006246; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FPbjaalRYKm9QivlQuryrahlBhRTkM5VSSKDElwEPLI=; b=Zg5Ds5JAZACw4o+g9jJZBmTt8O7GGia+Z7B28SSO5ghA9zHtjZE1/qnDyxS8RoiXAmkfk8 ex4KlIzW4nquEv8QGyIQ2pKi3iksw/Y6pjvL4WN++TgOzz0/0n7OYLKayrv0CmLTvUgIbD /5lwiFXCxC0LoZQPTpsavfkp3BMRR+o= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1702006246; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FPbjaalRYKm9QivlQuryrahlBhRTkM5VSSKDElwEPLI=; b=OxZEHxqIuxXo33jqY+SDnj7i5LU9K/udAF5suU/yqdh9vXcCozViAtxy8JbLlzwPkia4pw iGTEtKTw9v1/QIDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1702006246; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FPbjaalRYKm9QivlQuryrahlBhRTkM5VSSKDElwEPLI=; b=Zg5Ds5JAZACw4o+g9jJZBmTt8O7GGia+Z7B28SSO5ghA9zHtjZE1/qnDyxS8RoiXAmkfk8 ex4KlIzW4nquEv8QGyIQ2pKi3iksw/Y6pjvL4WN++TgOzz0/0n7OYLKayrv0CmLTvUgIbD /5lwiFXCxC0LoZQPTpsavfkp3BMRR+o= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1702006246; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FPbjaalRYKm9QivlQuryrahlBhRTkM5VSSKDElwEPLI=; b=OxZEHxqIuxXo33jqY+SDnj7i5LU9K/udAF5suU/yqdh9vXcCozViAtxy8JbLlzwPkia4pw iGTEtKTw9v1/QIDQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id D46581375D; Fri, 8 Dec 2023 03:30:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id k/10HeKNcmULMQAAD6G6ig (envelope-from ); Fri, 08 Dec 2023 03:30:42 +0000 From: NeilBrown To: Al Viro , Christian Brauner , Jens Axboe , Oleg Nesterov , Chuck Lever , Jeff Layton Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH 2/3] nfsd: Don't leave work of closing files to a work queue. Date: Fri, 8 Dec 2023 14:27:27 +1100 Message-ID: <20231208033006.5546-3-neilb@suse.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231208033006.5546-1-neilb@suse.de> References: <20231208033006.5546-1-neilb@suse.de> MIME-Version: 1.0 X-Spam-Score: 0.71 X-Spamd-Result: default: False [1.90 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_SEVEN(0.00)[9]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] Authentication-Results: smtp-out2.suse.de; none X-Spam-Score: 1.90 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 07 Dec 2023 19:31:10 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784682930198979841 X-GMAIL-MSGID: 1784682930198979841 The work of closing a file can have non-trivial cost. Doing it in a separate work queue thread means that cost isn't imposed on the nfsd threads and an imbalance can be created. This can result in files being queue for the work queue more quickly that the work queue can process them, resulting in unbounded growth of the queue and memory exhaustion. To avoid this work imbalance that exhausts memory, this patch moves all closing of files into the nfsd threads. This means that when the work imposes a cost, that cost appears where it would be expected - in the work of the nfsd thread. A subsequent patch will ensure the final __fput() is called in the same (nfsd) thread which calls filp_close(). Files opened for NFSv3 are never explicitly closed by the client and are kept open by the server in the "filecache", which responds to memory pressure, is garbage collected even when there is no pressure, and sometimes closes files when there is particular need such as for rename. These files currently have filp_close() called in a dedicated work queue, so their __fput() can have no effect on nfsd threads. This patch discards the work queue and instead has each nfsd thread call flip_close() on as many as 8 files from the filecache each time it acts on a client request (or finds there are no pending client requests). If there are more to be closed, more threads are woken. This spreads the work of __fput() over multiple threads and imposes any cost on those threads. The number 8 is somewhat arbitrary. It needs to be greater than 1 to ensure that files are closed more quickly than they can be added to the cache. It needs to be small enough to limit the per-request delays that will be imposed on clients when all threads are busy closing files. Signed-off-by: NeilBrown --- fs/nfsd/filecache.c | 67 +++++++++++++++++++++------------------------ fs/nfsd/filecache.h | 1 + fs/nfsd/nfssvc.c | 2 ++ 3 files changed, 34 insertions(+), 36 deletions(-) diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index e9734c7451b5..f9da4b0c4d42 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -61,13 +61,10 @@ static DEFINE_PER_CPU(unsigned long, nfsd_file_total_age); static DEFINE_PER_CPU(unsigned long, nfsd_file_evictions); struct nfsd_fcache_disposal { - struct work_struct work; spinlock_t lock; struct list_head freeme; }; -static struct workqueue_struct *nfsd_filecache_wq __read_mostly; - static struct kmem_cache *nfsd_file_slab; static struct kmem_cache *nfsd_file_mark_slab; static struct list_lru nfsd_file_lru; @@ -423,7 +420,37 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose) spin_lock(&l->lock); list_move_tail(&nf->nf_lru, &l->freeme); spin_unlock(&l->lock); - queue_work(nfsd_filecache_wq, &l->work); + svc_wake_up(nn->nfsd_serv); + } +} + +/** + * nfsd_file_net_dispose - deal with nfsd_files waiting to be disposed. + * @nn: nfsd_net in which to find files to be disposed. + * + * When files held open for nfsv3 are removed from the filecache, whether + * due to memory pressure or garbage collection, they are queued to + * a per-net-ns queue. This function completes the disposal, either + * directly or by waking another nfsd thread to help with the work. + */ +void nfsd_file_net_dispose(struct nfsd_net *nn) +{ + struct nfsd_fcache_disposal *l = nn->fcache_disposal; + + if (!list_empty(&l->freeme)) { + LIST_HEAD(dispose); + int i; + + spin_lock(&l->lock); + for (i = 0; i < 8 && !list_empty(&l->freeme); i++) + list_move(l->freeme.next, &dispose); + spin_unlock(&l->lock); + if (!list_empty(&l->freeme)) + /* Wake up another thread to share the work + * *before* doing any actual disposing. + */ + svc_wake_up(nn->nfsd_serv); + nfsd_file_dispose_list(&dispose); } } @@ -635,27 +662,6 @@ nfsd_file_close_inode_sync(struct inode *inode) } } -/** - * nfsd_file_delayed_close - close unused nfsd_files - * @work: dummy - * - * Scrape the freeme list for this nfsd_net, and then dispose of them - * all. - */ -static void -nfsd_file_delayed_close(struct work_struct *work) -{ - LIST_HEAD(head); - struct nfsd_fcache_disposal *l = container_of(work, - struct nfsd_fcache_disposal, work); - - spin_lock(&l->lock); - list_splice_init(&l->freeme, &head); - spin_unlock(&l->lock); - - nfsd_file_dispose_list(&head); -} - static int nfsd_file_lease_notifier_call(struct notifier_block *nb, unsigned long arg, void *data) @@ -718,10 +724,6 @@ nfsd_file_cache_init(void) return ret; ret = -ENOMEM; - nfsd_filecache_wq = alloc_workqueue("nfsd_filecache", 0, 0); - if (!nfsd_filecache_wq) - goto out; - nfsd_file_slab = kmem_cache_create("nfsd_file", sizeof(struct nfsd_file), 0, 0, NULL); if (!nfsd_file_slab) { @@ -736,7 +738,6 @@ nfsd_file_cache_init(void) goto out_err; } - ret = list_lru_init(&nfsd_file_lru); if (ret) { pr_err("nfsd: failed to init nfsd_file_lru: %d\n", ret); @@ -786,8 +787,6 @@ nfsd_file_cache_init(void) nfsd_file_slab = NULL; kmem_cache_destroy(nfsd_file_mark_slab); nfsd_file_mark_slab = NULL; - destroy_workqueue(nfsd_filecache_wq); - nfsd_filecache_wq = NULL; rhltable_destroy(&nfsd_file_rhltable); goto out; } @@ -833,7 +832,6 @@ nfsd_alloc_fcache_disposal(void) l = kmalloc(sizeof(*l), GFP_KERNEL); if (!l) return NULL; - INIT_WORK(&l->work, nfsd_file_delayed_close); spin_lock_init(&l->lock); INIT_LIST_HEAD(&l->freeme); return l; @@ -842,7 +840,6 @@ nfsd_alloc_fcache_disposal(void) static void nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l) { - cancel_work_sync(&l->work); nfsd_file_dispose_list(&l->freeme); kfree(l); } @@ -911,8 +908,6 @@ nfsd_file_cache_shutdown(void) fsnotify_wait_marks_destroyed(); kmem_cache_destroy(nfsd_file_mark_slab); nfsd_file_mark_slab = NULL; - destroy_workqueue(nfsd_filecache_wq); - nfsd_filecache_wq = NULL; rhltable_destroy(&nfsd_file_rhltable); for_each_possible_cpu(i) { diff --git a/fs/nfsd/filecache.h b/fs/nfsd/filecache.h index e54165a3224f..c61884def906 100644 --- a/fs/nfsd/filecache.h +++ b/fs/nfsd/filecache.h @@ -56,6 +56,7 @@ void nfsd_file_cache_shutdown_net(struct net *net); void nfsd_file_put(struct nfsd_file *nf); struct nfsd_file *nfsd_file_get(struct nfsd_file *nf); void nfsd_file_close_inode_sync(struct inode *inode); +void nfsd_file_net_dispose(struct nfsd_net *nn); bool nfsd_file_is_cached(struct inode *inode); __be32 nfsd_file_acquire_gc(struct svc_rqst *rqstp, struct svc_fh *fhp, unsigned int may_flags, struct nfsd_file **nfp); diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index fe61d9bbcc1f..26e203f42edd 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -956,6 +956,8 @@ nfsd(void *vrqstp) svc_recv(rqstp); validate_process_creds(); + + nfsd_file_net_dispose(nn); } atomic_dec(&nfsdstats.th_cnt); From patchwork Fri Dec 8 03:27:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: NeilBrown X-Patchwork-Id: 175552 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5220162vqy; Thu, 7 Dec 2023 19:31:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IE3+uqiAG+vfEujKUmCKzzyUKYgDK5Glt5tKY3ivUu43YYu8SYH/7MzZIVJzzrxC7fvxdKo X-Received: by 2002:a17:903:187:b0:1cf:d2c5:ae87 with SMTP id z7-20020a170903018700b001cfd2c5ae87mr3867018plg.11.1702006278188; Thu, 07 Dec 2023 19:31:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702006278; cv=none; d=google.com; s=arc-20160816; b=zT6I2KDSIqTCnAMt+Z8c+7Kpl+rRMBfL8bXR+ALWiT5UqO+heqbGnlWbg66LI2lk6H pcfyKrqjrIxHMQM5jhS0vDjwRXeYpvck4XLnsmUMP9ng2D7QZ7kiDbmU7SG/OHBMhIhf 5lxX6BE2mKwFP/4veDyeD2MaytOi9Qpoyf3R3/EiODS91IfgJpd4JbnarY7oYGxuU0i2 YntYgv1Jcl8yLP3ntW1KoLLlMWAkQ7A0rXZUlhYEf7lGlHJkV4IuKt3LCJDkDEQGFOXe 76fyn6H/aN8veZSEIfFznLUrWv5eduTSBfBXw4IA10rzzDPui53mbZtLjOwBK+N96K8L y59g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=XDQYdne0DAzlGGGJZTNP/1WiB4ya0IEng6FXcLsrx3E=; fh=I45k8M33HW2Wqj1OlgncPyvUt41clxPTDAFs+9Wisus=; b=rDGf2H2sJ9K2WhYnZiMO7rSW1yuueqN1orUy4enNSG8AY/VhBZeo9fUlJgrWNlpC1p RH5eZsdlZYofOPXU0RsdJQacQjBO/gUL6mEec/LC5wVuV49V+SzCH8L4uNCCkAGQ7Efq APgOFnMCFAcsE4gZJ3ZFzWfOio6/lOV47IRMKOHD0DhY+aK/16p+iW/DQj5QzjygLSJs TbiWVfdW6Jhdh81+bnTwXJZGKmd9BqQbLvqZf/nY3Uq365tEs/70YcEuv7/1IceIKaoA d6/Q3qp2x3a4Z9HXFo8IxyMHwdGzsjDcc82NxIXL6EzBqdkPf6PoND9MY4K1CQPQjzlS uzFg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=kxEJzfw8; dkim=neutral (no key) header.i=@suse.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id 19-20020a170902c21300b001cfc0272b48si800550pll.103.2023.12.07.19.31.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 19:31:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=kxEJzfw8; dkim=neutral (no key) header.i=@suse.de; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id C223B803D818; Thu, 7 Dec 2023 19:31:15 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229531AbjLHDay (ORCPT + 99 others); Thu, 7 Dec 2023 22:30:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235655AbjLHDav (ORCPT ); Thu, 7 Dec 2023 22:30:51 -0500 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2a07:de40:b251:101:10:150:64:1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EE261721; Thu, 7 Dec 2023 19:30:57 -0800 (PST) Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id A5E20220FA; Fri, 8 Dec 2023 03:30:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1702006255; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XDQYdne0DAzlGGGJZTNP/1WiB4ya0IEng6FXcLsrx3E=; b=kxEJzfw852I1pebVZpIvOZFm58YLyLf03hOuU+L5jy6BWRJK2dLEt3c9KZOrIcD2QX8FId VIr2VJxQAI6Q0j/JUVEroHwadNpO8y4TomUtw9vCg0ID/Edcbi0rCakZ2m0u3iI5d8pB/p DCdcyOgiklrgEguWBs0U9sOygeLf6tE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1702006255; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XDQYdne0DAzlGGGJZTNP/1WiB4ya0IEng6FXcLsrx3E=; b=bvOOsRMbaJQU2pYYXCg57Ki+nRANJ2GKR5gsoIHILcCMYq6reMSFF1EPSTsS/C/CVuMLV4 02nKP3nMq0U1reAg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 6FADB1375D; Fri, 8 Dec 2023 03:30:51 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap1.dmz-prg2.suse.org with ESMTPSA id WRj9AeuNcmUcMQAAD6G6ig (envelope-from ); Fri, 08 Dec 2023 03:30:51 +0000 From: NeilBrown To: Al Viro , Christian Brauner , Jens Axboe , Oleg Nesterov , Chuck Lever , Jeff Layton Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nfs@vger.kernel.org Subject: [PATCH 3/3] VFS: don't export flush_delayed_fput(). Date: Fri, 8 Dec 2023 14:27:28 +1100 Message-ID: <20231208033006.5546-4-neilb@suse.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231208033006.5546-1-neilb@suse.de> References: <20231208033006.5546-1-neilb@suse.de> MIME-Version: 1.0 X-Spamd-Result: default: False [4.90 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_SEVEN(0.00)[9]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-0.00)[36.00%] Authentication-Results: smtp-out1.suse.de; none X-Spam-Score: 4.90 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 07 Dec 2023 19:31:15 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784682935125952400 X-GMAIL-MSGID: 1784682935125952400 There are no more users of flush_delayed_fput(). Modules shouldn't need it. __fput_sync() is a better approach for any code which might benefit from flush_delayed_fput() Signed-off-by: NeilBrown --- fs/file_table.c | 1 - 1 file changed, 1 deletion(-) diff --git a/fs/file_table.c b/fs/file_table.c index de4a2915bfd4..de9155f82d4a 100644 --- a/fs/file_table.c +++ b/fs/file_table.c @@ -436,7 +436,6 @@ void flush_delayed_fput(void) { delayed_fput(NULL); } -EXPORT_SYMBOL_GPL(flush_delayed_fput); static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput);