From patchwork Wed Jan 31 08:51:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 194635 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp1769821dyb; Wed, 31 Jan 2024 01:29:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IGQKoZzr8cLdD1pyeCPMZVS1l/VXNpkVR5Nv6iIbF3uPZbLvI1gyszvnSFJiI2der7IsSEJ X-Received: by 2002:a17:90a:fe16:b0:295:f901:d78 with SMTP id ck22-20020a17090afe1600b00295f9010d78mr107777pjb.28.1706693345052; Wed, 31 Jan 2024 01:29:05 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706693345; cv=pass; d=google.com; s=arc-20160816; b=hnSIFnlQGlqZmiU8l/BNFYzHfOAd+1tAGbIKkuLQZDVTmsc1Ps8pEj/NS8ipsZR9z2 A73QRYIb3YkBxy1YqzKfxSY8bGpRCb9v3am9pYud5FHuvTUIyfsF3kqb5MF2SKu7WQhj z3ccqTjiiuMovV1vhxKfvXSZcjdG9XB+DfcKNtUUcec67nhHgXRkqWwYTiYN8DzaERoq MjHaZ5/wMypKrjMinWiYpBY3se5wjZnTpPjextIRqN+Srul1hGvx9A85em1L0U5e2GF4 EWA8IL9qmu4AQRVlaxhgwbWHtOKW5aSHYzn0moEj48YjNNUC8eULJBC7v/6nNnmZh/KZ XrBQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature:dkim-signature :dkim-signature:dkim-signature; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; fh=8u0ga93yMOvqDqPAHe9sHABxum7VbY8hTa1A/atnTGA=; b=lfmtUwCSEbYwpw+f1E+G0SADmDYu6wilK5m5bC84UBrhyB9+3+ko0qwRnEuJexH3Ml Tq+wam7C/tSE8EOjvozIcJcmCg/FzuREm/PAYDHlo/QqfIi6Bh3gZaeS0IgG1j423bOQ YjM9NfIXoxZwKFD6t2j01FH53hbbqLn302D95ErlDaTgztD4lMlUCJ2C8YsCY/P9IdHq 9r/+YPRLGWzvzZpN1cj9BeU0ZiqNLLp1Avrq+NSIUjcNpU6VEdC+F0G7axe+j6n++A6N UoIIKQwhL0YA3KKtuty2IhBokWG1papGnI9zZytZdpVg7n/ekm9uzEajHSmsBo0nCT3l 49Yw==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=tiYln1Hm; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=zxZdbmUa; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1 spf=pass spfdomain=suse.de dkim=pass dkdomain=suse.de dkim=pass dkdomain=suse.de dmarc=pass fromdomain=suse.de); spf=pass (google.com: domain of linux-kernel+bounces-46070-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46070-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de X-Forwarded-Encrypted: i=1; AJvYcCXUq+6y/GEuEwS/ThJtLcBNvnH5yrLkFkepA9kLLa24P2r2Euejb2bUe245aQw/ejm2f/iYCwTu2HKlZ4A97sOKgyYDag== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id a4-20020a1709027d8400b001d730061829si8971374plm.324.2024.01.31.01.29.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 01:29:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-46070-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=tiYln1Hm; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=zxZdbmUa; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1 spf=pass spfdomain=suse.de dkim=pass dkdomain=suse.de dkim=pass dkdomain=suse.de dmarc=pass fromdomain=suse.de); spf=pass (google.com: domain of linux-kernel+bounces-46070-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-46070-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id B40F6B2B310 for ; Wed, 31 Jan 2024 09:00:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5F1FA7F496; Wed, 31 Jan 2024 08:51:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="tiYln1Hm"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="h0NpJCYb"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="zxZdbmUa"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="aTM6YJEO" Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 777D379DA0 for ; Wed, 31 Jan 2024 08:51:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706691087; cv=none; b=CryKfJBLQM+ThBjyZJWpC71NJ53RieuX1AdrPyMYdabb8Tc8Unk6Pdtc7DcaA5+9SwJrGfIWrZybFDu+/HQUzQnUUP+yKa0i4nsk1q29w9rYTIeHUaUOjE37gZAp25T5K16FuB/GI9wA2jgIYqeMfbz7nusnHsnbjEnI+iCR5Ws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706691087; c=relaxed/simple; bh=xwdC8KCo1HnYmzjGNarX5b9itC4QycUjE+VsG0lmj5A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Yh+E1s+gsmwIUqndLBu3shp2nW7Konvqvn8Mmz5YsxFKbHEdHpeM2qgwldEeEyul/ecQGcDbpwCKT1dVn7oUyJIedtBRe/9BRYXIXaDltrgVX12oF/Z3HOt6DQkWf5t/M0MM7h8l+hFM7OxMhG6BSglfacCatHUH9pJ6h7jjw7Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=tiYln1Hm; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=h0NpJCYb; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=zxZdbmUa; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=aTM6YJEO; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id EEDD121FBD; Wed, 31 Jan 2024 08:51:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1706691083; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=tiYln1HmscGL3QmP60zbomHq+mTIKFDsc98LqNKgIQcmETHO8TjoNe+kd9978W0KqxvJni OoXoNQg5YoV8b9piBSuQkcsdUvlNYwmJxAxk4ZEUblIp0RtINC4w3biFlnlmm6x+STg3Am kRz+L+71MwTwmh2CZDmrlZn55r/s6U8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1706691083; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=h0NpJCYb8yjemb6lgcnX8fKE82ZOWl5smEL+eukPeKAfHJFhN7UigDa1zN8TENE5UG9KCM WWMkrg9/q0vS5TCQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1706691081; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=zxZdbmUak1zARqUdN2NkVqS92NFAdjZuG3P8ULK3UlXDDvNGFu4CzN0ywvywN9FpqxXf1s QN7eAX1Ksfpmyu88fOyBmvB6b7wepaV6EU4oqZe1YMZg2lCn7xnKIKP2E/NjfSSUH9Zele sY+1kNwIBWa74RfBE1i1bOmspIyiCq0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1706691081; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=aTM6YJEOOvTh3pFRJivPV9UXO1I2LX/wxbeRmtpsdPkusmXVpJbz9QQTnV/tfiz4P4kpal 7XqoQtIKEUJWE9Cg== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id DC896132FA; Wed, 31 Jan 2024 08:51:21 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap2.dmz-prg2.suse.org with ESMTPSA id hKRlNAkKumV7HAAAn2gu4w (envelope-from ); Wed, 31 Jan 2024 08:51:21 +0000 From: Daniel Wagner To: James Smart Cc: Keith Busch , Christoph Hellwig , Hannes Reinecke , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Daniel Wagner Subject: [PATCH v5 01/12] nvme-fc: do not wait in vain when unloading module Date: Wed, 31 Jan 2024 09:51:01 +0100 Message-ID: <20240131085112.21668-2-dwagner@suse.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240131085112.21668-1-dwagner@suse.de> References: <20240131085112.21668-1-dwagner@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Authentication-Results: smtp-out1.suse.de; none X-Spamd-Result: default: False [1.90 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.de:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Level: * X-Spam-Score: 1.90 X-Spam-Flag: NO X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789597680764814110 X-GMAIL-MSGID: 1789597680764814110 The module exit path has race between deleting all controllers and freeing 'left over IDs'. To prevent double free a synchronization between nvme_delete_ctrl and ida_destroy has been added by the initial commit. There is some logic around trying to prevent from hanging forever in wait_for_completion, though it does not handling all cases. E.g. blktests is able to reproduce the situation where the module unload hangs forever. If we completely rely on the cleanup code executed from the nvme_delete_ctrl path, all IDs will be freed eventually. This makes calling ida_destroy unnecessary. We only have to ensure that all nvme_delete_ctrl code has been executed before we leave nvme_fc_exit_module. This is done by flushing the nvme_delete_wq workqueue. While at it, remove the unused nvme_fc_wq workqueue too. Reviewed-by: Christoph Hellwig Signed-off-by: Daniel Wagner Reviewed-by: Hannes Reinecke --- drivers/nvme/host/fc.c | 47 ++++++------------------------------------ 1 file changed, 6 insertions(+), 41 deletions(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index e2308119f8f0..7006f4caac2f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -221,11 +221,6 @@ static LIST_HEAD(nvme_fc_lport_list); static DEFINE_IDA(nvme_fc_local_port_cnt); static DEFINE_IDA(nvme_fc_ctrl_cnt); -static struct workqueue_struct *nvme_fc_wq; - -static bool nvme_fc_waiting_to_unload; -static DECLARE_COMPLETION(nvme_fc_unload_proceed); - /* * These items are short-term. They will eventually be moved into * a generic FC class. See comments in module init. @@ -255,8 +250,6 @@ nvme_fc_free_lport(struct kref *ref) /* remove from transport list */ spin_lock_irqsave(&nvme_fc_lock, flags); list_del(&lport->port_list); - if (nvme_fc_waiting_to_unload && list_empty(&nvme_fc_lport_list)) - complete(&nvme_fc_unload_proceed); spin_unlock_irqrestore(&nvme_fc_lock, flags); ida_free(&nvme_fc_local_port_cnt, lport->localport.port_num); @@ -3894,10 +3887,6 @@ static int __init nvme_fc_init_module(void) { int ret; - nvme_fc_wq = alloc_workqueue("nvme_fc_wq", WQ_MEM_RECLAIM, 0); - if (!nvme_fc_wq) - return -ENOMEM; - /* * NOTE: * It is expected that in the future the kernel will combine @@ -3915,7 +3904,7 @@ static int __init nvme_fc_init_module(void) ret = class_register(&fc_class); if (ret) { pr_err("couldn't register class fc\n"); - goto out_destroy_wq; + return ret; } /* @@ -3939,8 +3928,6 @@ static int __init nvme_fc_init_module(void) device_destroy(&fc_class, MKDEV(0, 0)); out_destroy_class: class_unregister(&fc_class); -out_destroy_wq: - destroy_workqueue(nvme_fc_wq); return ret; } @@ -3960,45 +3947,23 @@ nvme_fc_delete_controllers(struct nvme_fc_rport *rport) spin_unlock(&rport->lock); } -static void -nvme_fc_cleanup_for_unload(void) +static void __exit nvme_fc_exit_module(void) { struct nvme_fc_lport *lport; struct nvme_fc_rport *rport; - - list_for_each_entry(lport, &nvme_fc_lport_list, port_list) { - list_for_each_entry(rport, &lport->endp_list, endp_list) { - nvme_fc_delete_controllers(rport); - } - } -} - -static void __exit nvme_fc_exit_module(void) -{ unsigned long flags; - bool need_cleanup = false; spin_lock_irqsave(&nvme_fc_lock, flags); - nvme_fc_waiting_to_unload = true; - if (!list_empty(&nvme_fc_lport_list)) { - need_cleanup = true; - nvme_fc_cleanup_for_unload(); - } + list_for_each_entry(lport, &nvme_fc_lport_list, port_list) + list_for_each_entry(rport, &lport->endp_list, endp_list) + nvme_fc_delete_controllers(rport); spin_unlock_irqrestore(&nvme_fc_lock, flags); - if (need_cleanup) { - pr_info("%s: waiting for ctlr deletes\n", __func__); - wait_for_completion(&nvme_fc_unload_proceed); - pr_info("%s: ctrl deletes complete\n", __func__); - } + flush_workqueue(nvme_delete_wq); nvmf_unregister_transport(&nvme_fc_transport); - ida_destroy(&nvme_fc_local_port_cnt); - ida_destroy(&nvme_fc_ctrl_cnt); - device_destroy(&fc_class, MKDEV(0, 0)); class_unregister(&fc_class); - destroy_workqueue(nvme_fc_wq); } module_init(nvme_fc_init_module);