From patchwork Tue Jan 30 09:49:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 193933 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2087:b0:106:209c:c626 with SMTP id gs7csp1107104dyb; Tue, 30 Jan 2024 01:53:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IH/fizotrswmyxnSpwsYdL5yS3VlVHDADr46fqSYoIW6FPRhbHKrKXHeB0CM2XBblN5rEf4 X-Received: by 2002:a17:906:55c7:b0:a31:1ab7:5152 with SMTP id z7-20020a17090655c700b00a311ab75152mr6435641ejp.51.1706608428624; Tue, 30 Jan 2024 01:53:48 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706608428; cv=pass; d=google.com; s=arc-20160816; b=HKYiDHmzwBPtP6369ixZ4sohbLTFQsUlJJpEOXC++LlH7cl3wLwvWi7yX2jUxpla1R 4ZbA5vOziQ63RnMCq9ANRN5Ri1rOp4Y7ZB389zYhXyuMJdvcsOgdPjrhO5NAlmkhL6An 1eFyLAHTFH0nJNxXoe79Pn3wlzXouFk2/BCm4ZBrh54lvWs9dFuV9Ntp4EpB/tUzec2v f+Tef2P5RbLVg3E6ttUUKUG2nsKTwyPrK/RllDPqK/wHox0aGo/pDcvis2B5u2FoSDwA Je+RoAti8i1pGFwr/GFPbffyz7zldDeaYmBN2Ze7hC34Un/xHw18QFO3bnFBiozW91+j mh6Q== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature:dkim-signature :dkim-signature:dkim-signature; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; fh=n5BdL0bf9gAg+lJbD9hdO95/5FlrOLRCMPX4XUPPjVI=; b=SbSWfASnudsaE8PGts8/9kcDtRH2mhi5iqmXSk5VGlnNL3CU8VLgPNSkYWs44cueBj c+CsDMCfdQN1i4uH6STa6L6OUap+eq160B4Zs+HHMi27ZsMD7aqD9WIu1mHqxHU7Ga56 jblAiow5TEGzsaQNDQ1N8yCATKC03QGxRgNJWv8v6RGPTfCCYfKUFsNX0KMOvT02ig51 kLmCAW4EeNTPnP1Xb1cqZwtcEDsyeASevgXUOtARfwwUEdcUjUxr/xQsnwuyyCtXMlhi IKSWYHntEShRYDTKUYQlN13yoXy8Xo1+HrmLe+rZsVSYWKwTjQwrn0noAczUfxCO3X85 v+7Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Y5HhdEQ2; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=pezbGGzR; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=CdPcYCjd; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1 spf=pass spfdomain=suse.de dkim=pass dkdomain=suse.de dkim=pass dkdomain=suse.de dmarc=pass fromdomain=suse.de); spf=pass (google.com: domain of linux-kernel+bounces-44306-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-44306-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id x15-20020a1709064a8f00b00a353d98ddebsi3215832eju.767.2024.01.30.01.53.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jan 2024 01:53:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-44306-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Y5HhdEQ2; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=pezbGGzR; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=CdPcYCjd; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519; arc=pass (i=1 spf=pass spfdomain=suse.de dkim=pass dkdomain=suse.de dkim=pass dkdomain=suse.de dmarc=pass fromdomain=suse.de); spf=pass (google.com: domain of linux-kernel+bounces-44306-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-44306-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 170591F21869 for ; Tue, 30 Jan 2024 09:53:48 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 048AD6A322; Tue, 30 Jan 2024 09:49:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="Y5HhdEQ2"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="pezbGGzR"; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b="CdPcYCjd"; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b="8ry3+ddc" Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5106C66B2A for ; Tue, 30 Jan 2024 09:49:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706608193; cv=none; b=rU/6n8qvrXK5JPpzqBTb6wv3hbTwK3+pQ2z5DEsI5kpiDJeU478IJgf7EDSu+ha91haoDx++1HHML5spdy+9vldRjCSLBS6lLnbyKbrBbOasmcpNBKYztvuaTaV10ThAnn0VEhjT8cRgwEmfAOXIfifZpIi2C7p/JTJqZsmjQN0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706608193; c=relaxed/simple; bh=xwdC8KCo1HnYmzjGNarX5b9itC4QycUjE+VsG0lmj5A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PEikAb8IXXcOh/8XUpo0A3LoMfee49WrRC60lGnPgl2dclZsad+q8qN4Cxdc7QnZXNGC2lIBIfuJNtGY7qtQp9Qs7F19FC42vGW6G1wKXdLdlDNfwlPwKRbA2pts7fnGw41pXtzusLOmXuw7Y2JSwzjskm+nwjJ2XMiP1zrU13c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de; spf=pass smtp.mailfrom=suse.de; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=Y5HhdEQ2; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=pezbGGzR; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.b=CdPcYCjd; dkim=permerror (0-bit key) header.d=suse.de header.i=@suse.de header.b=8ry3+ddc; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.de Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 652AA222D9; Tue, 30 Jan 2024 09:49:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1706608189; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=Y5HhdEQ2+8mSq7jMxOVGI09o4nRTc/dQG3W5v43J0F8tffe4pfV7moakteneRQ1jEXytwP 96H42e9hJvvjSxqZPUW2zgFvkFr7lk1rY5UYmx7aNBffm5taYE6PkdN8hH3padR0KOfz4x mEYY2HThtUs3LIe9ul9z7mHF4EyVUZw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1706608189; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=pezbGGzR66ss3jFic99GF1A2P7FWjPT0FT3aAU5ksolJn3nGBYEeFYpp2fXcXrpyCkl3GY R4IA2P3ZfcEZ4QAw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1706608188; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=CdPcYCjdFPBSPWvbFmXkMRfxn1eLxHdF3Bre9FmVOtC6F1PA2WJWlFedg/KSNVY/3ev2X1 d+6QrVAfu5Q8wc2ECiOjngVz6WFm9Ri3jOL6p9W9hzDGEcDTwYN3xhdeqJ/QvlmINVZIiT NaEf6rr9t0fIUAd/SjRzr/OsAZs/9pY= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1706608188; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oYsmbdq/eZQPw8pTg90yqgsJeRXWUAVcHMYkG4+lxfo=; b=8ry3+ddcK+It3X8cT/r7UsLYiH1/8hil0RnCw3F/rf97P22QC230tygdzanCEID4nxy4Dr ohFHqFuZh5aLw6Bg== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 51EFF13462; Tue, 30 Jan 2024 09:49:48 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap2.dmz-prg2.suse.org with ESMTPSA id 4+aREjzGuGWEbwAAn2gu4w (envelope-from ); Tue, 30 Jan 2024 09:49:48 +0000 From: Daniel Wagner To: James Smart Cc: Keith Busch , Christoph Hellwig , Hannes Reinecke , linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Daniel Wagner Subject: [PATCH v4 01/12] nvme-fc: do not wait in vain when unloading module Date: Tue, 30 Jan 2024 10:49:27 +0100 Message-ID: <20240130094938.1575-2-dwagner@suse.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130094938.1575-1-dwagner@suse.de> References: <20240130094938.1575-1-dwagner@suse.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Authentication-Results: smtp-out1.suse.de; none X-Spamd-Result: default: False [-2.10 / 50.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_MISSING_CHARSET(2.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; REPLY(-4.00)[]; BROKEN_CONTENT_TYPE(1.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; RCPT_COUNT_SEVEN(0.00)[7]; MID_CONTAINS_FROM(1.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[lst.de:email,suse.de:email]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; BAYES_HAM(-3.00)[100.00%] X-Spam-Level: X-Spam-Flag: NO X-Spam-Score: -2.10 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789508639699084477 X-GMAIL-MSGID: 1789508639699084477 The module exit path has race between deleting all controllers and freeing 'left over IDs'. To prevent double free a synchronization between nvme_delete_ctrl and ida_destroy has been added by the initial commit. There is some logic around trying to prevent from hanging forever in wait_for_completion, though it does not handling all cases. E.g. blktests is able to reproduce the situation where the module unload hangs forever. If we completely rely on the cleanup code executed from the nvme_delete_ctrl path, all IDs will be freed eventually. This makes calling ida_destroy unnecessary. We only have to ensure that all nvme_delete_ctrl code has been executed before we leave nvme_fc_exit_module. This is done by flushing the nvme_delete_wq workqueue. While at it, remove the unused nvme_fc_wq workqueue too. Reviewed-by: Christoph Hellwig Signed-off-by: Daniel Wagner --- drivers/nvme/host/fc.c | 47 ++++++------------------------------------ 1 file changed, 6 insertions(+), 41 deletions(-) diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index e2308119f8f0..7006f4caac2f 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -221,11 +221,6 @@ static LIST_HEAD(nvme_fc_lport_list); static DEFINE_IDA(nvme_fc_local_port_cnt); static DEFINE_IDA(nvme_fc_ctrl_cnt); -static struct workqueue_struct *nvme_fc_wq; - -static bool nvme_fc_waiting_to_unload; -static DECLARE_COMPLETION(nvme_fc_unload_proceed); - /* * These items are short-term. They will eventually be moved into * a generic FC class. See comments in module init. @@ -255,8 +250,6 @@ nvme_fc_free_lport(struct kref *ref) /* remove from transport list */ spin_lock_irqsave(&nvme_fc_lock, flags); list_del(&lport->port_list); - if (nvme_fc_waiting_to_unload && list_empty(&nvme_fc_lport_list)) - complete(&nvme_fc_unload_proceed); spin_unlock_irqrestore(&nvme_fc_lock, flags); ida_free(&nvme_fc_local_port_cnt, lport->localport.port_num); @@ -3894,10 +3887,6 @@ static int __init nvme_fc_init_module(void) { int ret; - nvme_fc_wq = alloc_workqueue("nvme_fc_wq", WQ_MEM_RECLAIM, 0); - if (!nvme_fc_wq) - return -ENOMEM; - /* * NOTE: * It is expected that in the future the kernel will combine @@ -3915,7 +3904,7 @@ static int __init nvme_fc_init_module(void) ret = class_register(&fc_class); if (ret) { pr_err("couldn't register class fc\n"); - goto out_destroy_wq; + return ret; } /* @@ -3939,8 +3928,6 @@ static int __init nvme_fc_init_module(void) device_destroy(&fc_class, MKDEV(0, 0)); out_destroy_class: class_unregister(&fc_class); -out_destroy_wq: - destroy_workqueue(nvme_fc_wq); return ret; } @@ -3960,45 +3947,23 @@ nvme_fc_delete_controllers(struct nvme_fc_rport *rport) spin_unlock(&rport->lock); } -static void -nvme_fc_cleanup_for_unload(void) +static void __exit nvme_fc_exit_module(void) { struct nvme_fc_lport *lport; struct nvme_fc_rport *rport; - - list_for_each_entry(lport, &nvme_fc_lport_list, port_list) { - list_for_each_entry(rport, &lport->endp_list, endp_list) { - nvme_fc_delete_controllers(rport); - } - } -} - -static void __exit nvme_fc_exit_module(void) -{ unsigned long flags; - bool need_cleanup = false; spin_lock_irqsave(&nvme_fc_lock, flags); - nvme_fc_waiting_to_unload = true; - if (!list_empty(&nvme_fc_lport_list)) { - need_cleanup = true; - nvme_fc_cleanup_for_unload(); - } + list_for_each_entry(lport, &nvme_fc_lport_list, port_list) + list_for_each_entry(rport, &lport->endp_list, endp_list) + nvme_fc_delete_controllers(rport); spin_unlock_irqrestore(&nvme_fc_lock, flags); - if (need_cleanup) { - pr_info("%s: waiting for ctlr deletes\n", __func__); - wait_for_completion(&nvme_fc_unload_proceed); - pr_info("%s: ctrl deletes complete\n", __func__); - } + flush_workqueue(nvme_delete_wq); nvmf_unregister_transport(&nvme_fc_transport); - ida_destroy(&nvme_fc_local_port_cnt); - ida_destroy(&nvme_fc_ctrl_cnt); - device_destroy(&fc_class, MKDEV(0, 0)); class_unregister(&fc_class); - destroy_workqueue(nvme_fc_wq); } module_init(nvme_fc_init_module);