From patchwork Thu May 4 09:12:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90021 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp174470vqo; Thu, 4 May 2023 02:34:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7b5fhuJ8O2iFnfmkY6gXp1roHI0RtdNlwdG0HqclCMdzw0GMGVv/cwzQEZTpM5NiZiXZa9 X-Received: by 2002:a05:6a20:1448:b0:f3:256:24d3 with SMTP id a8-20020a056a20144800b000f3025624d3mr2107593pzi.11.1683192856256; Thu, 04 May 2023 02:34:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192856; cv=none; d=google.com; s=arc-20160816; b=PZxAZt1W6//EH5jskZM/H+S2ob2ghoYBgvunMGR7XgLjzCStG7626Bc/l8xCO4sptd BnA8xW/ptjjhQsmhRV/pEet9qm0W+bp/Lb0rBGnMIllN8RO6PYgvgg43PzwzbeoVz8gY /Aht5ICkkVxUBDa9B9InuAmM10uXTKnMh4gb50j0GxZRbwkZMZ9cwj9v+S/bE1SCJvQ2 jU2OTtT0bNheVwUlyJipr+ho8LgvlLelvsS9VPATjHnEZtXDyw0mOV1LWlXyjlGCSTj0 /+JEsH9YCKFQ1UDD6BSqmJaec1AL7HLOHm9hncTRD2tbTVBsHz9IgPiC7Z8v/xj457+0 J8nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=FerwgbpotSBgm/052+82shHYk+aggnxdQyii9bK83Gk=; b=Gezt8zGXhV1ajMQa/EGUhzBEWV1P87e2dBTsVitPxCGY4KUddSULgU4P02RMG8UCYz H94xJjPlHjMS+FbG0NhCaZ/qR26lnFXCdf3Qk8JX3mHkCjg3BKBRxxPA+qg4UZrkI8vs LHDhamC+hHLZ0YUlsPXTTiuhWSgUsOgK8ULEmsfsBKPRuK7o+kMQCYS557e0KSGhvbdZ NVC7FE2GYa9+XRutXXG6MjF5+wOHFHiGYdik1iDYVntHP/KbbubEjycTZqogzQXiMiFY i5doCxM2TV9QqeKiiqY/FZFLnqVcprQ+iMLJNUrZYI5K5tn52A1HfIl6cFWVUgpjyucW yr9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=CrPIlzqI; dkim=neutral (no key) header.i=@suse.de header.b=Ofb0ixFj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m9-20020a654389000000b0052c4296d8casi6137983pgp.325.2023.05.04.02.34.03; Thu, 04 May 2023 02:34:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=CrPIlzqI; dkim=neutral (no key) header.i=@suse.de header.b=Ofb0ixFj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229900AbjEDJNJ (ORCPT + 99 others); Thu, 4 May 2023 05:13:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230139AbjEDJNG (ORCPT ); Thu, 4 May 2023 05:13:06 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D849448A for ; Thu, 4 May 2023 02:13:05 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 04CCA209A9; Thu, 4 May 2023 09:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191584; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FerwgbpotSBgm/052+82shHYk+aggnxdQyii9bK83Gk=; b=CrPIlzqI0JwtYTzc0Kz2O+wUUHm31AsHdIS0efgmOkmGE5tuwa4SttPz/hg+uwMdrBpuOJ L1YkWXADwCQvAKTbxYCtCyAJf/VoMGPga1uJqeOviE/mshs30MRMB839ZZ7U7vTFT2r9II r3ZVcwkiLAkg+g5yBhmRpiFEwLBjAwM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191584; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FerwgbpotSBgm/052+82shHYk+aggnxdQyii9bK83Gk=; b=Ofb0ixFjSghTjuGMOD2zvJj5ISEa1j2JhbRVo5tozg8WoiTZH7ZZ1UviLv6J8J/RhJ3r2S gzAvc1jojTCCyjDQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id EBF2713444; Thu, 4 May 2023 09:13:03 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id eieuOR93U2TYTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:03 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 1/9] nvme-rdma: stream line queue functions arguments Date: Thu, 4 May 2023 11:12:51 +0200 Message-Id: <20230504091259.29100-2-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955632463264951?= X-GMAIL-MSGID: =?utf-8?q?1764955632463264951?= In preparation to move common code from the fabrics driver to fabrics.c, we stream line the low level functions. This allows any common code just to pass in nvme subsystem global types, such as 'struct nvme_ctrl' instead of the driver specialized types 'struct nvme_rdma_ctrl'. Signed-off-by: Daniel Wagner --- drivers/nvme/host/rdma.c | 62 ++++++++++++++++++++++++++-------------- 1 file changed, 40 insertions(+), 22 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 0eb79696fb73..92e5d0ccf3a9 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -132,6 +132,11 @@ static inline struct nvme_rdma_ctrl *to_rdma_ctrl(struct nvme_ctrl *ctrl) return container_of(ctrl, struct nvme_rdma_ctrl, ctrl); } +static inline int nvme_rdma_queue_id(struct nvme_rdma_queue *queue) +{ + return queue - queue->ctrl->queues; +} + static LIST_HEAD(device_list); static DEFINE_MUTEX(device_list_mutex); @@ -566,13 +571,19 @@ static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue) return ret; } -static int nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, - int idx, size_t queue_size) +static int nvme_rdma_alloc_queue(struct nvme_ctrl *nctrl, int idx) { + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); struct nvme_rdma_queue *queue; struct sockaddr *src_addr = NULL; + size_t queue_size; int ret; + if (idx == 0) + queue_size = NVME_AQ_DEPTH; + else + queue_size = ctrl->ctrl.sqsize + 1; + queue = &ctrl->queues[idx]; mutex_init(&queue->queue_lock); queue->ctrl = ctrl; @@ -636,16 +647,22 @@ static void __nvme_rdma_stop_queue(struct nvme_rdma_queue *queue) ib_drain_qp(queue->qp); } -static void nvme_rdma_stop_queue(struct nvme_rdma_queue *queue) +static void nvme_rdma_stop_queue(struct nvme_ctrl *nctrl, int qid) { + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[qid]; + mutex_lock(&queue->queue_lock); if (test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags)) __nvme_rdma_stop_queue(queue); mutex_unlock(&queue->queue_lock); } -static void nvme_rdma_free_queue(struct nvme_rdma_queue *queue) +static void nvme_rdma_free_queue(struct nvme_ctrl *nctrl, int qid) { + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[qid]; + if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) return; @@ -659,7 +676,7 @@ static void nvme_rdma_free_io_queues(struct nvme_rdma_ctrl *ctrl) int i; for (i = 1; i < ctrl->ctrl.queue_count; i++) - nvme_rdma_free_queue(&ctrl->queues[i]); + nvme_rdma_free_queue(&ctrl->ctrl, i); } static void nvme_rdma_stop_io_queues(struct nvme_rdma_ctrl *ctrl) @@ -667,18 +684,19 @@ static void nvme_rdma_stop_io_queues(struct nvme_rdma_ctrl *ctrl) int i; for (i = 1; i < ctrl->ctrl.queue_count; i++) - nvme_rdma_stop_queue(&ctrl->queues[i]); + nvme_rdma_stop_queue(&ctrl->ctrl, i); } -static int nvme_rdma_start_queue(struct nvme_rdma_ctrl *ctrl, int idx) +static int nvme_rdma_start_queue(struct nvme_ctrl *nctrl, int idx) { + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); struct nvme_rdma_queue *queue = &ctrl->queues[idx]; int ret; if (idx) - ret = nvmf_connect_io_queue(&ctrl->ctrl, idx); + ret = nvmf_connect_io_queue(nctrl, idx); else - ret = nvmf_connect_admin_queue(&ctrl->ctrl); + ret = nvmf_connect_admin_queue(nctrl); if (!ret) { set_bit(NVME_RDMA_Q_LIVE, &queue->flags); @@ -697,7 +715,7 @@ static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl, int i, ret = 0; for (i = first; i < last; i++) { - ret = nvme_rdma_start_queue(ctrl, i); + ret = nvme_rdma_start_queue(&ctrl->ctrl, i); if (ret) goto out_stop_queues; } @@ -706,7 +724,7 @@ static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl, out_stop_queues: for (i--; i >= first; i--) - nvme_rdma_stop_queue(&ctrl->queues[i]); + nvme_rdma_stop_queue(&ctrl->ctrl, i); return ret; } @@ -768,8 +786,7 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) } for (i = 1; i < ctrl->ctrl.queue_count; i++) { - ret = nvme_rdma_alloc_queue(ctrl, i, - ctrl->ctrl.sqsize + 1); + ret = nvme_rdma_alloc_queue(&ctrl->ctrl, i); if (ret) goto out_free_queues; } @@ -778,7 +795,7 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) out_free_queues: for (i--; i >= 1; i--) - nvme_rdma_free_queue(&ctrl->queues[i]); + nvme_rdma_free_queue(&ctrl->ctrl, i); return ret; } @@ -806,7 +823,7 @@ static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl) sizeof(struct nvme_command), DMA_TO_DEVICE); ctrl->async_event_sqe.data = NULL; } - nvme_rdma_free_queue(&ctrl->queues[0]); + nvme_rdma_free_queue(&ctrl->ctrl, 0); } static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, @@ -815,7 +832,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, bool pi_capable = false; int error; - error = nvme_rdma_alloc_queue(ctrl, 0, NVME_AQ_DEPTH); + error = nvme_rdma_alloc_queue(&ctrl->ctrl, 0); if (error) return error; @@ -850,7 +867,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, } - error = nvme_rdma_start_queue(ctrl, 0); + error = nvme_rdma_start_queue(&ctrl->ctrl, 0); if (error) goto out_remove_admin_tag_set; @@ -877,7 +894,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, nvme_quiesce_admin_queue(&ctrl->ctrl); blk_sync_queue(ctrl->ctrl.admin_q); out_stop_queue: - nvme_rdma_stop_queue(&ctrl->queues[0]); + nvme_rdma_stop_queue(&ctrl->ctrl, 0); nvme_cancel_admin_tagset(&ctrl->ctrl); out_remove_admin_tag_set: if (new) @@ -889,7 +906,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, ctrl->async_event_sqe.data = NULL; } out_free_queue: - nvme_rdma_free_queue(&ctrl->queues[0]); + nvme_rdma_free_queue(&ctrl->ctrl, 0); return error; } @@ -962,7 +979,7 @@ static void nvme_rdma_teardown_admin_queue(struct nvme_rdma_ctrl *ctrl, { nvme_quiesce_admin_queue(&ctrl->ctrl); blk_sync_queue(ctrl->ctrl.admin_q); - nvme_rdma_stop_queue(&ctrl->queues[0]); + nvme_rdma_stop_queue(&ctrl->ctrl, 0); nvme_cancel_admin_tagset(&ctrl->ctrl); if (remove) { nvme_unquiesce_admin_queue(&ctrl->ctrl); @@ -1113,7 +1130,7 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) destroy_admin: nvme_quiesce_admin_queue(&ctrl->ctrl); blk_sync_queue(ctrl->ctrl.admin_q); - nvme_rdma_stop_queue(&ctrl->queues[0]); + nvme_rdma_stop_queue(&ctrl->ctrl, 0); nvme_cancel_admin_tagset(&ctrl->ctrl); if (new) nvme_remove_admin_tag_set(&ctrl->ctrl); @@ -1960,9 +1977,10 @@ static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id, static void nvme_rdma_complete_timed_out(struct request *rq) { struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); + struct nvme_ctrl *ctrl = &req->queue->ctrl->ctrl; struct nvme_rdma_queue *queue = req->queue; - nvme_rdma_stop_queue(queue); + nvme_rdma_stop_queue(ctrl, nvme_rdma_queue_id(queue)); nvmf_complete_timed_out_request(rq); } From patchwork Thu May 4 09:12:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90023 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp174817vqo; Thu, 4 May 2023 02:34:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4KbDJIiFp+zCLWbgd6J5/z0UkKANDGEh0QFCYs4HyUHp5HlmAHo02+7ofTI7HL2ATNIEgi X-Received: by 2002:a05:6a20:244c:b0:f3:5cf7:581b with SMTP id t12-20020a056a20244c00b000f35cf7581bmr2018655pzc.28.1683192899046; Thu, 04 May 2023 02:34:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192899; cv=none; d=google.com; s=arc-20160816; b=W6srWfnXB5QyOSvCa5W2iSH7CYwuJU+0cp5M+HcrBfIgZbX1rHmaNq+lKjKGlZx9Ex 8SG2i6Ddtz9y7WTr4LhX31VcMj06ZJ6o/l6GIYbW4tDzLmxNut0Vany1xAV27UIX8DN/ 7TX7WcYZ/EAaW27bfLwlgAu8IntdQME5kmQ6IITARSVD+rvx8bOGcWr2fuyYXEgAP06X enc1NFdxnShwB5RL6MwPod6jW0F1oxM9HJJ8ZUykpmuzEP/EYi6SFqDwZwKg/QzGCaGf /TBfLZv0yNJY1yTSzf41qv7xGDZcHFYA3ugRwDYnDpzkkY615Ey87WAnLtNEbBFbhT2e PLOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=ouaTATkWNixyD5d3su+1xUw5y9Ym8ke7nX0uHaKZbUg=; b=fGSVWLdZuRIuDD50GtQkvNgzRBO05V/h/E/NraQB/DHz2mjbirrAmj0S/V/eUORRje Bu9uzvaZpTZiSlj4n5txd0eQhPxge/2Pohe/0/IMaAm/Hg30Yht0bFozQwVwru+xXCeW F+YyqfmwuGjNxyN1qTv+Ufpuoz+3NEICLZUZuNg3E0QRoXO06hYZCnKUnuIg6xJPqRCH xPtIBMbXLiGkmkGCL5pafqRXcJWJHelfOVEezxAOFqV1Ze/wRi/omZqVRPEVJKgCEDRW QJYi9bk/Ea+KC1KPAqmfKsumuha6VNCFxjp0Kp4xuFQRyJDYerVYqnz2KFmyEGpQk5hD NlcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=lfPbEvfU; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=Bf3gHwkK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c1-20020a656181000000b0050fb1a00d3dsi34053192pgv.46.2023.05.04.02.34.45; Thu, 04 May 2023 02:34:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=lfPbEvfU; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=Bf3gHwkK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229681AbjEDJNN (ORCPT + 99 others); Thu, 4 May 2023 05:13:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230253AbjEDJNH (ORCPT ); Thu, 4 May 2023 05:13:07 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 328483ABB for ; Thu, 4 May 2023 02:13:05 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9125E209AC; Thu, 4 May 2023 09:13:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191584; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ouaTATkWNixyD5d3su+1xUw5y9Ym8ke7nX0uHaKZbUg=; b=lfPbEvfUwuVA73OXON1Y3/CaMNOfQEYaN5z9Ofcg0pBr2tCcLTpl5MzIxGuhU12MNFpYzh xL4eudbc8J5zY8e7wm93dJLtdojTazAKTyDf1siXPyf2r2OEYAbIA3uCIVfUvd482/Ys0x r82wRFeaSiaXFwh2agTkJ5RkeaH3df0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191584; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ouaTATkWNixyD5d3su+1xUw5y9Ym8ke7nX0uHaKZbUg=; b=Bf3gHwkKJIj1Xi3QkNnUxG7kM+ITdAVO2udEz1R/u5EMBeVCRc0kPXhOyElYn3CHARbBdO O+1XjbxY0cK6qXCQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 82CAE13444; Thu, 4 May 2023 09:13:04 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id AlzpHyB3U2TaTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:04 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 2/9] nvme-rdma: factor rdma specific queue init code out Date: Thu, 4 May 2023 11:12:52 +0200 Message-Id: <20230504091259.29100-3-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955676905855360?= X-GMAIL-MSGID: =?utf-8?q?1764955676905855360?= In preparation to move common code from the fabrics driver to fabrics.c, move the rmda queue specific initialization code into a separate function. Signed-off-by: Daniel Wagner --- drivers/nvme/host/rdma.c | 65 ++++++++++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 19 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 92e5d0ccf3a9..a78c66278b19 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -826,15 +826,16 @@ static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl) nvme_rdma_free_queue(&ctrl->ctrl, 0); } -static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, - bool new) +static int nvme_rdma_init_queue(struct nvme_ctrl *nctrl, int qid) { + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); bool pi_capable = false; int error; - error = nvme_rdma_alloc_queue(&ctrl->ctrl, 0); - if (error) - return error; + if (qid != 0) + /* only admin queue needs additional work. */ + return 0; + ctrl->device = ctrl->queues[0].device; ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev); @@ -854,6 +855,43 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, */ error = nvme_rdma_alloc_qe(ctrl->device->dev, &ctrl->async_event_sqe, sizeof(struct nvme_command), DMA_TO_DEVICE); + if (error) + return error; + + ctrl->ctrl.max_segments = ctrl->max_fr_pages; + ctrl->ctrl.max_hw_sectors = ctrl->max_fr_pages << (ilog2(SZ_4K) - 9); + if (pi_capable) + ctrl->ctrl.max_integrity_segments = ctrl->max_fr_pages; + else + ctrl->ctrl.max_integrity_segments = 0; + + return 0; +} + +static void nvme_rdma_deinit_queue(struct nvme_ctrl *nctrl, int qid) +{ + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + + if (qid != 0) + return; + + if (ctrl->async_event_sqe.data) { + nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe, + sizeof(struct nvme_command), DMA_TO_DEVICE); + ctrl->async_event_sqe.data = NULL; + } +} + +static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, + bool new) +{ + int error; + + error = nvme_rdma_alloc_queue(&ctrl->ctrl, 0); + if (error) + return error; + + error = nvme_rdma_init_queue(&ctrl->ctrl, 0); if (error) goto out_free_queue; @@ -863,7 +901,7 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, sizeof(struct nvme_rdma_request) + NVME_RDMA_DATA_SGL_SIZE); if (error) - goto out_free_async_qe; + goto out_deinit_admin_queue; } @@ -875,13 +913,6 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, if (error) goto out_stop_queue; - ctrl->ctrl.max_segments = ctrl->max_fr_pages; - ctrl->ctrl.max_hw_sectors = ctrl->max_fr_pages << (ilog2(SZ_4K) - 9); - if (pi_capable) - ctrl->ctrl.max_integrity_segments = ctrl->max_fr_pages; - else - ctrl->ctrl.max_integrity_segments = 0; - nvme_unquiesce_admin_queue(&ctrl->ctrl); error = nvme_init_ctrl_finish(&ctrl->ctrl, false); @@ -899,12 +930,8 @@ static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, out_remove_admin_tag_set: if (new) nvme_remove_admin_tag_set(&ctrl->ctrl); -out_free_async_qe: - if (ctrl->async_event_sqe.data) { - nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe, - sizeof(struct nvme_command), DMA_TO_DEVICE); - ctrl->async_event_sqe.data = NULL; - } +out_deinit_admin_queue: + nvme_rdma_deinit_queue(&ctrl->ctrl, 0); out_free_queue: nvme_rdma_free_queue(&ctrl->ctrl, 0); return error; From patchwork Thu May 4 09:12:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90016 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp173211vqo; Thu, 4 May 2023 02:31:33 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4pmtbAV54/ujLt4RBiq++PghHU+h44Xn41ClSACozxcHT1RTdUFy+iRG+x+aDFbjpLrn6Q X-Received: by 2002:a17:90a:d082:b0:23f:7d05:8762 with SMTP id k2-20020a17090ad08200b0023f7d058762mr1429191pju.23.1683192692963; Thu, 04 May 2023 02:31:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192692; cv=none; d=google.com; s=arc-20160816; b=E4zGeAUbOBjZm+yyq6ppFPrHFGeIRxfewKV84fo8hPBaX4EQxliNORelC/2owqMzVV ymLtb7LZQ8hJkquFfZmzPxmRgaxxFKEZ8hsyLDI4I2iSMqlx3o5oCVBpnrfkqAfCxSRw QJ/43hrw3/CJC9c05AR0GbvdunpI+evsw48lIDLeNEBxPOkVeRNynNAySlkubHoC7mJQ 62HARkJVwBFBSk44/+aqsB1KNNg3XIdSQskzVJ6K8+/yYgcQUejoe1KdFQZk3ZhDbMpp 986HfOvs0yfN0xrM9jTHraaaaRo4jIxueg4HCVvGiEJagkqYj+8YHkDK+9D4XGcNLhwh M1iA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=CJg6fOVMJfQiPdvfQeCkj1a2vcDOvIJAvZO16+3IyrU=; b=ZF/9GSWq6bSdwHTp6dSh2Nzs7YIbMyMpv7EfLsv0eYobHLC346GPzgftd4qFJTaK7A iCE1sKUri9aYiITNCxR2G+fufde0NwsE2oMAgfppkL6Ci0AyaBUYYmc0fF9/i5Ja2cvf iLtpVp6ARVQg6VuR6xv5R3RFMB3uscG+/DuXCCYyGEEMpMBug5fv6xDes4fXfrcw5ug8 mnczxvBalBFf0/+3T3c368/nzqA9xk1gxbiv5D6nEMLEW6rk9v/jYocHGeMRK5LovaKF GYP0r349M7oCnWAE9sH1+VeTSm+tCS6LLjZnh3+X+VI8QE3xkywiLYLdkmYdjeYt9L3k FNog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Pr9hQVSZ; dkim=neutral (no key) header.i=@suse.de header.b=QkTpdXmj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ja10-20020a170902efca00b001a97a54c3a2si2202276plb.197.2023.05.04.02.31.14; Thu, 04 May 2023 02:31:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=Pr9hQVSZ; dkim=neutral (no key) header.i=@suse.de header.b=QkTpdXmj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229822AbjEDJNQ (ORCPT + 99 others); Thu, 4 May 2023 05:13:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230265AbjEDJNI (ORCPT ); Thu, 4 May 2023 05:13:08 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E8E740C6 for ; Thu, 4 May 2023 02:13:06 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2998B33932; Thu, 4 May 2023 09:13:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191585; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CJg6fOVMJfQiPdvfQeCkj1a2vcDOvIJAvZO16+3IyrU=; b=Pr9hQVSZbxMqN2Kj/HQTRlKJ3TopK+kfGC9pt5fNTVIM+yUvM7ZlLRVz78SUbE8QnKM6Oi N5cLwrhWIPR0Bs2XxgIwZheqHiBld+msx3KjgLs2HrGOGvrYq7AIoA6cJ6sVZNlwLDETiO 2vpmAylR/hBQ0BmZCnuWIqt/CoAqfDE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191585; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CJg6fOVMJfQiPdvfQeCkj1a2vcDOvIJAvZO16+3IyrU=; b=QkTpdXmjR+kILeHfcRC22Wutk/+VTUuDQ0PmFc/leZgtuO7IOqIXuW8E9yEzq1ZayC/zHA WpjCqKNrljDvCHCg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1B4C013444; Thu, 4 May 2023 09:13:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 1UuoBiF3U2TcTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:05 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 3/9] nvme-tcp: move error and connect work to nvme_ctrl Date: Thu, 4 May 2023 11:12:53 +0200 Message-Id: <20230504091259.29100-4-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955461500652911?= X-GMAIL-MSGID: =?utf-8?q?1764955461500652911?= Move common data structures for fabrics to nvme_ctrl so that we are able to use them in fabrcis.c later. Signed-off-by: Daniel Wagner --- drivers/nvme/host/nvme.h | 3 +++ drivers/nvme/host/tcp.c | 24 ++++++++++-------------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index bf46f122e9e1..5aa30b00dd17 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -339,6 +339,9 @@ struct nvme_ctrl { struct work_struct ana_work; #endif + struct work_struct err_work; + struct delayed_work connect_work; + #ifdef CONFIG_NVME_AUTH struct work_struct dhchap_auth_work; struct mutex dhchap_auth_mutex; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 273c1f2760a4..74ccc84d244a 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -166,8 +166,6 @@ struct nvme_tcp_ctrl { struct sockaddr_storage src_addr; struct nvme_ctrl ctrl; - struct work_struct err_work; - struct delayed_work connect_work; struct nvme_tcp_request async_req; u32 io_queues[HCTX_MAX_TYPES]; }; @@ -527,7 +525,7 @@ static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) return; dev_warn(ctrl->device, "starting error recovery\n"); - queue_work(nvme_reset_wq, &to_tcp_ctrl(ctrl)->err_work); + queue_work(nvme_reset_wq, &ctrl->err_work); } static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, @@ -2025,7 +2023,7 @@ static void nvme_tcp_reconnect_or_remove(struct nvme_ctrl *ctrl) if (nvmf_should_reconnect(ctrl)) { dev_info(ctrl->device, "Reconnecting in %d seconds...\n", ctrl->opts->reconnect_delay); - queue_delayed_work(nvme_wq, &to_tcp_ctrl(ctrl)->connect_work, + queue_delayed_work(nvme_wq, &ctrl->connect_work, ctrl->opts->reconnect_delay * HZ); } else { dev_info(ctrl->device, "Removing controller...\n"); @@ -2107,9 +2105,8 @@ static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new) static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) { - struct nvme_tcp_ctrl *tcp_ctrl = container_of(to_delayed_work(work), - struct nvme_tcp_ctrl, connect_work); - struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; + struct nvme_ctrl *ctrl = container_of(to_delayed_work(work), + struct nvme_ctrl, connect_work); ++ctrl->nr_reconnects; @@ -2131,9 +2128,8 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) static void nvme_tcp_error_recovery_work(struct work_struct *work) { - struct nvme_tcp_ctrl *tcp_ctrl = container_of(work, - struct nvme_tcp_ctrl, err_work); - struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; + struct nvme_ctrl *ctrl = container_of(work, + struct nvme_ctrl, err_work); nvme_stop_keep_alive(ctrl); flush_work(&ctrl->async_event_work); @@ -2194,8 +2190,8 @@ static void nvme_reset_ctrl_work(struct work_struct *work) static void nvme_tcp_stop_ctrl(struct nvme_ctrl *ctrl) { - flush_work(&to_tcp_ctrl(ctrl)->err_work); - cancel_delayed_work_sync(&to_tcp_ctrl(ctrl)->connect_work); + flush_work(&ctrl->err_work); + cancel_delayed_work_sync(&ctrl->connect_work); } static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl) @@ -2581,9 +2577,9 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, ctrl->ctrl.sqsize = opts->queue_size - 1; ctrl->ctrl.kato = opts->kato; - INIT_DELAYED_WORK(&ctrl->connect_work, + INIT_DELAYED_WORK(&ctrl->ctrl.connect_work, nvme_tcp_reconnect_ctrl_work); - INIT_WORK(&ctrl->err_work, nvme_tcp_error_recovery_work); + INIT_WORK(&ctrl->ctrl.err_work, nvme_tcp_error_recovery_work); INIT_WORK(&ctrl->ctrl.reset_work, nvme_reset_ctrl_work); if (!(opts->mask & NVMF_OPT_TRSVCID)) { From patchwork Thu May 4 09:12:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90011 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp169687vqo; Thu, 4 May 2023 02:23:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7OZsEGuSp1vyf/R4bFJmcySdgBPUBuHy0HXZlxSIsNYgTqdU8rfsyt2o4l/VkupS2ITOoe X-Received: by 2002:a05:6a20:d904:b0:db:7055:d4e9 with SMTP id jd4-20020a056a20d90400b000db7055d4e9mr1864532pzb.27.1683192196710; Thu, 04 May 2023 02:23:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192196; cv=none; d=google.com; s=arc-20160816; b=J1ru04H7yGO2wJ/lW6uweoRyXsgzMEyn1Xkt86lOTVQ3Ox3m9Ywi6hrXhXZVJhwgvV YT9Nzo1ATpHIGaj8zaFHf8CYWqDnMXvXbjSy7YM+tifhFwfzS5J8ks2mhbkfvj4RdUNm Z8ouN3z6ILJOSQbOck5jtnNDQqH4GndqBTQiK6A1pAkH5Vpe5+OCM4fVZvpEnotgHmDv ZgEemHrmSUyuHktENlYqzSORLsosm3ZyTpeIAF5bd85tx7raz/vzrJ10yNkc42EFHerC i7E+itcOEh9+w6dt07nm1z8glUNLt8aVFsyQvmYms692aYil7S2nv6hOFtAf5A324sFn 6xxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=IZ0cClgnu8/kTFVRND7u+HMm+91w0te/Vhixjfj7uYY=; b=XxVlxu3EHXa4y09Yzau5IyeLoSuLRSpwRXROvH3T9hLtoD3OS4vxeTT1LAWs/dINa2 sqvp7fVa7+jFuFkPBPiNjgSdZrGG/T+xXC7o1Oz7CLTzTvEbdFAt0ojTv6+P9TXdBptX UlIkkuNVcRaj5fxH6hDm5zyo+U69b0QUnMH5ooFzrA/IM1iG2GEsH+CxFq0QaTBkuNtP 0SfZsqxsZ64Y/P/HkGdZyqrYP9vUdBrUgo8gzidekWlHQn12AoJsl1SIgugtJzfo9Gdo BsuHPq1mzP6MJDw6KEJVSnoRmaNme5/CTZp5FZDi7qnIMTAQFfKNL7BMzVWKmTMqmsN4 2Ucw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b="Us//lxeg"; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=DHe+CDDN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 82-20020a621755000000b0063d48d82e85si451873pfx.15.2023.05.04.02.23.02; Thu, 04 May 2023 02:23:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b="Us//lxeg"; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=DHe+CDDN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230265AbjEDJNT (ORCPT + 99 others); Thu, 4 May 2023 05:13:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230284AbjEDJNI (ORCPT ); Thu, 4 May 2023 05:13:08 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00F324497 for ; Thu, 4 May 2023 02:13:06 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AEA66209AB; Thu, 4 May 2023 09:13:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191585; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IZ0cClgnu8/kTFVRND7u+HMm+91w0te/Vhixjfj7uYY=; b=Us//lxegiQ+pNUvT0WoLCczQmKTkH6RAwHncUN5IfoCVQnTgp+9WGZF8XJr/bgIK6lJQP/ tw1FhR22z35sL/EB+DMcg8CMf6JY9GbY38AvUJMbhqLatlWF6sJpvOhaPsterX7JEg7bU9 Ff2t/VaHiODUS0XVjjDKUaXr3NodOZg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191585; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IZ0cClgnu8/kTFVRND7u+HMm+91w0te/Vhixjfj7uYY=; b=DHe+CDDNgUcq91cspWZJQ0DTo81yecLB9XtVMmP6Xug8X8HL56VgiGAuSQSUnNq6HriJBx MLo0BCDbyyWaZyAg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A0B6413444; Thu, 4 May 2023 09:13:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id wdJQJyF3U2TgTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:05 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 4/9] nvme-rdma: use error and connect work from nvme_ctrl Date: Thu, 4 May 2023 11:12:54 +0200 Message-Id: <20230504091259.29100-5-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764954941015748256?= X-GMAIL-MSGID: =?utf-8?q?1764954941015748256?= Use common data structures from nvme_ctrl so that we are able to use them in fabrics.c later. Signed-off-by: Daniel Wagner --- drivers/nvme/host/rdma.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index a78c66278b19..b0ab5a9d5fe0 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -106,12 +106,9 @@ struct nvme_rdma_ctrl { /* other member variables */ struct blk_mq_tag_set tag_set; - struct work_struct err_work; struct nvme_rdma_qe async_event_sqe; - struct delayed_work reconnect_work; - struct list_head list; struct blk_mq_tag_set admin_tag_set; @@ -1036,8 +1033,8 @@ static void nvme_rdma_stop_ctrl(struct nvme_ctrl *nctrl) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - flush_work(&ctrl->err_work); - cancel_delayed_work_sync(&ctrl->reconnect_work); + flush_work(&ctrl->ctrl.err_work); + cancel_delayed_work_sync(&ctrl->ctrl.connect_work); } static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) @@ -1069,7 +1066,7 @@ static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl) if (nvmf_should_reconnect(&ctrl->ctrl)) { dev_info(ctrl->ctrl.device, "Reconnecting in %d seconds...\n", ctrl->ctrl.opts->reconnect_delay); - queue_delayed_work(nvme_wq, &ctrl->reconnect_work, + queue_delayed_work(nvme_wq, &ctrl->ctrl.connect_work, ctrl->ctrl.opts->reconnect_delay * HZ); } else { nvme_delete_ctrl(&ctrl->ctrl); @@ -1167,8 +1164,9 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) { - struct nvme_rdma_ctrl *ctrl = container_of(to_delayed_work(work), - struct nvme_rdma_ctrl, reconnect_work); + struct nvme_ctrl *nctrl = container_of(to_delayed_work(work), + struct nvme_ctrl, connect_work); + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); ++ctrl->ctrl.nr_reconnects; @@ -1190,8 +1188,9 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) static void nvme_rdma_error_recovery_work(struct work_struct *work) { - struct nvme_rdma_ctrl *ctrl = container_of(work, - struct nvme_rdma_ctrl, err_work); + struct nvme_ctrl *nctrl = container_of(work, + struct nvme_ctrl, err_work); + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); nvme_stop_keep_alive(&ctrl->ctrl); flush_work(&ctrl->ctrl.async_event_work); @@ -1217,7 +1216,7 @@ static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl) return; dev_warn(ctrl->ctrl.device, "starting error recovery\n"); - queue_work(nvme_reset_wq, &ctrl->err_work); + queue_work(nvme_reset_wq, &ctrl->ctrl.err_work); } static void nvme_rdma_end_request(struct nvme_rdma_request *req) @@ -2369,9 +2368,9 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, goto out_free_ctrl; } - INIT_DELAYED_WORK(&ctrl->reconnect_work, + INIT_DELAYED_WORK(&ctrl->ctrl.connect_work, nvme_rdma_reconnect_ctrl_work); - INIT_WORK(&ctrl->err_work, nvme_rdma_error_recovery_work); + INIT_WORK(&ctrl->ctrl.err_work, nvme_rdma_error_recovery_work); INIT_WORK(&ctrl->ctrl.reset_work, nvme_rdma_reset_ctrl_work); ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues + From patchwork Thu May 4 09:12:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90015 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp172992vqo; Thu, 4 May 2023 02:31:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7DFvPZXo5TXyHBSUfu29PbMT0iRkBI2svhEdZXd2hEHU1DxumdvhyaFSaxEOPoscJlui1V X-Received: by 2002:a05:6a20:a395:b0:f2:bbf:f5f4 with SMTP id w21-20020a056a20a39500b000f20bbff5f4mr1422964pzk.52.1683192667784; Thu, 04 May 2023 02:31:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192667; cv=none; d=google.com; s=arc-20160816; b=mS4WkeiRRo7bBF2xKzrl+EOOO1EgHenmj29C0vD+kdb8ZdjVBtcyHgBriBepbOYLDP 4IWhd6h1i7V4ka9DJZcVj7wlq8wen95RqGsqMrcR5uF4wcYTtYkHLxXicNCAdWxU3yWI iW571mQboUAIhofXmJRq2611C73E/Qy04GR6Rq5j9Wdy+405dUEoMh75SoN6j3UZWQlk 9FHCVMJ6ty8owwHMpnWi3FKa5vfIK+5d9eaBv6bBg1PDvcSfTqJarEmAnWkpOQVqUcql RqAQuHPL7z2dHA0X5qCoGTcXHyOH6ssBsd/TOs4QfDAziT1Y7Mpxeq2IkBXZKj6S55aM loXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=jKAGQVKaUUQJ0wwOeUiQWcm9V9pH23Rwd+Th8ukWdzU=; b=ds3fuR025EtC2Lkx2Bq2eb8v2NdA041ZDaPeHRoIaI5vcUT860XJU8BzQ/V04Nowg8 eEs4mqvOZ0DxOdPV1f6ZfwkQHtRnxmZIZPMQY0nS+WDQzINzbcVMrhvckkDTiir5vgOq XxgxzVUMHWCeO7X5CkBcShrl+UtUp+P8dCOxJO5aV2IFe7Eh/ybRcX35NKvQ82m4Q7Li Tmk+z23nyBIDW2Y774gYNw1UiWpSwoLjg+IuHUBBTpXQidIcOBQXNmHqpcw9zb2tNqr1 LZ+SCBgRGkVNViomx9pwB2mI0KCKdiC26+nVBCyDCepmGWyyKiOUUvP+NfBieYhO3zNn M0eg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=uZ7bOMI9; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=LH63OfEV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w4-20020a656944000000b0050bcb8e3a8csi34018524pgq.828.2023.05.04.02.30.52; Thu, 04 May 2023 02:31:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=uZ7bOMI9; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=LH63OfEV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230354AbjEDJNY (ORCPT + 99 others); Thu, 4 May 2023 05:13:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230299AbjEDJNL (ORCPT ); Thu, 4 May 2023 05:13:11 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A9BD110E7 for ; Thu, 4 May 2023 02:13:07 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 4865E209AD; Thu, 4 May 2023 09:13:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191586; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jKAGQVKaUUQJ0wwOeUiQWcm9V9pH23Rwd+Th8ukWdzU=; b=uZ7bOMI9miJeH5qI7b8UnbZiQtsJRiarmxjqlTk8c1zns69jtqB/bv3YPHuV5iirx2CoUY sPkM+Q6HUbKXOjwJCdgsePKr1vlWmSHja70KZTd5IR0xh6fya6s6V1ARydcTqarO4gO0gi r3umVQxI2RJhXaSHceoJAMk09U9O4zg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191586; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jKAGQVKaUUQJ0wwOeUiQWcm9V9pH23Rwd+Th8ukWdzU=; b=LH63OfEVGSOWMN6H8Zcd3tK7gqcA43wPFLUvAvrEzLgZDGTBUeq/E+AZPlAZcPyVbxxKLg urTl3ro1kQyBg+CA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 3AD6F13444; Thu, 4 May 2023 09:13:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id y4dUDiJ3U2TkTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:06 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 5/9] nvme-fabrics: add fabric state machine Date: Thu, 4 May 2023 11:12:55 +0200 Message-Id: <20230504091259.29100-6-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955434912197214?= X-GMAIL-MSGID: =?utf-8?q?1764955434912197214?= The transports are sharing a lot of common code for the state machine. Add a generic static machine based on tcp transport. In this first step additional callbacks such as alloc_admin_tag_set() are in the callback API. These will be remove later again. This is just a for making the areas where more than moving code around is necessary a bit easier to review. This is approach just for discussion purpose and the proper series wont have these intermediate steps. I suppose, the later steps in this series would go in before the main state machine. Signed-off-by: Daniel Wagner --- drivers/nvme/host/fabrics.c | 426 ++++++++++++++++++++++++++++++++++++ drivers/nvme/host/fabrics.h | 25 +++ drivers/nvme/host/nvme.h | 3 + 3 files changed, 454 insertions(+) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index bbaa04a0c502..3d2cde17338d 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -1134,6 +1134,432 @@ nvmf_create_ctrl(struct device *dev, const char *buf) return ERR_PTR(ret); } +static int nvmf_start_io_queues(struct nvme_ctrl *ctrl, + int first, int last) +{ + int i, ret; + + for (i = first; i < last; i++) { + ret = ctrl->fabrics_ops->start_io_queue(ctrl, i); + if (ret) + goto out_stop_queues; + } + + return 0; + +out_stop_queues: + for (i--; i >= first; i--) + ctrl->fabrics_ops->stop_io_queue(ctrl, i); + return ret; +} + +static void nvmf_stop_io_queues(struct nvme_ctrl *ctrl) +{ + int i; + + for (i = 1; i < ctrl->queue_count; i++) + ctrl->fabrics_ops->stop_io_queue(ctrl, i); +} + +static int __nvmf_alloc_io_queues(struct nvme_ctrl *ctrl) +{ + int i, ret; + + for (i = 1; i < ctrl->queue_count; i++) { + ret = ctrl->fabrics_ops->alloc_io_queue(ctrl, i); + if (ret) + goto out_free_queues; + } + + return 0; + +out_free_queues: + for (i--; i >= 1; i--) + ctrl->fabrics_ops->free_io_queue(ctrl, i); + + return ret; +} + +static int nvmf_alloc_io_queues(struct nvme_ctrl *ctrl) +{ + unsigned int nr_io_queues; + int ret; + + nr_io_queues = ctrl->fabrics_ops->nr_io_queues(ctrl); + ret = nvme_set_queue_count(ctrl, &nr_io_queues); + if (ret) + return ret; + + if (nr_io_queues == 0) { + dev_err(ctrl->device, + "unable to set any I/O queues\n"); + return -ENOMEM; + } + + ctrl->queue_count = nr_io_queues + 1; + dev_info(ctrl->device, + "creating %d I/O queues.\n", nr_io_queues); + + ctrl->fabrics_ops->set_io_queues(ctrl, nr_io_queues); + + return __nvmf_alloc_io_queues(ctrl); +} + +static void nvmf_free_io_queues(struct nvme_ctrl *ctrl) +{ + int i; + + for (i = 1; i < ctrl->queue_count; i++) + ctrl->fabrics_ops->free_io_queue(ctrl, i); +} + +static int nvmf_configure_io_queues(struct nvme_ctrl *ctrl, bool new) +{ + int ret, nr_queues; + + ret = nvmf_alloc_io_queues(ctrl); + if (ret) + return ret; + + if (new) { + ret = ctrl->fabrics_ops->alloc_tag_set(ctrl); + if (ret) + goto out_free_io_queues; + } + + /* + * Only start IO queues for which we have allocated the tagset + * and limitted it to the available queues. On reconnects, the + * queue number might have changed. + */ + nr_queues = min(ctrl->tagset->nr_hw_queues + 1, ctrl->queue_count); + ret = nvmf_start_io_queues(ctrl, 1, nr_queues); + if (ret) + goto out_cleanup_connect_q; + + if (!new) { + nvme_unquiesce_io_queues(ctrl); + if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) { + /* + * If we timed out waiting for freeze we are likely to + * be stuck. Fail the controller initialization just + * to be safe. + */ + ret = -ENODEV; + goto out_wait_freeze_timed_out; + } + blk_mq_update_nr_hw_queues(ctrl->tagset, + ctrl->queue_count - 1); + nvme_unfreeze(ctrl); + } + + /* + * If the number of queues has increased (reconnect case) + * start all new queues now. + */ + ret = nvmf_start_io_queues(ctrl, nr_queues, + ctrl->tagset->nr_hw_queues + 1); + if (ret) + goto out_wait_freeze_timed_out; + + return 0; + +out_wait_freeze_timed_out: + nvme_quiesce_io_queues(ctrl); + nvme_sync_io_queues(ctrl); + nvmf_stop_io_queues(ctrl); +out_cleanup_connect_q: + nvme_cancel_tagset(ctrl); + if (new) + nvme_remove_io_tag_set(ctrl); +out_free_io_queues: + nvmf_free_io_queues(ctrl); + return ret; +} + +static int nvmf_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) +{ + int error; + + error = ctrl->fabrics_ops->alloc_admin_queue(ctrl); + if (error) + return error; + + if (new) { + error = ctrl->fabrics_ops->alloc_admin_tag_set(ctrl); + if (error) + goto out_free_admin_queue; + + } + + error = ctrl->fabrics_ops->start_admin_queue(ctrl); + if (error) + goto out_remove_admin_tag_set; + + error = nvme_enable_ctrl(ctrl); + if (error) + goto out_stop_queue; + + nvme_unquiesce_admin_queue(ctrl); + + error = nvme_init_ctrl_finish(ctrl, false); + if (error) + goto out_quiesce_queue; + + return 0; + +out_quiesce_queue: + nvme_quiesce_admin_queue(ctrl); + blk_sync_queue(ctrl->admin_q); +out_stop_queue: + ctrl->fabrics_ops->stop_admin_queue(ctrl); + nvme_cancel_admin_tagset(ctrl); +out_remove_admin_tag_set: + if (new) + nvme_remove_admin_tag_set(ctrl); +out_free_admin_queue: + ctrl->fabrics_ops->free_admin_queue(ctrl); + return error; +} + +static void nvmf_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove) +{ + nvmf_stop_io_queues(ctrl); + if (remove) + nvme_remove_io_tag_set(ctrl); + nvmf_free_io_queues(ctrl); +} + +static void nvmf_destroy_admin_queue(struct nvme_ctrl *ctrl, bool remove) +{ + ctrl->fabrics_ops->stop_admin_queue(ctrl); + if (remove) + nvme_remove_admin_tag_set(ctrl); + + ctrl->fabrics_ops->free_admin_queue(ctrl); +} + +static void nvmf_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove) +{ + nvme_quiesce_admin_queue(ctrl); + blk_sync_queue(ctrl->admin_q); + ctrl->fabrics_ops->stop_admin_queue(ctrl); + nvme_cancel_admin_tagset(ctrl); + if (remove) + nvme_unquiesce_admin_queue(ctrl); + nvmf_destroy_admin_queue(ctrl, remove); +} + +static void nvmf_teardown_io_queues(struct nvme_ctrl *ctrl, bool remove) +{ + if (ctrl->queue_count <= 1) + return; + nvme_quiesce_admin_queue(ctrl); + nvme_start_freeze(ctrl); + nvme_quiesce_io_queues(ctrl); + nvme_sync_io_queues(ctrl); + nvmf_stop_io_queues(ctrl); + nvme_cancel_tagset(ctrl); + if (remove) + nvme_unquiesce_io_queues(ctrl); + nvmf_destroy_io_queues(ctrl, remove); +} + +void nvmf_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown) +{ + nvmf_teardown_io_queues(ctrl, shutdown); + nvme_quiesce_admin_queue(ctrl); + nvme_disable_ctrl(ctrl, shutdown); + nvmf_teardown_admin_queue(ctrl, shutdown); +} +EXPORT_SYMBOL_GPL(nvmf_teardown_ctrl); + +void nvmf_stop_ctrl(struct nvme_ctrl *ctrl) +{ + flush_work(&ctrl->err_work); + cancel_delayed_work_sync(&ctrl->connect_work); +} +EXPORT_SYMBOL_GPL(nvmf_stop_ctrl); + +int nvmf_setup_ctrl(struct nvme_ctrl *ctrl, bool new) +{ + struct nvmf_ctrl_options *opts = ctrl->opts; + int ret; + + ret = nvmf_configure_admin_queue(ctrl, new); + if (ret) + return ret; + + if (ctrl->icdoff) { + ret = -EOPNOTSUPP; + dev_err(ctrl->device, "icdoff is not supported!\n"); + goto destroy_admin; + } + + if (!nvme_ctrl_sgl_supported(ctrl)) { + ret = -EOPNOTSUPP; + dev_err(ctrl->device, "Mandatory sgls are not supported!\n"); + goto destroy_admin; + } + + if (opts->queue_size > ctrl->sqsize + 1) + dev_warn(ctrl->device, + "queue_size %zu > ctrl sqsize %u, clamping down\n", + opts->queue_size, ctrl->sqsize + 1); + + if (ctrl->sqsize + 1 > ctrl->maxcmd) { + dev_warn(ctrl->device, + "sqsize %u > ctrl maxcmd %u, clamping down\n", + ctrl->sqsize + 1, ctrl->maxcmd); + ctrl->sqsize = ctrl->maxcmd - 1; + } + + if (ctrl->queue_count > 1) { + ret = nvmf_configure_io_queues(ctrl, new); + if (ret) + goto destroy_admin; + } + + if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_LIVE)) { + /* + * state change failure is ok if we started ctrl delete, + * unless we're during creation of a new controller to + * avoid races with teardown flow. + */ + WARN_ON_ONCE(ctrl->state != NVME_CTRL_DELETING && + ctrl->state != NVME_CTRL_DELETING_NOIO); + WARN_ON_ONCE(new); + ret = -EINVAL; + goto destroy_io; + } + + nvme_start_ctrl(ctrl); + return 0; + +destroy_io: + if (ctrl->queue_count > 1) { + nvme_quiesce_io_queues(ctrl); + nvme_sync_io_queues(ctrl); + nvmf_stop_io_queues(ctrl); + nvme_cancel_tagset(ctrl); + nvmf_destroy_io_queues(ctrl, new); + } +destroy_admin: + nvme_quiesce_admin_queue(ctrl); + blk_sync_queue(ctrl->admin_q); + ctrl->fabrics_ops->stop_admin_queue(ctrl); + nvme_cancel_admin_tagset(ctrl); + nvmf_destroy_admin_queue(ctrl, new); + return ret; +} +EXPORT_SYMBOL_GPL(nvmf_setup_ctrl); + +void nvmf_reconnect_or_remove(struct nvme_ctrl *ctrl) +{ + /* If we are resetting/deleting then do nothing */ + if (ctrl->state != NVME_CTRL_CONNECTING) { + WARN_ON_ONCE(ctrl->state == NVME_CTRL_NEW || + ctrl->state == NVME_CTRL_LIVE); + return; + } + + if (nvmf_should_reconnect(ctrl)) { + dev_info(ctrl->device, "Reconnecting in %d seconds...\n", + ctrl->opts->reconnect_delay); + queue_delayed_work(nvme_wq, &ctrl->connect_work, + ctrl->opts->reconnect_delay * HZ); + } else { + dev_info(ctrl->device, "Removing controller...\n"); + nvme_delete_ctrl(ctrl); + } +} +EXPORT_SYMBOL_GPL(nvmf_reconnect_or_remove); + +void nvmf_error_recovery_work(struct work_struct *work) +{ + struct nvme_ctrl *ctrl = container_of(work, + struct nvme_ctrl, err_work); + + nvme_stop_keep_alive(ctrl); + flush_work(&ctrl->async_event_work); + nvmf_teardown_io_queues(ctrl, false); + /* unquiesce to fail fast pending requests */ + nvme_unquiesce_io_queues(ctrl); + nvmf_teardown_admin_queue(ctrl, false); + nvme_unquiesce_admin_queue(ctrl); + nvme_auth_stop(ctrl); + + if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) { + /* state change failure is ok if we started ctrl delete */ + WARN_ON_ONCE(ctrl->state != NVME_CTRL_DELETING && + ctrl->state != NVME_CTRL_DELETING_NOIO); + return; + } + + nvmf_reconnect_or_remove(ctrl); +} +EXPORT_SYMBOL_GPL(nvmf_error_recovery_work); + +void nvmf_reset_ctrl_work(struct work_struct *work) +{ + struct nvme_ctrl *ctrl = + container_of(work, struct nvme_ctrl, reset_work); + + nvme_stop_ctrl(ctrl); + nvmf_teardown_ctrl(ctrl, false); + + if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) { + /* state change failure is ok if we started ctrl delete */ + WARN_ON_ONCE(ctrl->state != NVME_CTRL_DELETING && + ctrl->state != NVME_CTRL_DELETING_NOIO); + return; + } + + if (nvmf_setup_ctrl(ctrl, false)) + goto out_fail; + + return; + +out_fail: + ++ctrl->nr_reconnects; + nvmf_reconnect_or_remove(ctrl); +} +EXPORT_SYMBOL_GPL(nvmf_reset_ctrl_work); + +void nvmf_reconnect_ctrl_work(struct work_struct *work) +{ + struct nvme_ctrl *ctrl = container_of(to_delayed_work(work), + struct nvme_ctrl, connect_work); + + ++ctrl->nr_reconnects; + + if (nvmf_setup_ctrl(ctrl, false)) + goto requeue; + + dev_info(ctrl->device, "Successfully reconnected (%d attempt)\n", + ctrl->nr_reconnects); + + ctrl->nr_reconnects = 0; + + return; + +requeue: + dev_info(ctrl->device, "Failed reconnect attempt %d\n", + ctrl->nr_reconnects); + nvmf_reconnect_or_remove(ctrl); +} +EXPORT_SYMBOL_GPL(nvmf_reconnect_ctrl_work); + +void nvmf_error_recovery(struct nvme_ctrl *ctrl) +{ + if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) + return; + + dev_warn(ctrl->device, "starting error recovery\n"); + queue_work(nvme_reset_wq, &ctrl->err_work); +} +EXPORT_SYMBOL_GPL(nvmf_error_recovery); + static struct class *nvmf_class; static struct device *nvmf_device; static DEFINE_MUTEX(nvmf_dev_mutex); diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index dcac3df8a5f7..345d6de6bc86 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -172,6 +172,23 @@ struct nvmf_transport_ops { struct nvmf_ctrl_options *opts); }; +struct nvme_fabrics_ops { + int (*alloc_admin_queue)(struct nvme_ctrl *ctrl); + int (*start_admin_queue)(struct nvme_ctrl *ctrl); + void (*stop_admin_queue)(struct nvme_ctrl *ctrl); + void (*free_admin_queue)(struct nvme_ctrl *ctrl); + int (*alloc_io_queue)(struct nvme_ctrl *ctrl, int qid); + int (*start_io_queue)(struct nvme_ctrl *ctrl, int qid); + void (*stop_io_queue)(struct nvme_ctrl *ctrl, int qid); + void (*free_io_queue)(struct nvme_ctrl *ctrl, int qid); + + /* these should be replaced with a single one setup_transport() */ + int (*alloc_admin_tag_set)(struct nvme_ctrl *ctrl); + int (*alloc_tag_set)(struct nvme_ctrl *ctrl); + unsigned int (*nr_io_queues)(struct nvme_ctrl *ctrl); + void (*set_io_queues)(struct nvme_ctrl *ctrl, unsigned int nr_io_queues); +}; + static inline bool nvmf_ctlr_matches_baseopts(struct nvme_ctrl *ctrl, struct nvmf_ctrl_options *opts) @@ -215,5 +232,13 @@ int nvmf_get_address(struct nvme_ctrl *ctrl, char *buf, int size); bool nvmf_should_reconnect(struct nvme_ctrl *ctrl); bool nvmf_ip_options_match(struct nvme_ctrl *ctrl, struct nvmf_ctrl_options *opts); +int nvmf_setup_ctrl(struct nvme_ctrl *ctrl, bool new); +void nvmf_stop_ctrl(struct nvme_ctrl *ctrl); +void nvmf_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown); +void nvmf_reset_ctrl_work(struct work_struct *work); +void nvmf_reconnect_or_remove(struct nvme_ctrl *ctrl); +void nvmf_error_recovery_work(struct work_struct *work); +void nvmf_reconnect_ctrl_work(struct work_struct *work); +void nvmf_error_recovery(struct nvme_ctrl *ctrl); #endif /* _NVME_FABRICS_H */ diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 5aa30b00dd17..fcea2678094c 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -244,6 +244,8 @@ enum nvme_ctrl_flags { NVME_CTRL_STOPPED = 3, }; +struct nvme_fabrics_ops; + struct nvme_ctrl { bool comp_seen; enum nvme_ctrl_state state; @@ -251,6 +253,7 @@ struct nvme_ctrl { spinlock_t lock; struct mutex scan_lock; const struct nvme_ctrl_ops *ops; + const struct nvme_fabrics_ops *fabrics_ops; struct request_queue *admin_q; struct request_queue *connect_q; struct request_queue *fabrics_q; From patchwork Thu May 4 09:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90014 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp170192vqo; Thu, 4 May 2023 02:24:35 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6AKSYM8vrhlB07Q4r2sCzeKAQ8Wt6qGaVIFJ+VFhgVChB5Z1iNQFRULzs/Oi7k9KEtPZPz X-Received: by 2002:a17:902:ea07:b0:1a6:4a64:4d27 with SMTP id s7-20020a170902ea0700b001a64a644d27mr3893014plg.40.1683192275139; Thu, 04 May 2023 02:24:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192275; cv=none; d=google.com; s=arc-20160816; b=ayLh0l4uzyWuFS0pLCQk5wyESuwGPuP87HAIYiHIhRu3PPDDn7hvQM+QcEiJMGXAUs ZeP593dPHgtlxUK6vNN1Tc8fak5RPc4K9I4Jjv7rct2804ej1r92F8HSeH9gnF1Nhs2A bSqwDSQu8L3f36NBCy5Em2LcllA6ukZ0D7BPA/T7GJsCNqQjzOuDMlS2JBrR7/TvI7fW 1Zjs+NGAH1Zw8YTkaYbgskx/OGbudPBkOC5Hg27nS6OEjsEo6CoE2hyfBPPctn25UNkn GW+Eq0DYTNIZRz0CuiOJ3+6/Kya7hJYHRsEClrDBCEYn+u2oWgc3HOuc47yIunk5IELK 3oMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=Ne5S41rksW/94j/aPAVYR86xIk3eBjYReoVu8x8zS6Y=; b=kOY/0HZRmzoefPRTrCpr5KLWtzT5dxVFn0tQXu3mYR8NwZPgkQRnH3MCXiEFJBw/c8 HeYrkCIK5X/3NH0nv3a0OG5GyAi9TgqtGNVaceHnqJ9NuS7xIcf70jZs84lNYZeHo2me b+Ods65KbLugbKx7OFvoXUqpe8oq7ixq7R3Wq/mxBP7cGDHTNvXiexUS+GESjDTLoMVk hT4Yz37A6c1ojMBcJ9bimGWlYs4gdgsTbeAKBkiQz63MYB/i62xwXJtWZlyj63L2IAre P30nLo5R3BJb0OaAz7Vuc0G4FLEYG/daVefkdYdsejDrokjcmxzU6xujdDNrd7J54tvg qn0A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=QB07Bz7K; dkim=neutral (no key) header.i=@suse.de header.b=BxUFEGgH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z11-20020a170902cccb00b001a506617ab4si34395227ple.27.2023.05.04.02.24.19; Thu, 04 May 2023 02:24:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=QB07Bz7K; dkim=neutral (no key) header.i=@suse.de header.b=BxUFEGgH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230335AbjEDJN1 (ORCPT + 99 others); Thu, 4 May 2023 05:13:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230309AbjEDJNO (ORCPT ); Thu, 4 May 2023 05:13:14 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BC9944A1 for ; Thu, 4 May 2023 02:13:08 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D39D6209AE; Thu, 4 May 2023 09:13:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191586; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ne5S41rksW/94j/aPAVYR86xIk3eBjYReoVu8x8zS6Y=; b=QB07Bz7K4iEVfOY+6W98tVaKgnXTKUXPf0+OVvvGWRcKlv98G/thxH8oJ1OP5Dw/s/hV+m dZxxnEK6YP2o+1qTTLRmh4MN29oFBfV4S0t1TdYmW5FR7lYxR3gcGX7NSk/+lBm37PmuQO L/7xnPkt3/+vKKv2sCAUq5uYPQBmuVE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191586; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ne5S41rksW/94j/aPAVYR86xIk3eBjYReoVu8x8zS6Y=; b=BxUFEGgHb6iGYwXkg7SiwFxOiEkvbImXjlJ1MjD0g4KpuEWmAbGpDH3ceXy/LNZSnrGcsP zHKS1Hyz+9UI4xBQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id C2EDE13444; Thu, 4 May 2023 09:13:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id hm2WLyJ3U2TmTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:06 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 6/9] nvme-tcp: replace state machine with generic one Date: Thu, 4 May 2023 11:12:56 +0200 Message-Id: <20230504091259.29100-7-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955023118048520?= X-GMAIL-MSGID: =?utf-8?q?1764955023118048520?= Signed-off-by: Daniel Wagner --- drivers/nvme/host/tcp.c | 698 ++++++++++------------------------------ 1 file changed, 175 insertions(+), 523 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 74ccc84d244a..32c4346b7322 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -519,15 +519,6 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) queue->ddgst_remaining = 0; } -static void nvme_tcp_error_recovery(struct nvme_ctrl *ctrl) -{ - if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_RESETTING)) - return; - - dev_warn(ctrl->device, "starting error recovery\n"); - queue_work(nvme_reset_wq, &ctrl->err_work); -} - static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, struct nvme_completion *cqe) { @@ -539,7 +530,7 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue, dev_err(queue->ctrl->ctrl.device, "got bad cqe.command_id %#x on queue %d\n", cqe->command_id, nvme_tcp_queue_id(queue)); - nvme_tcp_error_recovery(&queue->ctrl->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); return -EINVAL; } @@ -581,7 +572,7 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue, dev_err(queue->ctrl->ctrl.device, "queue %d tag %#x SUCCESS set but not last PDU\n", nvme_tcp_queue_id(queue), rq->tag); - nvme_tcp_error_recovery(&queue->ctrl->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); return -EPROTO; } @@ -895,7 +886,7 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb, dev_err(queue->ctrl->ctrl.device, "receive failed: %d\n", result); queue->rd_enabled = false; - nvme_tcp_error_recovery(&queue->ctrl->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); return result; } } @@ -945,7 +936,7 @@ static void nvme_tcp_state_change(struct sock *sk) case TCP_LAST_ACK: case TCP_FIN_WAIT1: case TCP_FIN_WAIT2: - nvme_tcp_error_recovery(&queue->ctrl->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); break; default: dev_info(queue->ctrl->ctrl.device, @@ -1299,34 +1290,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) return 0; } -static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) -{ - struct page *page; - struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); - struct nvme_tcp_queue *queue = &ctrl->queues[qid]; - unsigned int noreclaim_flag; - - if (!test_and_clear_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - return; - - if (queue->hdr_digest || queue->data_digest) - nvme_tcp_free_crypto(queue); - - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } - - noreclaim_flag = memalloc_noreclaim_save(); - sock_release(queue->sock); - memalloc_noreclaim_restore(noreclaim_flag); - - kfree(queue->pdu); - mutex_destroy(&queue->send_mutex); - mutex_destroy(&queue->queue_lock); -} - static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue) { struct nvme_tcp_icreq_pdu *icreq; @@ -1488,10 +1451,9 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) queue->io_cpu = cpumask_next_wrap(n - 1, cpu_online_mask, -1, false); } -static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) +static int __nvme_tcp_alloc_queue(struct nvme_tcp_ctrl *ctrl, + struct nvme_tcp_queue *queue) { - struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); - struct nvme_tcp_queue *queue = &ctrl->queues[qid]; int ret, rcv_pdu_size; mutex_init(&queue->queue_lock); @@ -1501,16 +1463,10 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) mutex_init(&queue->send_mutex); INIT_WORK(&queue->io_work, nvme_tcp_io_work); - if (qid > 0) - queue->cmnd_capsule_len = nctrl->ioccsz * 16; - else - queue->cmnd_capsule_len = sizeof(struct nvme_command) + - NVME_TCP_ADMIN_CCSZ; - ret = sock_create(ctrl->addr.ss_family, SOCK_STREAM, IPPROTO_TCP, &queue->sock); if (ret) { - dev_err(nctrl->device, + dev_err(ctrl->ctrl.device, "failed to create socket: %d\n", ret); goto err_destroy_mutex; } @@ -1534,8 +1490,8 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) sock_set_priority(queue->sock->sk, so_priority); /* Set socket type of service */ - if (nctrl->opts->tos >= 0) - ip_sock_set_tos(queue->sock->sk, nctrl->opts->tos); + if (ctrl->ctrl.opts->tos >= 0) + ip_sock_set_tos(queue->sock->sk, ctrl->ctrl.opts->tos); /* Set 10 seconds timeout for icresp recvmsg */ queue->sock->sk->sk_rcvtimeo = 10 * HZ; @@ -1550,38 +1506,39 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) queue->pdu_offset = 0; sk_set_memalloc(queue->sock->sk); - if (nctrl->opts->mask & NVMF_OPT_HOST_TRADDR) { + if (ctrl->ctrl.opts->mask & NVMF_OPT_HOST_TRADDR) { ret = kernel_bind(queue->sock, (struct sockaddr *)&ctrl->src_addr, sizeof(ctrl->src_addr)); if (ret) { - dev_err(nctrl->device, + dev_err(ctrl->ctrl.device, "failed to bind queue %d socket %d\n", - qid, ret); + nvme_tcp_queue_id(queue), ret); goto err_sock; } } - if (nctrl->opts->mask & NVMF_OPT_HOST_IFACE) { - char *iface = nctrl->opts->host_iface; + if (ctrl->ctrl.opts->mask & NVMF_OPT_HOST_IFACE) { + char *iface = ctrl->ctrl.opts->host_iface; sockptr_t optval = KERNEL_SOCKPTR(iface); ret = sock_setsockopt(queue->sock, SOL_SOCKET, SO_BINDTODEVICE, optval, strlen(iface)); if (ret) { - dev_err(nctrl->device, + dev_err(ctrl->ctrl.device, "failed to bind to interface %s queue %d err %d\n", - iface, qid, ret); + iface, nvme_tcp_queue_id(queue), ret); goto err_sock; } } - queue->hdr_digest = nctrl->opts->hdr_digest; - queue->data_digest = nctrl->opts->data_digest; + queue->hdr_digest = ctrl->ctrl.opts->hdr_digest; + queue->data_digest = ctrl->ctrl.opts->data_digest; if (queue->hdr_digest || queue->data_digest) { ret = nvme_tcp_alloc_crypto(queue); if (ret) { - dev_err(nctrl->device, - "failed to allocate queue %d crypto\n", qid); + dev_err(ctrl->ctrl.device, + "failed to allocate queue %d crypto\n", + nvme_tcp_queue_id(queue)); goto err_sock; } } @@ -1594,13 +1551,13 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) goto err_crypto; } - dev_dbg(nctrl->device, "connecting queue %d\n", - nvme_tcp_queue_id(queue)); + dev_dbg(ctrl->ctrl.device, "connecting queue %d\n", + nvme_tcp_queue_id(queue)); ret = kernel_connect(queue->sock, (struct sockaddr *)&ctrl->addr, - sizeof(ctrl->addr), 0); + sizeof(ctrl->addr), 0); if (ret) { - dev_err(nctrl->device, + dev_err(ctrl->ctrl.device, "failed to connect socket: %d\n", ret); goto err_rcv_pdu; } @@ -1644,142 +1601,182 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid) return ret; } -static void nvme_tcp_restore_sock_calls(struct nvme_tcp_queue *queue) +static void __nvme_tcp_free_queue(struct nvme_tcp_ctrl *ctrl, + struct nvme_tcp_queue *queue) { - struct socket *sock = queue->sock; + struct page *page; + unsigned int noreclaim_flag; - write_lock_bh(&sock->sk->sk_callback_lock); - sock->sk->sk_user_data = NULL; - sock->sk->sk_data_ready = queue->data_ready; - sock->sk->sk_state_change = queue->state_change; - sock->sk->sk_write_space = queue->write_space; - write_unlock_bh(&sock->sk->sk_callback_lock); + if (!test_and_clear_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) + return; + + if (queue->hdr_digest || queue->data_digest) + nvme_tcp_free_crypto(queue); + + if (queue->pf_cache.va) { + page = virt_to_head_page(queue->pf_cache.va); + __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); + queue->pf_cache.va = NULL; + } + + noreclaim_flag = memalloc_noreclaim_save(); + sock_release(queue->sock); + memalloc_noreclaim_restore(noreclaim_flag); + + kfree(queue->pdu); + mutex_destroy(&queue->send_mutex); + mutex_destroy(&queue->queue_lock); } -static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) + +static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *nctrl) { - kernel_sock_shutdown(queue->sock, SHUT_RDWR); - nvme_tcp_restore_sock_calls(queue); - cancel_work_sync(&queue->io_work); + struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[0]; + int err; + + queue->cmnd_capsule_len = sizeof(struct nvme_command) + + NVME_TCP_ADMIN_CCSZ; + + err = __nvme_tcp_alloc_queue(ctrl, queue); + if (err) + return err; + + err = nvme_tcp_alloc_async_req(ctrl); + if (err) + __nvme_tcp_free_queue(ctrl, queue); + + return err; } -static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid) +static int nvme_tcp_alloc_io_queue(struct nvme_ctrl *nctrl, int qid) { struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; - if (!test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - return; + queue->cmnd_capsule_len = nctrl->ioccsz * 16; - mutex_lock(&queue->queue_lock); - if (test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) - __nvme_tcp_stop_queue(queue); - mutex_unlock(&queue->queue_lock); + return __nvme_tcp_alloc_queue(ctrl, queue); } -static int nvme_tcp_start_queue(struct nvme_ctrl *nctrl, int idx) +static void nvme_tcp_free_admin_queue(struct nvme_ctrl *nctrl) { struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); - int ret; - - if (idx) - ret = nvmf_connect_io_queue(nctrl, idx); - else - ret = nvmf_connect_admin_queue(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[0]; - if (!ret) { - set_bit(NVME_TCP_Q_LIVE, &ctrl->queues[idx].flags); - } else { - if (test_bit(NVME_TCP_Q_ALLOCATED, &ctrl->queues[idx].flags)) - __nvme_tcp_stop_queue(&ctrl->queues[idx]); - dev_err(nctrl->device, - "failed to connect queue: %d ret=%d\n", idx, ret); + if (ctrl->async_req.pdu) { + cancel_work_sync(&nctrl->async_event_work); + nvme_tcp_free_async_req(ctrl); + ctrl->async_req.pdu = NULL; } - return ret; + __nvme_tcp_free_queue(ctrl, queue); } -static void nvme_tcp_free_admin_queue(struct nvme_ctrl *ctrl) +static void nvme_tcp_free_io_queue(struct nvme_ctrl *nctrl, int qid) { - if (to_tcp_ctrl(ctrl)->async_req.pdu) { - cancel_work_sync(&ctrl->async_event_work); - nvme_tcp_free_async_req(to_tcp_ctrl(ctrl)); - to_tcp_ctrl(ctrl)->async_req.pdu = NULL; - } + struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[qid]; - nvme_tcp_free_queue(ctrl, 0); + __nvme_tcp_free_queue(ctrl, queue); } -static void nvme_tcp_free_io_queues(struct nvme_ctrl *ctrl) +static void nvme_tcp_restore_sock_calls(struct nvme_tcp_queue *queue) { - int i; + struct socket *sock = queue->sock; - for (i = 1; i < ctrl->queue_count; i++) - nvme_tcp_free_queue(ctrl, i); + write_lock_bh(&sock->sk->sk_callback_lock); + sock->sk->sk_user_data = NULL; + sock->sk->sk_data_ready = queue->data_ready; + sock->sk->sk_state_change = queue->state_change; + sock->sk->sk_write_space = queue->write_space; + write_unlock_bh(&sock->sk->sk_callback_lock); } -static void nvme_tcp_stop_io_queues(struct nvme_ctrl *ctrl) +static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) { - int i; - - for (i = 1; i < ctrl->queue_count; i++) - nvme_tcp_stop_queue(ctrl, i); + kernel_sock_shutdown(queue->sock, SHUT_RDWR); + nvme_tcp_restore_sock_calls(queue); + cancel_work_sync(&queue->io_work); } -static int nvme_tcp_start_io_queues(struct nvme_ctrl *ctrl, - int first, int last) +static int nvme_tcp_start_admin_queue(struct nvme_ctrl *nctrl) { - int i, ret; + struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[0]; + int ret; - for (i = first; i < last; i++) { - ret = nvme_tcp_start_queue(ctrl, i); - if (ret) - goto out_stop_queues; + ret = nvmf_connect_admin_queue(nctrl); + if (!ret) { + set_bit(NVME_TCP_Q_LIVE, &queue->flags); + } else { + if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) + __nvme_tcp_stop_queue(queue); + dev_err(nctrl->device, + "failed to connect queue: %d ret=%d\n", 0, ret); } - - return 0; - -out_stop_queues: - for (i--; i >= first; i--) - nvme_tcp_stop_queue(ctrl, i); return ret; } -static int nvme_tcp_alloc_admin_queue(struct nvme_ctrl *ctrl) +static int nvme_tcp_start_io_queue(struct nvme_ctrl *nctrl, int qid) { + struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[qid]; int ret; - ret = nvme_tcp_alloc_queue(ctrl, 0); - if (ret) - return ret; + ret = nvmf_connect_io_queue(nctrl, qid); + if (!ret) { + set_bit(NVME_TCP_Q_LIVE, &queue->flags); + } else { + if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) + __nvme_tcp_stop_queue(queue); + dev_err(nctrl->device, + "failed to connect queue: %d ret=%d\n", qid, ret); + } + return ret; +} - ret = nvme_tcp_alloc_async_req(to_tcp_ctrl(ctrl)); - if (ret) - goto out_free_queue; +static void nvme_tcp_stop_admin_queue(struct nvme_ctrl *nctrl) +{ + struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[0]; - return 0; + if (!test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) + return; -out_free_queue: - nvme_tcp_free_queue(ctrl, 0); - return ret; + mutex_lock(&queue->queue_lock); + if (test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) + __nvme_tcp_stop_queue(queue); + mutex_unlock(&queue->queue_lock); } -static int __nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) +static void nvme_tcp_stop_io_queue(struct nvme_ctrl *nctrl, int qid) { - int i, ret; + struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); + struct nvme_tcp_queue *queue = &ctrl->queues[qid]; - for (i = 1; i < ctrl->queue_count; i++) { - ret = nvme_tcp_alloc_queue(ctrl, i); - if (ret) - goto out_free_queues; - } + if (!test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) + return; - return 0; + mutex_lock(&queue->queue_lock); + if (test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) + __nvme_tcp_stop_queue(queue); + mutex_unlock(&queue->queue_lock); +} -out_free_queues: - for (i--; i >= 1; i--) - nvme_tcp_free_queue(ctrl, i); +static int nvme_tcp_alloc_admin_tag_set(struct nvme_ctrl *ctrl) +{ + return nvme_alloc_admin_tag_set(ctrl, &to_tcp_ctrl(ctrl)->admin_tag_set, + &nvme_tcp_admin_mq_ops, + sizeof(struct nvme_tcp_request)); +} + +static int nvme_tcp_alloc_tag_set(struct nvme_ctrl *ctrl) +{ + return nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, + &nvme_tcp_mq_ops, + ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, + sizeof(struct nvme_tcp_request)); - return ret; } static unsigned int nvme_tcp_nr_io_queues(struct nvme_ctrl *ctrl) @@ -1828,370 +1825,9 @@ static void nvme_tcp_set_io_queues(struct nvme_ctrl *nctrl, } } -static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl) -{ - unsigned int nr_io_queues; - int ret; - - nr_io_queues = nvme_tcp_nr_io_queues(ctrl); - ret = nvme_set_queue_count(ctrl, &nr_io_queues); - if (ret) - return ret; - - if (nr_io_queues == 0) { - dev_err(ctrl->device, - "unable to set any I/O queues\n"); - return -ENOMEM; - } - - ctrl->queue_count = nr_io_queues + 1; - dev_info(ctrl->device, - "creating %d I/O queues.\n", nr_io_queues); - - nvme_tcp_set_io_queues(ctrl, nr_io_queues); - - return __nvme_tcp_alloc_io_queues(ctrl); -} - -static void nvme_tcp_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove) -{ - nvme_tcp_stop_io_queues(ctrl); - if (remove) - nvme_remove_io_tag_set(ctrl); - nvme_tcp_free_io_queues(ctrl); -} - -static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) -{ - int ret, nr_queues; - - ret = nvme_tcp_alloc_io_queues(ctrl); - if (ret) - return ret; - - if (new) { - ret = nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, - &nvme_tcp_mq_ops, - ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, - sizeof(struct nvme_tcp_request)); - if (ret) - goto out_free_io_queues; - } - - /* - * Only start IO queues for which we have allocated the tagset - * and limitted it to the available queues. On reconnects, the - * queue number might have changed. - */ - nr_queues = min(ctrl->tagset->nr_hw_queues + 1, ctrl->queue_count); - ret = nvme_tcp_start_io_queues(ctrl, 1, nr_queues); - if (ret) - goto out_cleanup_connect_q; - - if (!new) { - nvme_unquiesce_io_queues(ctrl); - if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) { - /* - * If we timed out waiting for freeze we are likely to - * be stuck. Fail the controller initialization just - * to be safe. - */ - ret = -ENODEV; - goto out_wait_freeze_timed_out; - } - blk_mq_update_nr_hw_queues(ctrl->tagset, - ctrl->queue_count - 1); - nvme_unfreeze(ctrl); - } - - /* - * If the number of queues has increased (reconnect case) - * start all new queues now. - */ - ret = nvme_tcp_start_io_queues(ctrl, nr_queues, - ctrl->tagset->nr_hw_queues + 1); - if (ret) - goto out_wait_freeze_timed_out; - - return 0; - -out_wait_freeze_timed_out: - nvme_quiesce_io_queues(ctrl); - nvme_sync_io_queues(ctrl); - nvme_tcp_stop_io_queues(ctrl); -out_cleanup_connect_q: - nvme_cancel_tagset(ctrl); - if (new) - nvme_remove_io_tag_set(ctrl); -out_free_io_queues: - nvme_tcp_free_io_queues(ctrl); - return ret; -} - -static void nvme_tcp_destroy_admin_queue(struct nvme_ctrl *ctrl, bool remove) -{ - nvme_tcp_stop_queue(ctrl, 0); - if (remove) - nvme_remove_admin_tag_set(ctrl); - nvme_tcp_free_admin_queue(ctrl); -} - -static int nvme_tcp_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) -{ - int error; - - error = nvme_tcp_alloc_admin_queue(ctrl); - if (error) - return error; - - if (new) { - error = nvme_alloc_admin_tag_set(ctrl, - &to_tcp_ctrl(ctrl)->admin_tag_set, - &nvme_tcp_admin_mq_ops, - sizeof(struct nvme_tcp_request)); - if (error) - goto out_free_queue; - } - - error = nvme_tcp_start_queue(ctrl, 0); - if (error) - goto out_cleanup_tagset; - - error = nvme_enable_ctrl(ctrl); - if (error) - goto out_stop_queue; - - nvme_unquiesce_admin_queue(ctrl); - - error = nvme_init_ctrl_finish(ctrl, false); - if (error) - goto out_quiesce_queue; - - return 0; - -out_quiesce_queue: - nvme_quiesce_admin_queue(ctrl); - blk_sync_queue(ctrl->admin_q); -out_stop_queue: - nvme_tcp_stop_queue(ctrl, 0); - nvme_cancel_admin_tagset(ctrl); -out_cleanup_tagset: - if (new) - nvme_remove_admin_tag_set(ctrl); -out_free_queue: - nvme_tcp_free_admin_queue(ctrl); - return error; -} - -static void nvme_tcp_teardown_admin_queue(struct nvme_ctrl *ctrl, - bool remove) -{ - nvme_quiesce_admin_queue(ctrl); - blk_sync_queue(ctrl->admin_q); - nvme_tcp_stop_queue(ctrl, 0); - nvme_cancel_admin_tagset(ctrl); - if (remove) - nvme_unquiesce_admin_queue(ctrl); - nvme_tcp_destroy_admin_queue(ctrl, remove); -} - -static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl, - bool remove) -{ - if (ctrl->queue_count <= 1) - return; - nvme_quiesce_admin_queue(ctrl); - nvme_start_freeze(ctrl); - nvme_quiesce_io_queues(ctrl); - nvme_sync_io_queues(ctrl); - nvme_tcp_stop_io_queues(ctrl); - nvme_cancel_tagset(ctrl); - if (remove) - nvme_unquiesce_io_queues(ctrl); - nvme_tcp_destroy_io_queues(ctrl, remove); -} - -static void nvme_tcp_reconnect_or_remove(struct nvme_ctrl *ctrl) -{ - /* If we are resetting/deleting then do nothing */ - if (ctrl->state != NVME_CTRL_CONNECTING) { - WARN_ON_ONCE(ctrl->state == NVME_CTRL_NEW || - ctrl->state == NVME_CTRL_LIVE); - return; - } - - if (nvmf_should_reconnect(ctrl)) { - dev_info(ctrl->device, "Reconnecting in %d seconds...\n", - ctrl->opts->reconnect_delay); - queue_delayed_work(nvme_wq, &ctrl->connect_work, - ctrl->opts->reconnect_delay * HZ); - } else { - dev_info(ctrl->device, "Removing controller...\n"); - nvme_delete_ctrl(ctrl); - } -} - -static int nvme_tcp_setup_ctrl(struct nvme_ctrl *ctrl, bool new) -{ - struct nvmf_ctrl_options *opts = ctrl->opts; - int ret; - - ret = nvme_tcp_configure_admin_queue(ctrl, new); - if (ret) - return ret; - - if (ctrl->icdoff) { - ret = -EOPNOTSUPP; - dev_err(ctrl->device, "icdoff is not supported!\n"); - goto destroy_admin; - } - - if (!nvme_ctrl_sgl_supported(ctrl)) { - ret = -EOPNOTSUPP; - dev_err(ctrl->device, "Mandatory sgls are not supported!\n"); - goto destroy_admin; - } - - if (opts->queue_size > ctrl->sqsize + 1) - dev_warn(ctrl->device, - "queue_size %zu > ctrl sqsize %u, clamping down\n", - opts->queue_size, ctrl->sqsize + 1); - - if (ctrl->sqsize + 1 > ctrl->maxcmd) { - dev_warn(ctrl->device, - "sqsize %u > ctrl maxcmd %u, clamping down\n", - ctrl->sqsize + 1, ctrl->maxcmd); - ctrl->sqsize = ctrl->maxcmd - 1; - } - - if (ctrl->queue_count > 1) { - ret = nvme_tcp_configure_io_queues(ctrl, new); - if (ret) - goto destroy_admin; - } - - if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_LIVE)) { - /* - * state change failure is ok if we started ctrl delete, - * unless we're during creation of a new controller to - * avoid races with teardown flow. - */ - WARN_ON_ONCE(ctrl->state != NVME_CTRL_DELETING && - ctrl->state != NVME_CTRL_DELETING_NOIO); - WARN_ON_ONCE(new); - ret = -EINVAL; - goto destroy_io; - } - - nvme_start_ctrl(ctrl); - return 0; - -destroy_io: - if (ctrl->queue_count > 1) { - nvme_quiesce_io_queues(ctrl); - nvme_sync_io_queues(ctrl); - nvme_tcp_stop_io_queues(ctrl); - nvme_cancel_tagset(ctrl); - nvme_tcp_destroy_io_queues(ctrl, new); - } -destroy_admin: - nvme_quiesce_admin_queue(ctrl); - blk_sync_queue(ctrl->admin_q); - nvme_tcp_stop_queue(ctrl, 0); - nvme_cancel_admin_tagset(ctrl); - nvme_tcp_destroy_admin_queue(ctrl, new); - return ret; -} - -static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) -{ - struct nvme_ctrl *ctrl = container_of(to_delayed_work(work), - struct nvme_ctrl, connect_work); - - ++ctrl->nr_reconnects; - - if (nvme_tcp_setup_ctrl(ctrl, false)) - goto requeue; - - dev_info(ctrl->device, "Successfully reconnected (%d attempt)\n", - ctrl->nr_reconnects); - - ctrl->nr_reconnects = 0; - - return; - -requeue: - dev_info(ctrl->device, "Failed reconnect attempt %d\n", - ctrl->nr_reconnects); - nvme_tcp_reconnect_or_remove(ctrl); -} - -static void nvme_tcp_error_recovery_work(struct work_struct *work) -{ - struct nvme_ctrl *ctrl = container_of(work, - struct nvme_ctrl, err_work); - - nvme_stop_keep_alive(ctrl); - flush_work(&ctrl->async_event_work); - nvme_tcp_teardown_io_queues(ctrl, false); - /* unquiesce to fail fast pending requests */ - nvme_unquiesce_io_queues(ctrl); - nvme_tcp_teardown_admin_queue(ctrl, false); - nvme_unquiesce_admin_queue(ctrl); - nvme_auth_stop(ctrl); - - if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) { - /* state change failure is ok if we started ctrl delete */ - WARN_ON_ONCE(ctrl->state != NVME_CTRL_DELETING && - ctrl->state != NVME_CTRL_DELETING_NOIO); - return; - } - - nvme_tcp_reconnect_or_remove(ctrl); -} - -static void nvme_tcp_teardown_ctrl(struct nvme_ctrl *ctrl, bool shutdown) -{ - nvme_tcp_teardown_io_queues(ctrl, shutdown); - nvme_quiesce_admin_queue(ctrl); - nvme_disable_ctrl(ctrl, shutdown); - nvme_tcp_teardown_admin_queue(ctrl, shutdown); -} - static void nvme_tcp_delete_ctrl(struct nvme_ctrl *ctrl) { - nvme_tcp_teardown_ctrl(ctrl, true); -} - -static void nvme_reset_ctrl_work(struct work_struct *work) -{ - struct nvme_ctrl *ctrl = - container_of(work, struct nvme_ctrl, reset_work); - - nvme_stop_ctrl(ctrl); - nvme_tcp_teardown_ctrl(ctrl, false); - - if (!nvme_change_ctrl_state(ctrl, NVME_CTRL_CONNECTING)) { - /* state change failure is ok if we started ctrl delete */ - WARN_ON_ONCE(ctrl->state != NVME_CTRL_DELETING && - ctrl->state != NVME_CTRL_DELETING_NOIO); - return; - } - - if (nvme_tcp_setup_ctrl(ctrl, false)) - goto out_fail; - - return; - -out_fail: - ++ctrl->nr_reconnects; - nvme_tcp_reconnect_or_remove(ctrl); -} - -static void nvme_tcp_stop_ctrl(struct nvme_ctrl *ctrl) -{ - flush_work(&ctrl->err_work); - cancel_delayed_work_sync(&ctrl->connect_work); + nvmf_teardown_ctrl(ctrl, true); } static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl) @@ -2275,7 +1911,7 @@ static void nvme_tcp_complete_timed_out(struct request *rq) struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); struct nvme_ctrl *ctrl = &req->queue->ctrl->ctrl; - nvme_tcp_stop_queue(ctrl, nvme_tcp_queue_id(req->queue)); + nvme_tcp_stop_io_queue(ctrl, nvme_tcp_queue_id(req->queue)); nvmf_complete_timed_out_request(rq); } @@ -2314,7 +1950,7 @@ static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) * LIVE state should trigger the normal error recovery which will * handle completing this request. */ - nvme_tcp_error_recovery(ctrl); + nvmf_error_recovery(ctrl); return BLK_EH_RESET_TIMER; } @@ -2540,7 +2176,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ctrl_ops = { .submit_async_event = nvme_tcp_submit_async_event, .delete_ctrl = nvme_tcp_delete_ctrl, .get_address = nvme_tcp_get_address, - .stop_ctrl = nvme_tcp_stop_ctrl, + .stop_ctrl = nvmf_stop_ctrl, }; static bool @@ -2560,6 +2196,21 @@ nvme_tcp_existing_controller(struct nvmf_ctrl_options *opts) return found; } +static struct nvme_fabrics_ops nvme_tcp_fabrics_ops = { + .alloc_admin_queue = nvme_tcp_alloc_admin_queue, + .free_admin_queue = nvme_tcp_free_admin_queue, + .start_admin_queue = nvme_tcp_start_admin_queue, + .stop_admin_queue = nvme_tcp_stop_admin_queue, + .alloc_io_queue = nvme_tcp_alloc_io_queue, + .free_io_queue = nvme_tcp_free_io_queue, + .start_io_queue = nvme_tcp_start_io_queue, + .stop_io_queue = nvme_tcp_stop_io_queue, + .alloc_admin_tag_set = nvme_tcp_alloc_admin_tag_set, + .alloc_tag_set = nvme_tcp_alloc_tag_set, + .nr_io_queues = nvme_tcp_nr_io_queues, + .set_io_queues = nvme_tcp_set_io_queues, +}; + static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, struct nvmf_ctrl_options *opts) { @@ -2572,15 +2223,16 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, INIT_LIST_HEAD(&ctrl->list); ctrl->ctrl.opts = opts; + ctrl->ctrl.fabrics_ops = &nvme_tcp_fabrics_ops; ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues + opts->nr_poll_queues + 1; ctrl->ctrl.sqsize = opts->queue_size - 1; ctrl->ctrl.kato = opts->kato; INIT_DELAYED_WORK(&ctrl->ctrl.connect_work, - nvme_tcp_reconnect_ctrl_work); - INIT_WORK(&ctrl->ctrl.err_work, nvme_tcp_error_recovery_work); - INIT_WORK(&ctrl->ctrl.reset_work, nvme_reset_ctrl_work); + nvmf_reconnect_ctrl_work); + INIT_WORK(&ctrl->ctrl.err_work, nvmf_error_recovery_work); + INIT_WORK(&ctrl->ctrl.reset_work, nvmf_reset_ctrl_work); if (!(opts->mask & NVMF_OPT_TRSVCID)) { opts->trsvcid = @@ -2641,7 +2293,7 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, goto out_uninit_ctrl; } - ret = nvme_tcp_setup_ctrl(&ctrl->ctrl, true); + ret = nvmf_setup_ctrl(&ctrl->ctrl, true); if (ret) goto out_uninit_ctrl; From patchwork Thu May 4 09:12:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90020 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp174377vqo; Thu, 4 May 2023 02:34:03 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ47X00BrbgJlN7UJi+Ap4xZ3rv8BK4S5oFZIHn1AIBqoCwGUbF5J9/IU2rRUwL6mf2ra/EW X-Received: by 2002:a05:6a20:6a1d:b0:ef:cd5b:a5c7 with SMTP id p29-20020a056a206a1d00b000efcd5ba5c7mr1758516pzk.56.1683192843492; Thu, 04 May 2023 02:34:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192843; cv=none; d=google.com; s=arc-20160816; b=J+SLsi5wqUcPNPmZ9l/WGHui/tNN4rRTN5JsM0UWquKzUftsN4NRXGuXEMh2KpFcij 72M/kNfwcE8wEwhhXOKohn6fux31ldbbrEL8B45QIKZdUanKjKpJeFvQwqV+kySySnd0 OITos72wd7bXh9C6aD5W1w/dJ8P6Pmvipk/FcTjhRXK+XUy5xSSbdlRZJ0GuwEbe5lwr 9Loa05qOlao/oCs41NbC8r4ckHYWQk+SapHUPZefjw+fePAxVNdO3ZZr2XzWMuJUsRkX XaJI3uz4fl8iIv8Mb4y90IgiLm3NYtUk8Z6n3JrRcD3uPIEggCSKrGs/6BVz7GSAAeAN s67A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=wRpPk2OAt1XjanwaNas9Trw8tVdU+qjHxgEEKaT3M8k=; b=z9Br+n/5QmoctHER7Uao3BGZEpeEbVjTRJFT47O40SeMtulXQnhpnuIB/SzaBqMCMW QHMX94VEOechOF8rGSs77tAJrrLTvK4qfWJUKSKV/3J4jx6MhmxN8xG0cvz6XYe0Jv/D M7LOIlZ2RgRGJFK03JyaF7OXy/TdULz0VmqY+uFWuQyTMSAiUohks/k8u+iTk0KTdg83 RB8NgSOXcqHd/kfnMJv4JyMtaPcVQsr1Gr6++qELc8cOf70y7J1yQSumimJblzwDwl80 isuu43mzDP07x4gZEbRFjNIXNJUC1BzIXhbsArU9hHwbPGjkmfDPt/iJKiHDbwB96ylF C/AA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=tsRL6eny; dkim=neutral (no key) header.i=@suse.de header.b=bVao6M3q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y10-20020a63fa0a000000b005001391ef6bsi34659217pgh.108.2023.05.04.02.33.49; Thu, 04 May 2023 02:34:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=tsRL6eny; dkim=neutral (no key) header.i=@suse.de header.b=bVao6M3q; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230376AbjEDJNb (ORCPT + 99 others); Thu, 4 May 2023 05:13:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230324AbjEDJNQ (ORCPT ); Thu, 4 May 2023 05:13:16 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8B693ABF for ; Thu, 4 May 2023 02:13:08 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6D22333933; Thu, 4 May 2023 09:13:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wRpPk2OAt1XjanwaNas9Trw8tVdU+qjHxgEEKaT3M8k=; b=tsRL6enyVg3JADqIgAi/8v8/VS66p/iqZy4ZQoGgRd5ZATVqbnKQDLSixPSIg9ftTAKC5i ZdpxDq02zsM4EkayMddRZVlw5gYLgM935jN6pLKuoYOKjgCKYf2YxYOLvXOuam5XsW7Q84 cA5cGHsNPH387j/YsIHA576D2v1IFdE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191587; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wRpPk2OAt1XjanwaNas9Trw8tVdU+qjHxgEEKaT3M8k=; b=bVao6M3qpKLZuXT1/NibtApS5MzcgAlI8zo3j0d4QE9xW/1bV+hkCc/gXuyyRQbrfxeysd I+9YBiqdwYlCxEDw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 5C37713444; Thu, 4 May 2023 09:13:07 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id doEtFiN3U2ToTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:07 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 7/9] nvme-rdma: replace state machine with generic one Date: Thu, 4 May 2023 11:12:57 +0200 Message-Id: <20230504091259.29100-8-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955619184796053?= X-GMAIL-MSGID: =?utf-8?q?1764955619184796053?= Signed-off-by: Daniel Wagner --- drivers/nvme/host/rdma.c | 703 ++++++++++----------------------------- 1 file changed, 173 insertions(+), 530 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index b0ab5a9d5fe0..1fde65e8c2b5 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -568,35 +568,16 @@ static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue) return ret; } -static int nvme_rdma_alloc_queue(struct nvme_ctrl *nctrl, int idx) +static int __nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, + struct nvme_rdma_queue *queue) { - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - struct nvme_rdma_queue *queue; struct sockaddr *src_addr = NULL; - size_t queue_size; int ret; - if (idx == 0) - queue_size = NVME_AQ_DEPTH; - else - queue_size = ctrl->ctrl.sqsize + 1; - - queue = &ctrl->queues[idx]; mutex_init(&queue->queue_lock); queue->ctrl = ctrl; - if (idx && ctrl->ctrl.max_integrity_segments) - queue->pi_support = true; - else - queue->pi_support = false; init_completion(&queue->cm_done); - if (idx > 0) - queue->cmnd_capsule_len = ctrl->ctrl.ioccsz * 16; - else - queue->cmnd_capsule_len = sizeof(struct nvme_command); - - queue->queue_size = queue_size; - queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue, RDMA_PS_TCP, IB_QPT_RC); if (IS_ERR(queue->cm_id)) { @@ -638,62 +619,120 @@ static int nvme_rdma_alloc_queue(struct nvme_ctrl *nctrl, int idx) return ret; } -static void __nvme_rdma_stop_queue(struct nvme_rdma_queue *queue) +static void __nvme_rdma_free_queue(struct nvme_rdma_ctrl *ctrl, + struct nvme_rdma_queue *queue) { - rdma_disconnect(queue->cm_id); - ib_drain_qp(queue->qp); + if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) + return; + + rdma_destroy_id(queue->cm_id); + nvme_rdma_destroy_queue_ib(queue); + mutex_destroy(&queue->queue_lock); } -static void nvme_rdma_stop_queue(struct nvme_ctrl *nctrl, int qid) +static int nvme_rdma_alloc_admin_queue(struct nvme_ctrl *nctrl) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - struct nvme_rdma_queue *queue = &ctrl->queues[qid]; + struct nvme_rdma_queue *queue = &ctrl->queues[0]; + bool pi_capable = false; + int ret; - mutex_lock(&queue->queue_lock); - if (test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags)) - __nvme_rdma_stop_queue(queue); - mutex_unlock(&queue->queue_lock); + queue->cmnd_capsule_len = sizeof(struct nvme_command); + queue->queue_size = NVME_AQ_DEPTH; + queue->pi_support = false; + + ret = __nvme_rdma_alloc_queue(ctrl, queue); + if (ret) + return ret; + + ctrl->device = queue->device; + nctrl->numa_node = ibdev_to_node(ctrl->device->dev); + + /* T10-PI support */ + if (ctrl->device->dev->attrs.kernel_cap_flags & + IBK_INTEGRITY_HANDOVER) + pi_capable = true; + + ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev, + pi_capable); + + /* + * Bind the async event SQE DMA mapping to the admin queue lifetime. + * It's safe, since any chage in the underlying RDMA device will issue + * error recovery and queue re-creation. + */ + ret = nvme_rdma_alloc_qe(ctrl->device->dev, &ctrl->async_event_sqe, + sizeof(struct nvme_command), DMA_TO_DEVICE); + if (ret) { + __nvme_rdma_free_queue(ctrl, queue); + return ret; + } + + ctrl->ctrl.max_segments = ctrl->max_fr_pages; + ctrl->ctrl.max_hw_sectors = ctrl->max_fr_pages << (ilog2(SZ_4K) - 9); + if (pi_capable) + ctrl->ctrl.max_integrity_segments = ctrl->max_fr_pages; + else + ctrl->ctrl.max_integrity_segments = 0; + + return 0; } -static void nvme_rdma_free_queue(struct nvme_ctrl *nctrl, int qid) +static int nvme_rdma_alloc_io_queue(struct nvme_ctrl *nctrl, int qid) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); struct nvme_rdma_queue *queue = &ctrl->queues[qid]; - if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) - return; + queue->cmnd_capsule_len = ctrl->ctrl.ioccsz * 16; + queue->queue_size = ctrl->ctrl.sqsize + 1; + if (ctrl->ctrl.max_integrity_segments) + queue->pi_support = true; + else + queue->pi_support = false; - rdma_destroy_id(queue->cm_id); - nvme_rdma_destroy_queue_ib(queue); - mutex_destroy(&queue->queue_lock); + return __nvme_rdma_alloc_queue(ctrl, queue); } -static void nvme_rdma_free_io_queues(struct nvme_rdma_ctrl *ctrl) +static void nvme_rdma_free_admin_queue(struct nvme_ctrl *nctrl) { - int i; + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[0]; + + if (ctrl->async_event_sqe.data) { + cancel_work_sync(&ctrl->ctrl.async_event_work); + nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe, + sizeof(struct nvme_command), DMA_TO_DEVICE); + ctrl->async_event_sqe.data = NULL; + } - for (i = 1; i < ctrl->ctrl.queue_count; i++) - nvme_rdma_free_queue(&ctrl->ctrl, i); + __nvme_rdma_free_queue(ctrl, queue); } -static void nvme_rdma_stop_io_queues(struct nvme_rdma_ctrl *ctrl) +static void nvme_rdma_free_io_queue(struct nvme_ctrl *nctrl, int qid) { - int i; + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[qid]; - for (i = 1; i < ctrl->ctrl.queue_count; i++) - nvme_rdma_stop_queue(&ctrl->ctrl, i); + __nvme_rdma_free_queue(ctrl, queue); +} + +static void __nvme_rdma_stop_queue(struct nvme_rdma_queue *queue) +{ + mutex_lock(&queue->queue_lock); + if (test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags)) { + rdma_disconnect(queue->cm_id); + ib_drain_qp(queue->qp); + } + mutex_unlock(&queue->queue_lock); } -static int nvme_rdma_start_queue(struct nvme_ctrl *nctrl, int idx) +static int nvme_rdma_start_admin_queue(struct nvme_ctrl *nctrl) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - struct nvme_rdma_queue *queue = &ctrl->queues[idx]; + struct nvme_rdma_queue *queue = &ctrl->queues[0]; int ret; - if (idx) - ret = nvmf_connect_io_queue(nctrl, idx); - else - ret = nvmf_connect_admin_queue(nctrl); + ret = nvmf_connect_admin_queue(nctrl); if (!ret) { set_bit(NVME_RDMA_Q_LIVE, &queue->flags); @@ -701,58 +740,74 @@ static int nvme_rdma_start_queue(struct nvme_ctrl *nctrl, int idx) if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) __nvme_rdma_stop_queue(queue); dev_info(ctrl->ctrl.device, - "failed to connect queue: %d ret=%d\n", idx, ret); + "failed to connect queue: %d ret=%d\n", 0, ret); } return ret; } -static int nvme_rdma_start_io_queues(struct nvme_rdma_ctrl *ctrl, - int first, int last) +static int nvme_rdma_start_io_queue(struct nvme_ctrl *nctrl, int idx) { - int i, ret = 0; + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[idx]; + int ret; - for (i = first; i < last; i++) { - ret = nvme_rdma_start_queue(&ctrl->ctrl, i); - if (ret) - goto out_stop_queues; + ret = nvmf_connect_io_queue(nctrl, idx); + if (!ret) { + set_bit(NVME_RDMA_Q_LIVE, &queue->flags); + } else { + if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) + __nvme_rdma_stop_queue(queue); + dev_info(ctrl->ctrl.device, + "failed to connect queue: %d ret=%d\n", idx, ret); } + return ret; +} - return 0; +static void nvme_rdma_stop_admin_queue(struct nvme_ctrl *nctrl) +{ + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[0]; -out_stop_queues: - for (i--; i >= first; i--) - nvme_rdma_stop_queue(&ctrl->ctrl, i); - return ret; + __nvme_rdma_stop_queue(queue); } -static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) +static void nvme_rdma_stop_io_queue(struct nvme_ctrl *nctrl, int qid) { - struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); + struct nvme_rdma_queue *queue = &ctrl->queues[qid]; + + __nvme_rdma_stop_queue(queue); +} + +static unsigned int nvme_rdma_nr_io_queues(struct nvme_ctrl *ctrl) +{ + struct ib_device *ibdev = to_rdma_ctrl(ctrl)->device->dev; + struct nvmf_ctrl_options *opts = ctrl->opts; + unsigned int nr_io_queues; + + nr_io_queues = min_t(unsigned int, ibdev->num_comp_vectors, + min(opts->nr_io_queues, num_online_cpus())); + nr_io_queues += min_t(unsigned int, ibdev->num_comp_vectors, + min(opts->nr_write_queues, num_online_cpus())); + nr_io_queues += min(opts->nr_poll_queues, num_online_cpus()); + + return nr_io_queues; +} + +static void nvme_rdma_set_io_queues(struct nvme_ctrl *nctrl, + unsigned int nr_io_queues) +{ + struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); struct ib_device *ibdev = ctrl->device->dev; - unsigned int nr_io_queues, nr_default_queues; + struct nvmf_ctrl_options *opts = ctrl->ctrl.opts; + unsigned int nr_default_queues; unsigned int nr_read_queues, nr_poll_queues; - int i, ret; nr_read_queues = min_t(unsigned int, ibdev->num_comp_vectors, min(opts->nr_io_queues, num_online_cpus())); nr_default_queues = min_t(unsigned int, ibdev->num_comp_vectors, min(opts->nr_write_queues, num_online_cpus())); nr_poll_queues = min(opts->nr_poll_queues, num_online_cpus()); - nr_io_queues = nr_read_queues + nr_default_queues + nr_poll_queues; - - ret = nvme_set_queue_count(&ctrl->ctrl, &nr_io_queues); - if (ret) - return ret; - - if (nr_io_queues == 0) { - dev_err(ctrl->ctrl.device, - "unable to set any I/O queues\n"); - return -ENOMEM; - } - - ctrl->ctrl.queue_count = nr_io_queues + 1; - dev_info(ctrl->ctrl.device, - "creating %d I/O queues.\n", nr_io_queues); if (opts->nr_write_queues && nr_read_queues < nr_io_queues) { /* @@ -781,20 +836,6 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl) ctrl->io_queues[HCTX_TYPE_POLL] = min(nr_poll_queues, nr_io_queues); } - - for (i = 1; i < ctrl->ctrl.queue_count; i++) { - ret = nvme_rdma_alloc_queue(&ctrl->ctrl, i); - if (ret) - goto out_free_queues; - } - - return 0; - -out_free_queues: - for (i--; i >= 1; i--) - nvme_rdma_free_queue(&ctrl->ctrl, i); - - return ret; } static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl) @@ -812,231 +853,6 @@ static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl) cmd_size); } -static void nvme_rdma_destroy_admin_queue(struct nvme_rdma_ctrl *ctrl) -{ - if (ctrl->async_event_sqe.data) { - cancel_work_sync(&ctrl->ctrl.async_event_work); - nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe, - sizeof(struct nvme_command), DMA_TO_DEVICE); - ctrl->async_event_sqe.data = NULL; - } - nvme_rdma_free_queue(&ctrl->ctrl, 0); -} - -static int nvme_rdma_init_queue(struct nvme_ctrl *nctrl, int qid) -{ - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - bool pi_capable = false; - int error; - - if (qid != 0) - /* only admin queue needs additional work. */ - return 0; - - - ctrl->device = ctrl->queues[0].device; - ctrl->ctrl.numa_node = ibdev_to_node(ctrl->device->dev); - - /* T10-PI support */ - if (ctrl->device->dev->attrs.kernel_cap_flags & - IBK_INTEGRITY_HANDOVER) - pi_capable = true; - - ctrl->max_fr_pages = nvme_rdma_get_max_fr_pages(ctrl->device->dev, - pi_capable); - - /* - * Bind the async event SQE DMA mapping to the admin queue lifetime. - * It's safe, since any chage in the underlying RDMA device will issue - * error recovery and queue re-creation. - */ - error = nvme_rdma_alloc_qe(ctrl->device->dev, &ctrl->async_event_sqe, - sizeof(struct nvme_command), DMA_TO_DEVICE); - if (error) - return error; - - ctrl->ctrl.max_segments = ctrl->max_fr_pages; - ctrl->ctrl.max_hw_sectors = ctrl->max_fr_pages << (ilog2(SZ_4K) - 9); - if (pi_capable) - ctrl->ctrl.max_integrity_segments = ctrl->max_fr_pages; - else - ctrl->ctrl.max_integrity_segments = 0; - - return 0; -} - -static void nvme_rdma_deinit_queue(struct nvme_ctrl *nctrl, int qid) -{ - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - - if (qid != 0) - return; - - if (ctrl->async_event_sqe.data) { - nvme_rdma_free_qe(ctrl->device->dev, &ctrl->async_event_sqe, - sizeof(struct nvme_command), DMA_TO_DEVICE); - ctrl->async_event_sqe.data = NULL; - } -} - -static int nvme_rdma_configure_admin_queue(struct nvme_rdma_ctrl *ctrl, - bool new) -{ - int error; - - error = nvme_rdma_alloc_queue(&ctrl->ctrl, 0); - if (error) - return error; - - error = nvme_rdma_init_queue(&ctrl->ctrl, 0); - if (error) - goto out_free_queue; - - if (new) { - error = nvme_alloc_admin_tag_set(&ctrl->ctrl, - &ctrl->admin_tag_set, &nvme_rdma_admin_mq_ops, - sizeof(struct nvme_rdma_request) + - NVME_RDMA_DATA_SGL_SIZE); - if (error) - goto out_deinit_admin_queue; - - } - - error = nvme_rdma_start_queue(&ctrl->ctrl, 0); - if (error) - goto out_remove_admin_tag_set; - - error = nvme_enable_ctrl(&ctrl->ctrl); - if (error) - goto out_stop_queue; - - nvme_unquiesce_admin_queue(&ctrl->ctrl); - - error = nvme_init_ctrl_finish(&ctrl->ctrl, false); - if (error) - goto out_quiesce_queue; - - return 0; - -out_quiesce_queue: - nvme_quiesce_admin_queue(&ctrl->ctrl); - blk_sync_queue(ctrl->ctrl.admin_q); -out_stop_queue: - nvme_rdma_stop_queue(&ctrl->ctrl, 0); - nvme_cancel_admin_tagset(&ctrl->ctrl); -out_remove_admin_tag_set: - if (new) - nvme_remove_admin_tag_set(&ctrl->ctrl); -out_deinit_admin_queue: - nvme_rdma_deinit_queue(&ctrl->ctrl, 0); -out_free_queue: - nvme_rdma_free_queue(&ctrl->ctrl, 0); - return error; -} - -static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) -{ - int ret, nr_queues; - - ret = nvme_rdma_alloc_io_queues(ctrl); - if (ret) - return ret; - - if (new) { - ret = nvme_rdma_alloc_tag_set(&ctrl->ctrl); - if (ret) - goto out_free_io_queues; - } - - /* - * Only start IO queues for which we have allocated the tagset - * and limitted it to the available queues. On reconnects, the - * queue number might have changed. - */ - nr_queues = min(ctrl->tag_set.nr_hw_queues + 1, ctrl->ctrl.queue_count); - ret = nvme_rdma_start_io_queues(ctrl, 1, nr_queues); - if (ret) - goto out_cleanup_tagset; - - if (!new) { - nvme_unquiesce_io_queues(&ctrl->ctrl); - if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) { - /* - * If we timed out waiting for freeze we are likely to - * be stuck. Fail the controller initialization just - * to be safe. - */ - ret = -ENODEV; - goto out_wait_freeze_timed_out; - } - blk_mq_update_nr_hw_queues(ctrl->ctrl.tagset, - ctrl->ctrl.queue_count - 1); - nvme_unfreeze(&ctrl->ctrl); - } - - /* - * If the number of queues has increased (reconnect case) - * start all new queues now. - */ - ret = nvme_rdma_start_io_queues(ctrl, nr_queues, - ctrl->tag_set.nr_hw_queues + 1); - if (ret) - goto out_wait_freeze_timed_out; - - return 0; - -out_wait_freeze_timed_out: - nvme_quiesce_io_queues(&ctrl->ctrl); - nvme_sync_io_queues(&ctrl->ctrl); - nvme_rdma_stop_io_queues(ctrl); -out_cleanup_tagset: - nvme_cancel_tagset(&ctrl->ctrl); - if (new) - nvme_remove_io_tag_set(&ctrl->ctrl); -out_free_io_queues: - nvme_rdma_free_io_queues(ctrl); - return ret; -} - -static void nvme_rdma_teardown_admin_queue(struct nvme_rdma_ctrl *ctrl, - bool remove) -{ - nvme_quiesce_admin_queue(&ctrl->ctrl); - blk_sync_queue(ctrl->ctrl.admin_q); - nvme_rdma_stop_queue(&ctrl->ctrl, 0); - nvme_cancel_admin_tagset(&ctrl->ctrl); - if (remove) { - nvme_unquiesce_admin_queue(&ctrl->ctrl); - nvme_remove_admin_tag_set(&ctrl->ctrl); - } - nvme_rdma_destroy_admin_queue(ctrl); -} - -static void nvme_rdma_teardown_io_queues(struct nvme_rdma_ctrl *ctrl, - bool remove) -{ - if (ctrl->ctrl.queue_count > 1) { - nvme_start_freeze(&ctrl->ctrl); - nvme_quiesce_io_queues(&ctrl->ctrl); - nvme_sync_io_queues(&ctrl->ctrl); - nvme_rdma_stop_io_queues(ctrl); - nvme_cancel_tagset(&ctrl->ctrl); - if (remove) { - nvme_unquiesce_io_queues(&ctrl->ctrl); - nvme_remove_io_tag_set(&ctrl->ctrl); - } - nvme_rdma_free_io_queues(ctrl); - } -} - -static void nvme_rdma_stop_ctrl(struct nvme_ctrl *nctrl) -{ - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - - flush_work(&ctrl->ctrl.err_work); - cancel_delayed_work_sync(&ctrl->ctrl.connect_work); -} - static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); @@ -1054,169 +870,13 @@ static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) kfree(ctrl); } -static void nvme_rdma_reconnect_or_remove(struct nvme_rdma_ctrl *ctrl) -{ - /* If we are resetting/deleting then do nothing */ - if (ctrl->ctrl.state != NVME_CTRL_CONNECTING) { - WARN_ON_ONCE(ctrl->ctrl.state == NVME_CTRL_NEW || - ctrl->ctrl.state == NVME_CTRL_LIVE); - return; - } - - if (nvmf_should_reconnect(&ctrl->ctrl)) { - dev_info(ctrl->ctrl.device, "Reconnecting in %d seconds...\n", - ctrl->ctrl.opts->reconnect_delay); - queue_delayed_work(nvme_wq, &ctrl->ctrl.connect_work, - ctrl->ctrl.opts->reconnect_delay * HZ); - } else { - nvme_delete_ctrl(&ctrl->ctrl); - } -} - -static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) -{ - int ret; - bool changed; - - ret = nvme_rdma_configure_admin_queue(ctrl, new); - if (ret) - return ret; - - if (ctrl->ctrl.icdoff) { - ret = -EOPNOTSUPP; - dev_err(ctrl->ctrl.device, "icdoff is not supported!\n"); - goto destroy_admin; - } - - if (!(ctrl->ctrl.sgls & (1 << 2))) { - ret = -EOPNOTSUPP; - dev_err(ctrl->ctrl.device, - "Mandatory keyed sgls are not supported!\n"); - goto destroy_admin; - } - - if (ctrl->ctrl.opts->queue_size > ctrl->ctrl.sqsize + 1) { - dev_warn(ctrl->ctrl.device, - "queue_size %zu > ctrl sqsize %u, clamping down\n", - ctrl->ctrl.opts->queue_size, ctrl->ctrl.sqsize + 1); - } - - if (ctrl->ctrl.sqsize + 1 > NVME_RDMA_MAX_QUEUE_SIZE) { - dev_warn(ctrl->ctrl.device, - "ctrl sqsize %u > max queue size %u, clamping down\n", - ctrl->ctrl.sqsize + 1, NVME_RDMA_MAX_QUEUE_SIZE); - ctrl->ctrl.sqsize = NVME_RDMA_MAX_QUEUE_SIZE - 1; - } - - if (ctrl->ctrl.sqsize + 1 > ctrl->ctrl.maxcmd) { - dev_warn(ctrl->ctrl.device, - "sqsize %u > ctrl maxcmd %u, clamping down\n", - ctrl->ctrl.sqsize + 1, ctrl->ctrl.maxcmd); - ctrl->ctrl.sqsize = ctrl->ctrl.maxcmd - 1; - } - - if (ctrl->ctrl.sgls & (1 << 20)) - ctrl->use_inline_data = true; - - if (ctrl->ctrl.queue_count > 1) { - ret = nvme_rdma_configure_io_queues(ctrl, new); - if (ret) - goto destroy_admin; - } - - changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_LIVE); - if (!changed) { - /* - * state change failure is ok if we started ctrl delete, - * unless we're during creation of a new controller to - * avoid races with teardown flow. - */ - WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING && - ctrl->ctrl.state != NVME_CTRL_DELETING_NOIO); - WARN_ON_ONCE(new); - ret = -EINVAL; - goto destroy_io; - } - - nvme_start_ctrl(&ctrl->ctrl); - return 0; - -destroy_io: - if (ctrl->ctrl.queue_count > 1) { - nvme_quiesce_io_queues(&ctrl->ctrl); - nvme_sync_io_queues(&ctrl->ctrl); - nvme_rdma_stop_io_queues(ctrl); - nvme_cancel_tagset(&ctrl->ctrl); - if (new) - nvme_remove_io_tag_set(&ctrl->ctrl); - nvme_rdma_free_io_queues(ctrl); - } -destroy_admin: - nvme_quiesce_admin_queue(&ctrl->ctrl); - blk_sync_queue(ctrl->ctrl.admin_q); - nvme_rdma_stop_queue(&ctrl->ctrl, 0); - nvme_cancel_admin_tagset(&ctrl->ctrl); - if (new) - nvme_remove_admin_tag_set(&ctrl->ctrl); - nvme_rdma_destroy_admin_queue(ctrl); - return ret; -} - -static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) -{ - struct nvme_ctrl *nctrl = container_of(to_delayed_work(work), - struct nvme_ctrl, connect_work); - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - - ++ctrl->ctrl.nr_reconnects; - - if (nvme_rdma_setup_ctrl(ctrl, false)) - goto requeue; - - dev_info(ctrl->ctrl.device, "Successfully reconnected (%d attempts)\n", - ctrl->ctrl.nr_reconnects); - - ctrl->ctrl.nr_reconnects = 0; - - return; - -requeue: - dev_info(ctrl->ctrl.device, "Failed reconnect attempt %d\n", - ctrl->ctrl.nr_reconnects); - nvme_rdma_reconnect_or_remove(ctrl); -} - -static void nvme_rdma_error_recovery_work(struct work_struct *work) +static int nvme_rdma_alloc_admin_tag_set(struct nvme_ctrl *ctrl) { - struct nvme_ctrl *nctrl = container_of(work, - struct nvme_ctrl, err_work); - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - nvme_stop_keep_alive(&ctrl->ctrl); - flush_work(&ctrl->ctrl.async_event_work); - nvme_rdma_teardown_io_queues(ctrl, false); - nvme_unquiesce_io_queues(&ctrl->ctrl); - nvme_rdma_teardown_admin_queue(ctrl, false); - nvme_unquiesce_admin_queue(&ctrl->ctrl); - nvme_auth_stop(&ctrl->ctrl); - - if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) { - /* state change failure is ok if we started ctrl delete */ - WARN_ON_ONCE(ctrl->ctrl.state != NVME_CTRL_DELETING && - ctrl->ctrl.state != NVME_CTRL_DELETING_NOIO); - return; - } - - nvme_rdma_reconnect_or_remove(ctrl); -} - -static void nvme_rdma_error_recovery(struct nvme_rdma_ctrl *ctrl) -{ - if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_RESETTING)) - return; - - dev_warn(ctrl->ctrl.device, "starting error recovery\n"); - queue_work(nvme_reset_wq, &ctrl->ctrl.err_work); + return nvme_alloc_admin_tag_set(ctrl, &to_rdma_ctrl(ctrl)->admin_tag_set, + &nvme_rdma_admin_mq_ops, + sizeof(struct nvme_rdma_request) + + NVME_RDMA_DATA_SGL_SIZE); } static void nvme_rdma_end_request(struct nvme_rdma_request *req) @@ -1240,7 +900,7 @@ static void nvme_rdma_wr_error(struct ib_cq *cq, struct ib_wc *wc, "%s for CQE 0x%p failed with status %s (%d)\n", op, wc->wr_cqe, ib_wc_status_msg(wc->status), wc->status); - nvme_rdma_error_recovery(ctrl); + nvmf_error_recovery(&ctrl->ctrl); } static void nvme_rdma_memreg_done(struct ib_cq *cq, struct ib_wc *wc) @@ -1759,7 +1419,7 @@ static void nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, dev_err(queue->ctrl->ctrl.device, "got bad command_id %#x on QP %#x\n", cqe->command_id, queue->qp->qp_num); - nvme_rdma_error_recovery(queue->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); return; } req = blk_mq_rq_to_pdu(rq); @@ -1773,7 +1433,7 @@ static void nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, dev_err(queue->ctrl->ctrl.device, "Bogus remote invalidation for rkey %#x\n", req->mr ? req->mr->rkey : 0); - nvme_rdma_error_recovery(queue->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); } } else if (req->mr) { int ret; @@ -1783,7 +1443,7 @@ static void nvme_rdma_process_nvme_rsp(struct nvme_rdma_queue *queue, dev_err(queue->ctrl->ctrl.device, "Queueing INV WR for rkey %#x failed (%d)\n", req->mr->rkey, ret); - nvme_rdma_error_recovery(queue->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); } /* the local invalidation completion will end the request */ return; @@ -1810,7 +1470,7 @@ static void nvme_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc) if (unlikely(wc->byte_len < len)) { dev_err(queue->ctrl->ctrl.device, "Unexpected nvme completion length(%d)\n", wc->byte_len); - nvme_rdma_error_recovery(queue->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); return; } @@ -1980,7 +1640,7 @@ static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id, case RDMA_CM_EVENT_TIMEWAIT_EXIT: dev_dbg(queue->ctrl->ctrl.device, "disconnect received - connection closed\n"); - nvme_rdma_error_recovery(queue->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); break; case RDMA_CM_EVENT_DEVICE_REMOVAL: /* device removal is handled via the ib_client API */ @@ -1988,7 +1648,7 @@ static int nvme_rdma_cm_handler(struct rdma_cm_id *cm_id, default: dev_err(queue->ctrl->ctrl.device, "Unexpected RDMA CM event (%d)\n", ev->event); - nvme_rdma_error_recovery(queue->ctrl); + nvmf_error_recovery(&queue->ctrl->ctrl); break; } @@ -2006,7 +1666,7 @@ static void nvme_rdma_complete_timed_out(struct request *rq) struct nvme_ctrl *ctrl = &req->queue->ctrl->ctrl; struct nvme_rdma_queue *queue = req->queue; - nvme_rdma_stop_queue(ctrl, nvme_rdma_queue_id(queue)); + nvme_rdma_stop_io_queue(ctrl, nvme_rdma_queue_id(queue)); nvmf_complete_timed_out_request(rq); } @@ -2041,7 +1701,7 @@ static enum blk_eh_timer_return nvme_rdma_timeout(struct request *rq) * LIVE state should trigger the normal error recovery which will * handle completing this request. */ - nvme_rdma_error_recovery(ctrl); + nvmf_error_recovery(&ctrl->ctrl); return BLK_EH_RESET_TIMER; } @@ -2242,41 +1902,9 @@ static const struct blk_mq_ops nvme_rdma_admin_mq_ops = { .timeout = nvme_rdma_timeout, }; -static void nvme_rdma_shutdown_ctrl(struct nvme_rdma_ctrl *ctrl, bool shutdown) -{ - nvme_rdma_teardown_io_queues(ctrl, shutdown); - nvme_quiesce_admin_queue(&ctrl->ctrl); - nvme_disable_ctrl(&ctrl->ctrl, shutdown); - nvme_rdma_teardown_admin_queue(ctrl, shutdown); -} - static void nvme_rdma_delete_ctrl(struct nvme_ctrl *ctrl) { - nvme_rdma_shutdown_ctrl(to_rdma_ctrl(ctrl), true); -} - -static void nvme_rdma_reset_ctrl_work(struct work_struct *work) -{ - struct nvme_rdma_ctrl *ctrl = - container_of(work, struct nvme_rdma_ctrl, ctrl.reset_work); - - nvme_stop_ctrl(&ctrl->ctrl); - nvme_rdma_shutdown_ctrl(ctrl, false); - - if (!nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING)) { - /* state change failure should never happen */ - WARN_ON_ONCE(1); - return; - } - - if (nvme_rdma_setup_ctrl(ctrl, false)) - goto out_fail; - - return; - -out_fail: - ++ctrl->ctrl.nr_reconnects; - nvme_rdma_reconnect_or_remove(ctrl); + nvmf_teardown_ctrl(ctrl, true); } static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = { @@ -2290,7 +1918,7 @@ static const struct nvme_ctrl_ops nvme_rdma_ctrl_ops = { .submit_async_event = nvme_rdma_submit_async_event, .delete_ctrl = nvme_rdma_delete_ctrl, .get_address = nvmf_get_address, - .stop_ctrl = nvme_rdma_stop_ctrl, + .stop_ctrl = nvmf_stop_ctrl, }; /* @@ -2322,6 +1950,21 @@ nvme_rdma_existing_controller(struct nvmf_ctrl_options *opts) return found; } +static struct nvme_fabrics_ops nvme_rdma_fabrics_ops = { + .alloc_admin_queue = nvme_rdma_alloc_admin_queue, + .free_admin_queue = nvme_rdma_free_admin_queue, + .start_admin_queue = nvme_rdma_start_admin_queue, + .stop_admin_queue = nvme_rdma_stop_admin_queue, + .alloc_io_queue = nvme_rdma_alloc_io_queue, + .free_io_queue = nvme_rdma_free_io_queue, + .start_io_queue = nvme_rdma_start_io_queue, + .stop_io_queue = nvme_rdma_stop_io_queue, + .alloc_admin_tag_set = nvme_rdma_alloc_admin_tag_set, + .alloc_tag_set = nvme_rdma_alloc_tag_set, + .nr_io_queues = nvme_rdma_nr_io_queues, + .set_io_queues = nvme_rdma_set_io_queues, +}; + static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, struct nvmf_ctrl_options *opts) { @@ -2333,6 +1976,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, if (!ctrl) return ERR_PTR(-ENOMEM); ctrl->ctrl.opts = opts; + ctrl->ctrl.fabrics_ops = &nvme_rdma_fabrics_ops; INIT_LIST_HEAD(&ctrl->list); if (!(opts->mask & NVMF_OPT_TRSVCID)) { @@ -2369,10 +2013,9 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, } INIT_DELAYED_WORK(&ctrl->ctrl.connect_work, - nvme_rdma_reconnect_ctrl_work); - INIT_WORK(&ctrl->ctrl.err_work, nvme_rdma_error_recovery_work); - INIT_WORK(&ctrl->ctrl.reset_work, nvme_rdma_reset_ctrl_work); - + nvmf_reconnect_ctrl_work); + INIT_WORK(&ctrl->ctrl.err_work, nvmf_error_recovery_work); + INIT_WORK(&ctrl->ctrl.reset_work, nvmf_reset_ctrl_work); ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues + opts->nr_poll_queues + 1; ctrl->ctrl.sqsize = opts->queue_size - 1; @@ -2392,7 +2035,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, changed = nvme_change_ctrl_state(&ctrl->ctrl, NVME_CTRL_CONNECTING); WARN_ON_ONCE(!changed); - ret = nvme_rdma_setup_ctrl(ctrl, true); + ret = nvmf_setup_ctrl(&ctrl->ctrl, true); if (ret) goto out_uninit_ctrl; From patchwork Thu May 4 09:12:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90012 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp169682vqo; Thu, 4 May 2023 02:23:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5N6KUBvxOsmfVzeXz7YkSvEooizvf5McY5dg7ssYozxjT8euwQ+Q002/WZmFS9N+MGglId X-Received: by 2002:a05:6a20:3d95:b0:f3:4da0:a25d with SMTP id s21-20020a056a203d9500b000f34da0a25dmr1970781pzi.13.1683192195996; Thu, 04 May 2023 02:23:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192195; cv=none; d=google.com; s=arc-20160816; b=ZO0yiIN14VteHTTu684Jc4/ksVkBmEn7RAq1cbZWlk582evhVnupwVUqg57Mf9MF/w ge+ZkY/2fxvQly+Fct7BqBdvHoL3PrxewBR8LL8K8P0tQoSfCGviASqPO4Jj9esXV4QN gwqpFJ6onyGmdxE62lbJzth94vINnQ+bG+/Ccjp2y02Oa0QfeJqgd76eqO+4TzmoopuN /cCwq8w9RY/dBcIU0fTv3l7bp2m+UOntfcPSuURvfA/wVyAGt7ZL5xAuBJbJn5H8VA4f 0qUcnWCk30Olgpo2G9WX5eTYFrWg4QsjbbMXaFYHSNR7a2ouURylThAo+qr+LkWGzYwz /gAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=/MT2eOEXTDtx6TbMI4OROEQ94vFPqSXJcCQGfexDgnU=; b=hIaPsd87jXR4IfkUiXhefLxQsZ1eYox6LuK/rJN9L9IdUyeH6L7DXgnd8kbD+pS00a EYxwZ7T6cukMOkegSdBaLw/h/ykZwf8r5V/Ms3MqKTZzzre8QmLqL2XBemvXiMZI7bLt KDzPCAQqw3ArwcUq8SfdrSWcgkjsUklksxeJm3b2Svnj0FWXGCT08UBsXsrgSvPW+lxc vetqgwPr1oVOg2PTOnzoxOR5JYIDIMfAa69l6CB//0Fx53EDJNRPCGOv2dY8S17XCfmR vg7coLxw1kIDYbDH+J+xIXwBrpepBiaapSvX7LSW8j0DlqsFqYPr09iyFxab1GIen1at zYsQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=dJeCpK6o; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=lka5R4+8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 126-20020a630284000000b0050bf22172d3si6630555pgc.490.2023.05.04.02.23.03; Thu, 04 May 2023 02:23:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=dJeCpK6o; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=lka5R4+8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230381AbjEDJNh (ORCPT + 99 others); Thu, 4 May 2023 05:13:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230332AbjEDJNQ (ORCPT ); Thu, 4 May 2023 05:13:16 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93BE44684 for ; Thu, 4 May 2023 02:13:09 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 172E133934; Thu, 4 May 2023 09:13:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191588; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/MT2eOEXTDtx6TbMI4OROEQ94vFPqSXJcCQGfexDgnU=; b=dJeCpK6oMVSvr8smGf+ihxMQdwqUVDqmYUlVJyIus/e84HDjTS8QRQwhOWTAwbH8DtE3MF a7OpBR3Fo+QiZCUcUa3pXSsYjlmNUvQe6f9MYuxl26kdoPhwiWt+bRp6YwM3XwuMse21OQ xq2q99L5AiFr8w6ZhWYbtHC/d40ApgI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191588; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/MT2eOEXTDtx6TbMI4OROEQ94vFPqSXJcCQGfexDgnU=; b=lka5R4+8+IgQ341avap5uL4bPGE7krFsKwHiwzAuOhwRIvtGiZdeZsF6lDomI0QJV1q8sn iZL0VWtbTFCBjbDQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 0827F13444; Thu, 4 May 2023 09:13:08 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 28f9ASR3U2TrTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:08 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 8/9] nvme: move queue flags to middle layer Date: Thu, 4 May 2023 11:12:58 +0200 Message-Id: <20230504091259.29100-9-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764954940043564906?= X-GMAIL-MSGID: =?utf-8?q?1764954940043564906?= The queue flags are used to track the state of the queue (deleted, live, ...). Move this generic feature into the fabric middle layer. Unfortunately, rdma uses an transport flag (TR_READY) which is not used in the generic part of the state machine. Signed-off-by: Daniel Wagner --- drivers/nvme/host/fabrics.c | 157 +++++++++++++++++++++++++++++------- drivers/nvme/host/nvme.h | 19 ++++- drivers/nvme/host/rdma.c | 75 ++++++----------- drivers/nvme/host/tcp.c | 87 +++++++------------- 4 files changed, 197 insertions(+), 141 deletions(-) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index 3d2cde17338d..5f212cb9421a 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -1134,13 +1134,117 @@ nvmf_create_ctrl(struct device *dev, const char *buf) return ERR_PTR(ret); } +static int __nvmf_alloc_admin_queue(struct nvme_ctrl *ctrl) +{ + int ret; + + ret = ctrl->fabrics_ops->alloc_admin_queue(ctrl); + if (ret) + return ret; + + set_bit(NVME_FABRICS_Q_ALLOCATED, ctrl->queues[0].flags); + + return 0; +} + +static void __nvmf_free_admin_queue(struct nvme_ctrl *ctrl) +{ + if (!test_and_clear_bit(NVME_FABRICS_Q_ALLOCATED, + ctrl->queues[0].flags)) + return; + + ctrl->fabrics_ops->free_admin_queue(ctrl); +} + +static int __nvmf_start_admin_queue(struct nvme_ctrl *ctrl) +{ + int ret; + + if (!test_bit(NVME_FABRICS_Q_ALLOCATED, ctrl->queues[0].flags)) + return -EINVAL; + + ret = ctrl->fabrics_ops->start_admin_queue(ctrl); + if (ret) { + dev_err(ctrl->device, + "failed to start admin queue: ret=%d\n", ret); + return ret; + } + + set_bit(NVME_FABRICS_Q_LIVE, ctrl->queues[0].flags); + + return 0; +} + +static void __nvmf_stop_admin_queue(struct nvme_ctrl *ctrl) +{ + if (!test_bit(NVME_FABRICS_Q_ALLOCATED, ctrl->queues[0].flags)) + return; + + mutex_lock(ctrl->queues[0].queue_lock); + if (test_and_clear_bit(NVME_FABRICS_Q_LIVE, ctrl->queues[0].flags)) + ctrl->fabrics_ops->stop_admin_queue(ctrl); + mutex_unlock(ctrl->queues[0].queue_lock); +} + +static int __nvmf_alloc_io_queue(struct nvme_ctrl *ctrl, int qid) +{ + int ret; + + ret = ctrl->fabrics_ops->alloc_io_queue(ctrl, qid); + if (ret) { + dev_err(ctrl->device, + "failed to start I/O queue: %d ret=%d\n", qid, ret); + return ret; + } + + set_bit(NVME_FABRICS_Q_ALLOCATED, ctrl->queues[qid].flags); + + return 0; +} + +static void __nvmf_free_io_queue(struct nvme_ctrl *ctrl, int qid) +{ + if (!test_and_clear_bit(NVME_FABRICS_Q_ALLOCATED, + ctrl->queues[qid].flags)) + return; + + ctrl->fabrics_ops->free_io_queue(ctrl, qid); +} + +static int __nvmf_start_io_queue(struct nvme_ctrl *ctrl, int qid) +{ + int ret; + + if (!test_bit(NVME_FABRICS_Q_ALLOCATED, ctrl->queues[0].flags)) + return -EINVAL; + + ret = ctrl->fabrics_ops->start_io_queue(ctrl, qid); + if (ret) + return ret; + + set_bit(NVME_FABRICS_Q_LIVE, ctrl->queues[qid].flags); + + return 0; +} + +static void __nvmf_stop_io_queue(struct nvme_ctrl *ctrl, int qid) +{ + if (!test_bit(NVME_FABRICS_Q_ALLOCATED, ctrl->queues[qid].flags)) + return; + + mutex_lock(ctrl->queues[qid].queue_lock); + if (test_and_clear_bit(NVME_FABRICS_Q_LIVE, ctrl->queues[qid].flags)) + ctrl->fabrics_ops->stop_io_queue(ctrl, qid); + mutex_unlock(ctrl->queues[qid].queue_lock); +} + static int nvmf_start_io_queues(struct nvme_ctrl *ctrl, int first, int last) { int i, ret; for (i = first; i < last; i++) { - ret = ctrl->fabrics_ops->start_io_queue(ctrl, i); + ret = __nvmf_start_io_queue(ctrl, i); if (ret) goto out_stop_queues; } @@ -1149,7 +1253,7 @@ static int nvmf_start_io_queues(struct nvme_ctrl *ctrl, out_stop_queues: for (i--; i >= first; i--) - ctrl->fabrics_ops->stop_io_queue(ctrl, i); + __nvmf_stop_io_queue(ctrl, i); return ret; } @@ -1158,7 +1262,7 @@ static void nvmf_stop_io_queues(struct nvme_ctrl *ctrl) int i; for (i = 1; i < ctrl->queue_count; i++) - ctrl->fabrics_ops->stop_io_queue(ctrl, i); + __nvmf_stop_io_queue(ctrl, i); } static int __nvmf_alloc_io_queues(struct nvme_ctrl *ctrl) @@ -1166,7 +1270,7 @@ static int __nvmf_alloc_io_queues(struct nvme_ctrl *ctrl) int i, ret; for (i = 1; i < ctrl->queue_count; i++) { - ret = ctrl->fabrics_ops->alloc_io_queue(ctrl, i); + ret = __nvmf_alloc_io_queue(ctrl, i); if (ret) goto out_free_queues; } @@ -1175,7 +1279,7 @@ static int __nvmf_alloc_io_queues(struct nvme_ctrl *ctrl) out_free_queues: for (i--; i >= 1; i--) - ctrl->fabrics_ops->free_io_queue(ctrl, i); + __nvmf_free_io_queue(ctrl, i); return ret; } @@ -1198,7 +1302,7 @@ static int nvmf_alloc_io_queues(struct nvme_ctrl *ctrl) ctrl->queue_count = nr_io_queues + 1; dev_info(ctrl->device, - "creating %d I/O queues.\n", nr_io_queues); + "creating %d I/O queues.\n", nr_io_queues); ctrl->fabrics_ops->set_io_queues(ctrl, nr_io_queues); @@ -1210,7 +1314,7 @@ static void nvmf_free_io_queues(struct nvme_ctrl *ctrl) int i; for (i = 1; i < ctrl->queue_count; i++) - ctrl->fabrics_ops->free_io_queue(ctrl, i); + __nvmf_free_io_queue(ctrl, i); } static int nvmf_configure_io_queues(struct nvme_ctrl *ctrl, bool new) @@ -1279,31 +1383,31 @@ static int nvmf_configure_io_queues(struct nvme_ctrl *ctrl, bool new) static int nvmf_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) { - int error; + int ret; - error = ctrl->fabrics_ops->alloc_admin_queue(ctrl); - if (error) - return error; + ret = __nvmf_alloc_admin_queue(ctrl); + if (ret) + return ret; if (new) { - error = ctrl->fabrics_ops->alloc_admin_tag_set(ctrl); - if (error) + ret = ctrl->fabrics_ops->alloc_admin_tag_set(ctrl); + if (ret) goto out_free_admin_queue; } - error = ctrl->fabrics_ops->start_admin_queue(ctrl); - if (error) + ret = __nvmf_start_admin_queue(ctrl); + if (ret) goto out_remove_admin_tag_set; - error = nvme_enable_ctrl(ctrl); - if (error) + ret = nvme_enable_ctrl(ctrl); + if (ret) goto out_stop_queue; nvme_unquiesce_admin_queue(ctrl); - error = nvme_init_ctrl_finish(ctrl, false); - if (error) + ret = nvme_init_ctrl_finish(ctrl, false); + if (ret) goto out_quiesce_queue; return 0; @@ -1312,14 +1416,14 @@ static int nvmf_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) nvme_quiesce_admin_queue(ctrl); blk_sync_queue(ctrl->admin_q); out_stop_queue: - ctrl->fabrics_ops->stop_admin_queue(ctrl); + __nvmf_stop_admin_queue(ctrl); nvme_cancel_admin_tagset(ctrl); out_remove_admin_tag_set: if (new) nvme_remove_admin_tag_set(ctrl); out_free_admin_queue: - ctrl->fabrics_ops->free_admin_queue(ctrl); - return error; + __nvmf_free_admin_queue(ctrl); + return ret; } static void nvmf_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove) @@ -1332,18 +1436,17 @@ static void nvmf_destroy_io_queues(struct nvme_ctrl *ctrl, bool remove) static void nvmf_destroy_admin_queue(struct nvme_ctrl *ctrl, bool remove) { - ctrl->fabrics_ops->stop_admin_queue(ctrl); + __nvmf_stop_admin_queue(ctrl); if (remove) nvme_remove_admin_tag_set(ctrl); - - ctrl->fabrics_ops->free_admin_queue(ctrl); + __nvmf_free_admin_queue(ctrl); } static void nvmf_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove) { nvme_quiesce_admin_queue(ctrl); blk_sync_queue(ctrl->admin_q); - ctrl->fabrics_ops->stop_admin_queue(ctrl); + __nvmf_stop_admin_queue(ctrl); nvme_cancel_admin_tagset(ctrl); if (remove) nvme_unquiesce_admin_queue(ctrl); @@ -1447,7 +1550,7 @@ int nvmf_setup_ctrl(struct nvme_ctrl *ctrl, bool new) destroy_admin: nvme_quiesce_admin_queue(ctrl); blk_sync_queue(ctrl->admin_q); - ctrl->fabrics_ops->stop_admin_queue(ctrl); + __nvmf_stop_admin_queue(ctrl); nvme_cancel_admin_tagset(ctrl); nvmf_destroy_admin_queue(ctrl, new); return ret; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index fcea2678094c..0810bc2a9e13 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -246,6 +246,18 @@ enum nvme_ctrl_flags { struct nvme_fabrics_ops; +enum nvme_fabrics_queue_flags { + NVME_FABRICS_Q_ALLOCATED = 0, + NVME_FABRICS_Q_TR_READY = 1, + NVME_FABRICS_Q_LIVE = 2, + NVME_FABRICS_Q_POLLING = 3, +}; + +struct nvme_fabrics_queue { + unsigned long *flags; + struct mutex *queue_lock; +}; + struct nvme_ctrl { bool comp_seen; enum nvme_ctrl_state state; @@ -253,7 +265,6 @@ struct nvme_ctrl { spinlock_t lock; struct mutex scan_lock; const struct nvme_ctrl_ops *ops; - const struct nvme_fabrics_ops *fabrics_ops; struct request_queue *admin_q; struct request_queue *connect_q; struct request_queue *fabrics_q; @@ -342,8 +353,10 @@ struct nvme_ctrl { struct work_struct ana_work; #endif - struct work_struct err_work; - struct delayed_work connect_work; + const struct nvme_fabrics_ops *fabrics_ops; + struct nvme_fabrics_queue *queues; + struct work_struct err_work; + struct delayed_work connect_work; #ifdef CONFIG_NVME_AUTH struct work_struct dhchap_auth_work; diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 1fde65e8c2b5..023316fdc2c6 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -3,6 +3,7 @@ * NVMe over Fabrics RDMA host code. * Copyright (c) 2015-2016 HGST, a Western Digital Company. */ +#include "linux/gfp_types.h" #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include #include @@ -76,12 +77,6 @@ struct nvme_rdma_request { bool use_sig_mr; }; -enum nvme_rdma_queue_flags { - NVME_RDMA_Q_ALLOCATED = 0, - NVME_RDMA_Q_LIVE = 1, - NVME_RDMA_Q_TR_READY = 2, -}; - struct nvme_rdma_queue { struct nvme_rdma_qe *rsp_ring; int queue_size; @@ -425,7 +420,7 @@ static void nvme_rdma_destroy_queue_ib(struct nvme_rdma_queue *queue) struct nvme_rdma_device *dev; struct ib_device *ibdev; - if (!test_and_clear_bit(NVME_RDMA_Q_TR_READY, &queue->flags)) + if (!test_and_clear_bit(NVME_FABRICS_Q_TR_READY, &queue->flags)) return; dev = queue->device; @@ -550,7 +545,7 @@ static int nvme_rdma_create_queue_ib(struct nvme_rdma_queue *queue) } } - set_bit(NVME_RDMA_Q_TR_READY, &queue->flags); + set_bit(NVME_FABRICS_Q_TR_READY, &queue->flags); return 0; @@ -572,12 +567,17 @@ static int __nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, struct nvme_rdma_queue *queue) { struct sockaddr *src_addr = NULL; + struct nvme_fabrics_queue *fqueue; int ret; mutex_init(&queue->queue_lock); queue->ctrl = ctrl; init_completion(&queue->cm_done); + fqueue = &ctrl->ctrl.queues[nvme_rdma_queue_id(queue)]; + fqueue->flags = &queue->flags; + fqueue->queue_lock = &queue->queue_lock; + queue->cm_id = rdma_create_id(&init_net, nvme_rdma_cm_handler, queue, RDMA_PS_TCP, IB_QPT_RC); if (IS_ERR(queue->cm_id)) { @@ -607,8 +607,6 @@ static int __nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, goto out_destroy_cm_id; } - set_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags); - return 0; out_destroy_cm_id: @@ -622,9 +620,6 @@ static int __nvme_rdma_alloc_queue(struct nvme_rdma_ctrl *ctrl, static void __nvme_rdma_free_queue(struct nvme_rdma_ctrl *ctrl, struct nvme_rdma_queue *queue) { - if (!test_and_clear_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) - return; - rdma_destroy_id(queue->cm_id); nvme_rdma_destroy_queue_ib(queue); mutex_destroy(&queue->queue_lock); @@ -718,49 +713,18 @@ static void nvme_rdma_free_io_queue(struct nvme_ctrl *nctrl, int qid) static void __nvme_rdma_stop_queue(struct nvme_rdma_queue *queue) { - mutex_lock(&queue->queue_lock); - if (test_and_clear_bit(NVME_RDMA_Q_LIVE, &queue->flags)) { - rdma_disconnect(queue->cm_id); - ib_drain_qp(queue->qp); - } - mutex_unlock(&queue->queue_lock); + rdma_disconnect(queue->cm_id); + ib_drain_qp(queue->qp); } -static int nvme_rdma_start_admin_queue(struct nvme_ctrl *nctrl) +static int nvme_rdma_start_admin_queue(struct nvme_ctrl *ctrl) { - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - struct nvme_rdma_queue *queue = &ctrl->queues[0]; - int ret; - - ret = nvmf_connect_admin_queue(nctrl); - - if (!ret) { - set_bit(NVME_RDMA_Q_LIVE, &queue->flags); - } else { - if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) - __nvme_rdma_stop_queue(queue); - dev_info(ctrl->ctrl.device, - "failed to connect queue: %d ret=%d\n", 0, ret); - } - return ret; + return nvmf_connect_admin_queue(ctrl); } -static int nvme_rdma_start_io_queue(struct nvme_ctrl *nctrl, int idx) +static int nvme_rdma_start_io_queue(struct nvme_ctrl *ctrl, int idx) { - struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); - struct nvme_rdma_queue *queue = &ctrl->queues[idx]; - int ret; - - ret = nvmf_connect_io_queue(nctrl, idx); - if (!ret) { - set_bit(NVME_RDMA_Q_LIVE, &queue->flags); - } else { - if (test_bit(NVME_RDMA_Q_ALLOCATED, &queue->flags)) - __nvme_rdma_stop_queue(queue); - dev_info(ctrl->ctrl.device, - "failed to connect queue: %d ret=%d\n", idx, ret); - } - return ret; + return nvmf_connect_io_queue(ctrl, idx); } static void nvme_rdma_stop_admin_queue(struct nvme_ctrl *nctrl) @@ -1715,7 +1679,7 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_hw_ctx *hctx, struct nvme_rdma_qe *sqe = &req->sqe; struct nvme_command *c = nvme_req(rq)->cmd; struct ib_device *dev; - bool queue_ready = test_bit(NVME_RDMA_Q_LIVE, &queue->flags); + bool queue_ready = test_bit(NVME_FABRICS_Q_LIVE, &queue->flags); blk_status_t ret; int err; @@ -2027,6 +1991,12 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, if (!ctrl->queues) goto out_free_ctrl; + ctrl->ctrl.queues = kcalloc(ctrl->ctrl.queue_count, + sizeof(*ctrl->ctrl.queues), + GFP_KERNEL); + if (!ctrl->ctrl.queues) + goto out_free_ctrl_queues; + ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_rdma_ctrl_ops, 0 /* no quirks, we're perfect! */); if (ret) @@ -2054,7 +2024,10 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, if (ret > 0) ret = -EIO; return ERR_PTR(ret); + kfree(ctrl->queues); out_kfree_queues: + kfree(ctrl->ctrl.queues); +out_free_ctrl_queues: kfree(ctrl->queues); out_free_ctrl: kfree(ctrl); diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 32c4346b7322..dfdf35b32adc 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -100,12 +100,6 @@ struct nvme_tcp_request { enum nvme_tcp_send_state state; }; -enum nvme_tcp_queue_flags { - NVME_TCP_Q_ALLOCATED = 0, - NVME_TCP_Q_LIVE = 1, - NVME_TCP_Q_POLLING = 2, -}; - enum nvme_tcp_recv_state { NVME_TCP_RECV_PDU = 0, NVME_TCP_RECV_DATA, @@ -903,7 +897,7 @@ static void nvme_tcp_data_ready(struct sock *sk) read_lock_bh(&sk->sk_callback_lock); queue = sk->sk_user_data; if (likely(queue && queue->rd_enabled) && - !test_bit(NVME_TCP_Q_POLLING, &queue->flags)) + !test_bit(NVME_FABRICS_Q_POLLING, &queue->flags)) queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work); read_unlock_bh(&sk->sk_callback_lock); } @@ -1454,6 +1448,7 @@ static void nvme_tcp_set_queue_io_cpu(struct nvme_tcp_queue *queue) static int __nvme_tcp_alloc_queue(struct nvme_tcp_ctrl *ctrl, struct nvme_tcp_queue *queue) { + struct nvme_fabrics_queue *fqueue; int ret, rcv_pdu_size; mutex_init(&queue->queue_lock); @@ -1463,6 +1458,10 @@ static int __nvme_tcp_alloc_queue(struct nvme_tcp_ctrl *ctrl, mutex_init(&queue->send_mutex); INIT_WORK(&queue->io_work, nvme_tcp_io_work); + fqueue = &ctrl->ctrl.queues[nvme_tcp_queue_id(queue)]; + fqueue->flags = &queue->flags; + fqueue->queue_lock = &queue->queue_lock; + ret = sock_create(ctrl->addr.ss_family, SOCK_STREAM, IPPROTO_TCP, &queue->sock); if (ret) { @@ -1567,7 +1566,6 @@ static int __nvme_tcp_alloc_queue(struct nvme_tcp_ctrl *ctrl, goto err_init_connect; queue->rd_enabled = true; - set_bit(NVME_TCP_Q_ALLOCATED, &queue->flags); nvme_tcp_init_recv_ctx(queue); write_lock_bh(&queue->sock->sk->sk_callback_lock); @@ -1607,9 +1605,6 @@ static void __nvme_tcp_free_queue(struct nvme_tcp_ctrl *ctrl, struct page *page; unsigned int noreclaim_flag; - if (!test_and_clear_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - return; - if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); @@ -1699,40 +1694,14 @@ static void __nvme_tcp_stop_queue(struct nvme_tcp_queue *queue) cancel_work_sync(&queue->io_work); } -static int nvme_tcp_start_admin_queue(struct nvme_ctrl *nctrl) +static int nvme_tcp_start_admin_queue(struct nvme_ctrl *ctrl) { - struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); - struct nvme_tcp_queue *queue = &ctrl->queues[0]; - int ret; - - ret = nvmf_connect_admin_queue(nctrl); - if (!ret) { - set_bit(NVME_TCP_Q_LIVE, &queue->flags); - } else { - if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - __nvme_tcp_stop_queue(queue); - dev_err(nctrl->device, - "failed to connect queue: %d ret=%d\n", 0, ret); - } - return ret; + return nvmf_connect_admin_queue(ctrl); } -static int nvme_tcp_start_io_queue(struct nvme_ctrl *nctrl, int qid) +static int nvme_tcp_start_io_queue(struct nvme_ctrl *ctrl, int qid) { - struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); - struct nvme_tcp_queue *queue = &ctrl->queues[qid]; - int ret; - - ret = nvmf_connect_io_queue(nctrl, qid); - if (!ret) { - set_bit(NVME_TCP_Q_LIVE, &queue->flags); - } else { - if (test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - __nvme_tcp_stop_queue(queue); - dev_err(nctrl->device, - "failed to connect queue: %d ret=%d\n", qid, ret); - } - return ret; + return nvmf_connect_io_queue(ctrl, qid); } static void nvme_tcp_stop_admin_queue(struct nvme_ctrl *nctrl) @@ -1740,13 +1709,7 @@ static void nvme_tcp_stop_admin_queue(struct nvme_ctrl *nctrl) struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[0]; - if (!test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - return; - - mutex_lock(&queue->queue_lock); - if (test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) - __nvme_tcp_stop_queue(queue); - mutex_unlock(&queue->queue_lock); + __nvme_tcp_stop_queue(queue); } static void nvme_tcp_stop_io_queue(struct nvme_ctrl *nctrl, int qid) @@ -1754,13 +1717,7 @@ static void nvme_tcp_stop_io_queue(struct nvme_ctrl *nctrl, int qid) struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; - if (!test_bit(NVME_TCP_Q_ALLOCATED, &queue->flags)) - return; - - mutex_lock(&queue->queue_lock); - if (test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) - __nvme_tcp_stop_queue(queue); - mutex_unlock(&queue->queue_lock); + __nvme_tcp_stop_queue(queue); } static int nvme_tcp_alloc_admin_tag_set(struct nvme_ctrl *ctrl) @@ -1843,6 +1800,7 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl) nvmf_free_options(nctrl->opts); free_ctrl: + kfree(ctrl->ctrl.queues); kfree(ctrl->queues); kfree(ctrl); } @@ -2043,7 +2001,7 @@ static blk_status_t nvme_tcp_queue_rq(struct blk_mq_hw_ctx *hctx, struct nvme_tcp_queue *queue = hctx->driver_data; struct request *rq = bd->rq; struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); - bool queue_ready = test_bit(NVME_TCP_Q_LIVE, &queue->flags); + bool queue_ready = test_bit(NVME_FABRICS_Q_LIVE, &queue->flags); blk_status_t ret; if (!nvme_check_ready(&queue->ctrl->ctrl, rq, queue_ready)) @@ -2108,14 +2066,14 @@ static int nvme_tcp_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob) struct nvme_tcp_queue *queue = hctx->driver_data; struct sock *sk = queue->sock->sk; - if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags)) + if (!test_bit(NVME_FABRICS_Q_LIVE, &queue->flags)) return 0; - set_bit(NVME_TCP_Q_POLLING, &queue->flags); + set_bit(NVME_FABRICS_Q_POLLING, &queue->flags); if (sk_can_busy_loop(sk) && skb_queue_empty_lockless(&sk->sk_receive_queue)) sk_busy_loop(sk, true); nvme_tcp_try_recv(queue); - clear_bit(NVME_TCP_Q_POLLING, &queue->flags); + clear_bit(NVME_FABRICS_Q_POLLING, &queue->flags); return queue->nr_cqe; } @@ -2129,7 +2087,7 @@ static int nvme_tcp_get_address(struct nvme_ctrl *ctrl, char *buf, int size) mutex_lock(&queue->queue_lock); - if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags)) + if (!test_bit(NVME_CTRL_LIVE, &queue->flags)) goto done; ret = kernel_getsockname(queue->sock, (struct sockaddr *)&src_addr); if (ret > 0) { @@ -2282,6 +2240,13 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, ret = -ENOMEM; goto out_free_ctrl; } + ctrl->ctrl.queues = kcalloc(ctrl->ctrl.queue_count, + sizeof(*ctrl->ctrl.queues), + GFP_KERNEL); + if (!ctrl->ctrl.queues) { + ret = -ENOMEM; + goto out_free_ctrl_queue; + } ret = nvme_init_ctrl(&ctrl->ctrl, dev, &nvme_tcp_ctrl_ops, 0); if (ret) @@ -2313,6 +2278,8 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, ret = -EIO; return ERR_PTR(ret); out_kfree_queues: + kfree(ctrl->ctrl.queues); +out_free_ctrl_queue: kfree(ctrl->queues); out_free_ctrl: kfree(ctrl); From patchwork Thu May 4 09:12:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Wagner X-Patchwork-Id: 90022 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp174644vqo; Thu, 4 May 2023 02:34:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7r72tB0kKb0HVSmGenaMv4DuxkMsBObq5wlmKhQ3QsibzTYW0MuSI2ST9anMBJxC/pEMp/ X-Received: by 2002:a05:6a20:914c:b0:f0:a283:4854 with SMTP id x12-20020a056a20914c00b000f0a2834854mr1593874pzc.13.1683192876400; Thu, 04 May 2023 02:34:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683192876; cv=none; d=google.com; s=arc-20160816; b=DmvEb3ep7ADyWs5T2drM7AkG/KdjevIXejd5Jc5hbTAOkFNxD8Y3R+UYgkGm9plFpj KfgmL5DnCaX4XAEY6s9eHCC4lAWYBjF0NbRjsJWA6+ODaaJ9ZRzWX4jUZWREKmJ8zgIs zgctP6Syi8b3smmRBncJkovXmO+jf6Eyucapa1TphnKXQuolcYKqt6wfUkVmGJfClN7G svNnYkPTsPDWdqnRl4Q+Tej/fYnyl23URcilw4Bb+7TxY3gsrOl48voDkM0LuCTtDIkp 3ol4niEwNjJvGUqacNNnhw4+Er8YYvSON948ER5j7wt4+SSldFg7AC2dkvKcoPvUoZfq oFjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=qGF7UmaBH/FVgI9XfFWCjX+CSRSvTZ2zRs9tkAUw7Fk=; b=ZxWaoqSEeEHOsoOyGTAxls4MpR/VmNZBm7FerUuRwXetkwrlu6sqJxmlly6ay6OrxK hTrnFjNAYESDFOd5lC2+OJ5Z4PJOMUja9gwDtFsV7RfPssfY3uRbyeNlkmz7jdgHLIWM gmY2rhTUjgimoKhTD8pHWZuNvU0qj8zoNpBQOFCXcztPsrXN9oiZz8RzVQWZD5TqmSxN IcCc/DvakiK73LRg+bwbh0JtJ1LpTpF42nxqeQRyyO+DY5Nh9iMnMqNRN6AXWV4GQVG6 htjW+f5U2LKhBS+YAeaV99kR1gGxm+Q10v9Yk4HBplq6DEP5S5KR+ME2ic63Q2bQ7vgD vm4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=GX6GAEn3; dkim=neutral (no key) header.i=@suse.de header.b=yiSVTOZo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i19-20020aa796f3000000b0063b1739532esi35840555pfq.139.2023.05.04.02.34.24; Thu, 04 May 2023 02:34:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=GX6GAEn3; dkim=neutral (no key) header.i=@suse.de header.b=yiSVTOZo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230059AbjEDJNn (ORCPT + 99 others); Thu, 4 May 2023 05:13:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37544 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230322AbjEDJNR (ORCPT ); Thu, 4 May 2023 05:13:17 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 42EA24490 for ; Thu, 4 May 2023 02:13:10 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B739133935; Thu, 4 May 2023 09:13:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1683191588; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qGF7UmaBH/FVgI9XfFWCjX+CSRSvTZ2zRs9tkAUw7Fk=; b=GX6GAEn3ppc93iRTLV0kkAvs/dM+3zlwLLqDzuVndhO3wsNo/7a1z1b+cLYtw3OKzYEV1D k7bsp0JadQ1CrZE4pvishN0Ht5V9+gXJ1v3JM7Iy7j1iemS9p68ujlwotdH7KKMvJXBYd3 yyvCGqE+Gcm+Pewwp0rkhQsxQ+3tWn8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1683191588; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qGF7UmaBH/FVgI9XfFWCjX+CSRSvTZ2zRs9tkAUw7Fk=; b=yiSVTOZoew2qpdCKheUmp9Q+RUXV+PRCkg7/E5WpPBbKXmyMWP1ryiLYWEl6yZYMzfXwS4 qZWBmXNVGBxBXJBw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A592E13444; Thu, 4 May 2023 09:13:08 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id SCqvJyR3U2TwTwAAMHmgww (envelope-from ); Thu, 04 May 2023 09:13:08 +0000 From: Daniel Wagner To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Chaitanya Kulkarni , Sagi Grimberg , Hannes Reinecke , James Smart , Daniel Wagner Subject: [RFC v3 9/9] nvme: introduce setup_transport() Date: Thu, 4 May 2023 11:12:59 +0200 Message-Id: <20230504091259.29100-10-dwagner@suse.de> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20230504091259.29100-1-dwagner@suse.de> References: <20230504091259.29100-1-dwagner@suse.de> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764955653648034288?= X-GMAIL-MSGID: =?utf-8?q?1764955653648034288?= Do the tag allocation in the new setup function. Nope, this doesn't work because the tag allocation wants to map the but we haven't allocated them yet. Signed-off-by: Daniel Wagner --- drivers/nvme/host/fabrics.c | 26 ++++------------ drivers/nvme/host/fabrics.h | 5 ++-- drivers/nvme/host/rdma.c | 60 +++++++++++++++++++++---------------- drivers/nvme/host/tcp.c | 31 +++++++++++-------- 4 files changed, 59 insertions(+), 63 deletions(-) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index 5f212cb9421a..06e9cf0c9e84 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -1325,12 +1325,6 @@ static int nvmf_configure_io_queues(struct nvme_ctrl *ctrl, bool new) if (ret) return ret; - if (new) { - ret = ctrl->fabrics_ops->alloc_tag_set(ctrl); - if (ret) - goto out_free_io_queues; - } - /* * Only start IO queues for which we have allocated the tagset * and limitted it to the available queues. On reconnects, the @@ -1374,9 +1368,6 @@ static int nvmf_configure_io_queues(struct nvme_ctrl *ctrl, bool new) nvmf_stop_io_queues(ctrl); out_cleanup_connect_q: nvme_cancel_tagset(ctrl); - if (new) - nvme_remove_io_tag_set(ctrl); -out_free_io_queues: nvmf_free_io_queues(ctrl); return ret; } @@ -1389,16 +1380,9 @@ static int nvmf_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) if (ret) return ret; - if (new) { - ret = ctrl->fabrics_ops->alloc_admin_tag_set(ctrl); - if (ret) - goto out_free_admin_queue; - - } - ret = __nvmf_start_admin_queue(ctrl); if (ret) - goto out_remove_admin_tag_set; + goto out_remove_admin_queue; ret = nvme_enable_ctrl(ctrl); if (ret) @@ -1418,10 +1402,7 @@ static int nvmf_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) out_stop_queue: __nvmf_stop_admin_queue(ctrl); nvme_cancel_admin_tagset(ctrl); -out_remove_admin_tag_set: - if (new) - nvme_remove_admin_tag_set(ctrl); -out_free_admin_queue: +out_remove_admin_queue: __nvmf_free_admin_queue(ctrl); return ret; } @@ -1489,6 +1470,9 @@ int nvmf_setup_ctrl(struct nvme_ctrl *ctrl, bool new) struct nvmf_ctrl_options *opts = ctrl->opts; int ret; + if (new) + ctrl->fabrics_ops->setup_transport(ctrl); + ret = nvmf_configure_admin_queue(ctrl, new); if (ret) return ret; diff --git a/drivers/nvme/host/fabrics.h b/drivers/nvme/host/fabrics.h index 345d6de6bc86..ad4734df7342 100644 --- a/drivers/nvme/host/fabrics.h +++ b/drivers/nvme/host/fabrics.h @@ -173,6 +173,7 @@ struct nvmf_transport_ops { }; struct nvme_fabrics_ops { + int (*setup_transport)(struct nvme_ctrl *ctrl); int (*alloc_admin_queue)(struct nvme_ctrl *ctrl); int (*start_admin_queue)(struct nvme_ctrl *ctrl); void (*stop_admin_queue)(struct nvme_ctrl *ctrl); @@ -182,9 +183,7 @@ struct nvme_fabrics_ops { void (*stop_io_queue)(struct nvme_ctrl *ctrl, int qid); void (*free_io_queue)(struct nvme_ctrl *ctrl, int qid); - /* these should be replaced with a single one setup_transport() */ - int (*alloc_admin_tag_set)(struct nvme_ctrl *ctrl); - int (*alloc_tag_set)(struct nvme_ctrl *ctrl); + /* there should move to setup_transport() as well */ unsigned int (*nr_io_queues)(struct nvme_ctrl *ctrl); void (*set_io_queues)(struct nvme_ctrl *ctrl, unsigned int nr_io_queues); }; diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 023316fdc2c6..015a6bde732a 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -743,6 +743,39 @@ static void nvme_rdma_stop_io_queue(struct nvme_ctrl *nctrl, int qid) __nvme_rdma_stop_queue(queue); } +static int nvme_rdma_setup_transport(struct nvme_ctrl *ctrl) +{ + unsigned int cmd_size; + int ret; + + ret = nvme_alloc_admin_tag_set(ctrl, &to_rdma_ctrl(ctrl)->admin_tag_set, + &nvme_rdma_admin_mq_ops, + sizeof(struct nvme_rdma_request) + + NVME_RDMA_DATA_SGL_SIZE); + if (ret) + return ret; + + cmd_size = sizeof(struct nvme_rdma_request) + + NVME_RDMA_DATA_SGL_SIZE; + + if (ctrl->max_integrity_segments) + cmd_size += sizeof(struct nvme_rdma_sgl) + + NVME_RDMA_METADATA_SGL_SIZE; + + ret = nvme_alloc_io_tag_set(ctrl, &to_rdma_ctrl(ctrl)->tag_set, + &nvme_rdma_mq_ops, + ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, + cmd_size); + if (ret) + goto out_free_admin_tag_set; + + return 0; + +out_free_admin_tag_set: + nvme_remove_admin_tag_set(ctrl); + return ret; +} + static unsigned int nvme_rdma_nr_io_queues(struct nvme_ctrl *ctrl) { struct ib_device *ibdev = to_rdma_ctrl(ctrl)->device->dev; @@ -802,21 +835,6 @@ static void nvme_rdma_set_io_queues(struct nvme_ctrl *nctrl, } } -static int nvme_rdma_alloc_tag_set(struct nvme_ctrl *ctrl) -{ - unsigned int cmd_size = sizeof(struct nvme_rdma_request) + - NVME_RDMA_DATA_SGL_SIZE; - - if (ctrl->max_integrity_segments) - cmd_size += sizeof(struct nvme_rdma_sgl) + - NVME_RDMA_METADATA_SGL_SIZE; - - return nvme_alloc_io_tag_set(ctrl, &to_rdma_ctrl(ctrl)->tag_set, - &nvme_rdma_mq_ops, - ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, - cmd_size); -} - static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(nctrl); @@ -834,15 +852,6 @@ static void nvme_rdma_free_ctrl(struct nvme_ctrl *nctrl) kfree(ctrl); } -static int nvme_rdma_alloc_admin_tag_set(struct nvme_ctrl *ctrl) -{ - - return nvme_alloc_admin_tag_set(ctrl, &to_rdma_ctrl(ctrl)->admin_tag_set, - &nvme_rdma_admin_mq_ops, - sizeof(struct nvme_rdma_request) + - NVME_RDMA_DATA_SGL_SIZE); -} - static void nvme_rdma_end_request(struct nvme_rdma_request *req) { struct request *rq = blk_mq_rq_from_pdu(req); @@ -1915,6 +1924,7 @@ nvme_rdma_existing_controller(struct nvmf_ctrl_options *opts) } static struct nvme_fabrics_ops nvme_rdma_fabrics_ops = { + .setup_transport = nvme_rdma_setup_transport, .alloc_admin_queue = nvme_rdma_alloc_admin_queue, .free_admin_queue = nvme_rdma_free_admin_queue, .start_admin_queue = nvme_rdma_start_admin_queue, @@ -1923,8 +1933,6 @@ static struct nvme_fabrics_ops nvme_rdma_fabrics_ops = { .free_io_queue = nvme_rdma_free_io_queue, .start_io_queue = nvme_rdma_start_io_queue, .stop_io_queue = nvme_rdma_stop_io_queue, - .alloc_admin_tag_set = nvme_rdma_alloc_admin_tag_set, - .alloc_tag_set = nvme_rdma_alloc_tag_set, .nr_io_queues = nvme_rdma_nr_io_queues, .set_io_queues = nvme_rdma_set_io_queues, }; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index dfdf35b32adc..f91575b944a2 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -1720,20 +1720,26 @@ static void nvme_tcp_stop_io_queue(struct nvme_ctrl *nctrl, int qid) __nvme_tcp_stop_queue(queue); } -static int nvme_tcp_alloc_admin_tag_set(struct nvme_ctrl *ctrl) +static int nvme_tcp_setup_transport(struct nvme_ctrl *ctrl) { - return nvme_alloc_admin_tag_set(ctrl, &to_tcp_ctrl(ctrl)->admin_tag_set, - &nvme_tcp_admin_mq_ops, - sizeof(struct nvme_tcp_request)); -} + int ret; -static int nvme_tcp_alloc_tag_set(struct nvme_ctrl *ctrl) -{ - return nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, - &nvme_tcp_mq_ops, - ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, - sizeof(struct nvme_tcp_request)); + ret = nvme_alloc_admin_tag_set(ctrl, &to_tcp_ctrl(ctrl)->admin_tag_set, + &nvme_tcp_admin_mq_ops, + sizeof(struct nvme_tcp_request)); + if (ret) + return ret; + + ret = nvme_alloc_io_tag_set(ctrl, &to_tcp_ctrl(ctrl)->tag_set, + &nvme_tcp_mq_ops, + ctrl->opts->nr_poll_queues ? HCTX_MAX_TYPES : 2, + sizeof(struct nvme_tcp_request)); + if (ret) + goto out_free_admin_tag_set; +out_free_admin_tag_set: + nvme_remove_admin_tag_set(ctrl); + return ret; } static unsigned int nvme_tcp_nr_io_queues(struct nvme_ctrl *ctrl) @@ -2155,6 +2161,7 @@ nvme_tcp_existing_controller(struct nvmf_ctrl_options *opts) } static struct nvme_fabrics_ops nvme_tcp_fabrics_ops = { + .setup_transport = nvme_tcp_setup_transport, .alloc_admin_queue = nvme_tcp_alloc_admin_queue, .free_admin_queue = nvme_tcp_free_admin_queue, .start_admin_queue = nvme_tcp_start_admin_queue, @@ -2163,8 +2170,6 @@ static struct nvme_fabrics_ops nvme_tcp_fabrics_ops = { .free_io_queue = nvme_tcp_free_io_queue, .start_io_queue = nvme_tcp_start_io_queue, .stop_io_queue = nvme_tcp_stop_io_queue, - .alloc_admin_tag_set = nvme_tcp_alloc_admin_tag_set, - .alloc_tag_set = nvme_tcp_alloc_tag_set, .nr_io_queues = nvme_tcp_nr_io_queues, .set_io_queues = nvme_tcp_set_io_queues, };