From patchwork Sat Oct 22 07:25:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Greg KH X-Patchwork-Id: 7702 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4242:0:0:0:0:0 with SMTP id s2csp1109392wrr; Sat, 22 Oct 2022 01:48:18 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6nQ1Yd0XQEgbg46A2s9gRJuR6UtfFIq2rf6PeM9fXHw7+2eQJATtE+m6tTGqOJdPIDsdqm X-Received: by 2002:a17:906:ef8c:b0:78d:46b7:6847 with SMTP id ze12-20020a170906ef8c00b0078d46b76847mr18879600ejb.241.1666428498329; Sat, 22 Oct 2022 01:48:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666428498; cv=none; d=google.com; s=arc-20160816; b=x66nWxM6dTumuHK2CFywFzMH1JgZIpTwArd64n90pFKaV/JLNY3YlQsavziNjZsRQ3 ydu+9mo5aBVcrWx+a17AW56ogbi3skU6T3Mg1RY+VmPAg4lHG7GbbROQLeSm/ZV/lAqr TUTHVfsuDgnp94hT+pkKCCVKF0qHwuK7muwuYvrAbGFSc0pFCUEgze2p3iIuV2MyI8wq HnChftu9YMvAMhlO9xoNyvVYCuhP0easvukfHdIkZ9w3uXlHKGky1ddAFivKeTNWxkK0 oR6I6/MunGsmTqkyqrbcbnOuQENxsdwZAcclW8aSxkbkmuYz+7MtQopZTNDS/G4LX7yT O/Pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=yVfSfWzjyZxG/GevmHUGwsWrc+WFjFeaFlNwI8q26SU=; b=bZvMRQvpx71N+/s1IKmK+8DREoTjbLJg27WE354E8GwXIhnjf5rjDuKP4Jj0t0FZL/ BhH12JW8Gn0en/MSqEd9Yf6ZTW4rAhq5a9pbNCkQoEz6c/TJh9YLXgfyzRYNlaNgPyCR kOmX4Qsg0HrYEusatLZg3lHX4FgWTg4MXS5q/Wt8B2aqb6Z6/S4oHN4MQQjYFIAQMM2W 5Yy3QnIdysMrbRqmqveaCbMC372gsw/k5XZi1G9yQ8xzzCIyAmVl2O/tXBOVUOPoN2fK 1LxiKYaHS23u6tf0CftuCHS3RigAgHoCp45EMxiZh8uvjF+2v7IcUT2EJJ2T5e5FSILC ziyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=rl7eMwlz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id di13-20020a170906730d00b0078d1e08f1e5si21594297ejc.919.2022.10.22.01.47.54; Sat, 22 Oct 2022 01:48:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=rl7eMwlz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234778AbiJVIrV (ORCPT + 99 others); Sat, 22 Oct 2022 04:47:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235157AbiJVIor (ORCPT ); Sat, 22 Oct 2022 04:44:47 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9AB672CB8DA; Sat, 22 Oct 2022 01:08:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 33AB6B82E10; Sat, 22 Oct 2022 07:56:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9981BC4314B; Sat, 22 Oct 2022 07:56:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1666425405; bh=tehG72v3c95uVxTzV5i+1h2vcogIiUZ8LZxNLsn5eu4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rl7eMwlzP3hVD2rOnT0D1yw91H66Jnw/IbjKv201vatLYrgcdPYuxhHkCicVrAKF3 CLKq82v1mEBUVr8M0OM2/yk0ZqnhW7NYNDxJkPIC0Sh3pnfngx1qNnnPBSwsiBFRQV PVcmKnWyLZb4So6CfQmxom9bqQQUVHDZcf/T0694= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Lee Duncan , Mike Christie , "Martin K. Petersen" , Sasha Levin Subject: [PATCH 5.19 460/717] scsi: iscsi: Run recv path from workqueue Date: Sat, 22 Oct 2022 09:25:39 +0200 Message-Id: <20221022072518.651221542@linuxfoundation.org> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221022072415.034382448@linuxfoundation.org> References: <20221022072415.034382448@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 X-Spam-Status: No, score=-7.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747376928981908162?= X-GMAIL-MSGID: =?utf-8?q?1747376928981908162?= From: Mike Christie [ Upstream commit f1d269765ee29da56b32818b7a08054484ed89f2 ] We don't always want to run the recv path from the network softirq because when we have to have multiple sessions sharing the same CPUs, some sessions can eat up the NAPI softirq budget and affect other sessions or users. Allow us to queue the recv handling to the iscsi workqueue so we can have the scheduler/wq code try to balance the work and CPU use across all sessions' worker threads. Note: It wasn't the original intent of the change but a nice side effect is that for some workloads/configs we get a nice performance boost. For a simple read heavy test: fio --direct=1 --filename=/dev/dm-0 --rw=randread --bs=256K --ioengine=libaio --iodepth=128 --numjobs=4 where the iscsi threads, fio jobs, and rps_cpus share CPUs we see a 32% throughput boost. We also see increases for small I/O IOPs tests but it's not as high. Link: https://lore.kernel.org/r/20220616224557.115234-4-michael.christie@oracle.com Reviewed-by: Lee Duncan Signed-off-by: Mike Christie Signed-off-by: Martin K. Petersen Stable-dep-of: 57569c37f0ad ("scsi: iscsi: iscsi_tcp: Fix null-ptr-deref while calling getpeername()") Signed-off-by: Sasha Levin --- drivers/scsi/iscsi_tcp.c | 65 ++++++++++++++++++++++++++++++++-------- drivers/scsi/iscsi_tcp.h | 2 ++ 2 files changed, 54 insertions(+), 13 deletions(-) diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c index da1dc345b873..10d7f2b7dd0e 100644 --- a/drivers/scsi/iscsi_tcp.c +++ b/drivers/scsi/iscsi_tcp.c @@ -52,6 +52,10 @@ static struct iscsi_transport iscsi_sw_tcp_transport; static unsigned int iscsi_max_lun = ~0; module_param_named(max_lun, iscsi_max_lun, uint, S_IRUGO); +static bool iscsi_recv_from_iscsi_q; +module_param_named(recv_from_iscsi_q, iscsi_recv_from_iscsi_q, bool, 0644); +MODULE_PARM_DESC(recv_from_iscsi_q, "Set to true to read iSCSI data/headers from the iscsi_q workqueue. The default is false which will perform reads from the network softirq context."); + static int iscsi_sw_tcp_dbg; module_param_named(debug_iscsi_tcp, iscsi_sw_tcp_dbg, int, S_IRUGO | S_IWUSR); @@ -122,20 +126,13 @@ static inline int iscsi_sw_sk_state_check(struct sock *sk) return 0; } -static void iscsi_sw_tcp_data_ready(struct sock *sk) +static void iscsi_sw_tcp_recv_data(struct iscsi_conn *conn) { - struct iscsi_conn *conn; - struct iscsi_tcp_conn *tcp_conn; + struct iscsi_tcp_conn *tcp_conn = conn->dd_data; + struct iscsi_sw_tcp_conn *tcp_sw_conn = tcp_conn->dd_data; + struct sock *sk = tcp_sw_conn->sock->sk; read_descriptor_t rd_desc; - read_lock_bh(&sk->sk_callback_lock); - conn = sk->sk_user_data; - if (!conn) { - read_unlock_bh(&sk->sk_callback_lock); - return; - } - tcp_conn = conn->dd_data; - /* * Use rd_desc to pass 'conn' to iscsi_tcp_recv. * We set count to 1 because we want the network layer to @@ -144,13 +141,48 @@ static void iscsi_sw_tcp_data_ready(struct sock *sk) */ rd_desc.arg.data = conn; rd_desc.count = 1; - tcp_read_sock(sk, &rd_desc, iscsi_sw_tcp_recv); - iscsi_sw_sk_state_check(sk); + tcp_read_sock(sk, &rd_desc, iscsi_sw_tcp_recv); /* If we had to (atomically) map a highmem page, * unmap it now. */ iscsi_tcp_segment_unmap(&tcp_conn->in.segment); + + iscsi_sw_sk_state_check(sk); +} + +static void iscsi_sw_tcp_recv_data_work(struct work_struct *work) +{ + struct iscsi_conn *conn = container_of(work, struct iscsi_conn, + recvwork); + struct iscsi_tcp_conn *tcp_conn = conn->dd_data; + struct iscsi_sw_tcp_conn *tcp_sw_conn = tcp_conn->dd_data; + struct sock *sk = tcp_sw_conn->sock->sk; + + lock_sock(sk); + iscsi_sw_tcp_recv_data(conn); + release_sock(sk); +} + +static void iscsi_sw_tcp_data_ready(struct sock *sk) +{ + struct iscsi_sw_tcp_conn *tcp_sw_conn; + struct iscsi_tcp_conn *tcp_conn; + struct iscsi_conn *conn; + + read_lock_bh(&sk->sk_callback_lock); + conn = sk->sk_user_data; + if (!conn) { + read_unlock_bh(&sk->sk_callback_lock); + return; + } + tcp_conn = conn->dd_data; + tcp_sw_conn = tcp_conn->dd_data; + + if (tcp_sw_conn->queue_recv) + iscsi_conn_queue_recv(conn); + else + iscsi_sw_tcp_recv_data(conn); read_unlock_bh(&sk->sk_callback_lock); } @@ -276,6 +308,9 @@ static int iscsi_sw_tcp_xmit_segment(struct iscsi_tcp_conn *tcp_conn, if (segment->total_copied + segment->size < segment->total_size) flags |= MSG_MORE; + if (tcp_sw_conn->queue_recv) + flags |= MSG_DONTWAIT; + /* Use sendpage if we can; else fall back to sendmsg */ if (!segment->data) { sg = segment->sg; @@ -557,6 +592,8 @@ iscsi_sw_tcp_conn_create(struct iscsi_cls_session *cls_session, conn = cls_conn->dd_data; tcp_conn = conn->dd_data; tcp_sw_conn = tcp_conn->dd_data; + INIT_WORK(&conn->recvwork, iscsi_sw_tcp_recv_data_work); + tcp_sw_conn->queue_recv = iscsi_recv_from_iscsi_q; tfm = crypto_alloc_ahash("crc32c", 0, CRYPTO_ALG_ASYNC); if (IS_ERR(tfm)) @@ -610,6 +647,8 @@ static void iscsi_sw_tcp_release_conn(struct iscsi_conn *conn) iscsi_sw_tcp_conn_restore_callbacks(conn); sock_put(sock->sk); + iscsi_suspend_rx(conn); + spin_lock_bh(&session->frwd_lock); tcp_sw_conn->sock = NULL; spin_unlock_bh(&session->frwd_lock); diff --git a/drivers/scsi/iscsi_tcp.h b/drivers/scsi/iscsi_tcp.h index 791453195099..850a018aefb9 100644 --- a/drivers/scsi/iscsi_tcp.h +++ b/drivers/scsi/iscsi_tcp.h @@ -28,6 +28,8 @@ struct iscsi_sw_tcp_send { struct iscsi_sw_tcp_conn { struct socket *sock; + struct work_struct recvwork; + bool queue_recv; struct iscsi_sw_tcp_send out; /* old values for socket callbacks */