From patchwork Sun Apr 23 12:17:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86699 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2167143vqo; Sun, 23 Apr 2023 05:18:56 -0700 (PDT) X-Google-Smtp-Source: AKy350at8S7QVPuHA4EPnfpvfvQKjYKFaToZR8Y9qutoOWYN5NybAahAritntusaciW5GwKZOSkx X-Received: by 2002:a17:902:e943:b0:1a6:5fa2:3293 with SMTP id b3-20020a170902e94300b001a65fa23293mr11110953pll.56.1682252335996; Sun, 23 Apr 2023 05:18:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682252335; cv=none; d=google.com; s=arc-20160816; b=rg42XvO9/3Vm43nzfO9FtR+611pWLKeHMUGiv50hm3cpy+jopcy+9+rMLMfTZBjSgy szOGFg2c/BHmeCrnYuAmWUBM8OCkQVSlXk0fq6WkJh+4MC2ZmUr1JcmzlP0YAxaffDig oZuk+pffD+N2vFgL551zbRecdZIeHZoRO1X8d158z/iYP1LW1UfGTNvDljL3kkY9Z90U xAfe5yyUFBqEFwlGi/5JAv+dpV9spIbeK+pQGT1EVZ223SB0S7pXVqnd3Q7XCg5JcxcL +hC8ZNZGj6XC0tJRB+yQW164ashU8KiVtgt9yJU7ROd3eiwb8I2bKWf6z4K9THo/dnSc O1Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=YkI8nfWdk85GUVbTaS0cXVCvjHkcgnSSecqNPswoDos=; b=p+9648u1q2+D050+IAC5Bwq16Qajjr5FPT5ygR1qaNf3Rwe6/Q9DuhZKZbqhTmJVXi zYMv92fKigpApCftQ/hdI82GmZb/aPNUpoaARcOQ2RUBp12+w8uVDtHLsVKu7evL///o Y2nGL7gvOFc8PkaVnj2P7sssg+o360R7MRaXT0m3EqvmPPlQdH+X+umYiZRNvojZ9/iD hn2RHgiWl3iu3D6cySx0nvvV/b5BbPMuos+x++qopXUTZwFNc4AdqhVVo2S++zMQXTVt zb+cks2ZL45ppX57DAKVTmVhTAbQJQbZ0TTO/V9lcUx2bDPCJtpwX38+jF9uohKIj/3Y Cxog== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u14-20020a170902e80e00b001a6fe422894si9385438plg.200.2023.04.23.05.18.41; Sun, 23 Apr 2023 05:18:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229580AbjDWMSC (ORCPT + 99 others); Sun, 23 Apr 2023 08:18:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229565AbjDWMSB (ORCPT ); Sun, 23 Apr 2023 08:18:01 -0400 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8411B10F0; Sun, 23 Apr 2023 05:17:59 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046050;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vgjux1M_1682252274; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vgjux1M_1682252274) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:17:56 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 1/9] net/smc: Decouple ism_dev from SMC-D device dump Date: Sun, 23 Apr 2023 20:17:43 +0800 Message-Id: <1682252271-2544-2-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763969425159774841?= X-GMAIL-MSGID: =?utf-8?q?1763969425159774841?= This patch helps to decouple SMC-D device and ISM device, allowing different underlying device forms, such as non-PCI devices. Signed-off-by: Wen Gu --- include/net/smc.h | 1 + net/smc/smc_ism.c | 6 +++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/include/net/smc.h b/include/net/smc.h index a002552..963ce9c 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -77,6 +77,7 @@ struct smcd_ops { struct smcd_dev { const struct smcd_ops *ops; void *priv; + struct device *parent_pci_dev; struct list_head list; spinlock_t lock; struct smc_connection **conn; diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index fbee249..4249fb9 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -231,11 +231,9 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, struct smc_pci_dev smc_pci_dev; struct nlattr *port_attrs; struct nlattr *attrs; - struct ism_dev *ism; int use_cnt = 0; void *nlh; - ism = smcd->priv; nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq, &smc_gen_nl_family, NLM_F_MULTI, SMC_NETLINK_GET_DEV_SMCD); @@ -250,7 +248,8 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, if (nla_put_u8(skb, SMC_NLA_DEV_IS_CRIT, use_cnt > 0)) goto errattr; memset(&smc_pci_dev, 0, sizeof(smc_pci_dev)); - smc_set_pci_values(to_pci_dev(ism->dev.parent), &smc_pci_dev); + if (smcd->parent_pci_dev) + smc_set_pci_values(to_pci_dev(smcd->parent_pci_dev), &smc_pci_dev); if (nla_put_u32(skb, SMC_NLA_DEV_PCI_FID, smc_pci_dev.pci_fid)) goto errattr; if (nla_put_u16(skb, SMC_NLA_DEV_PCI_CHID, smc_pci_dev.pci_pchid)) @@ -420,6 +419,7 @@ static void smcd_register_dev(struct ism_dev *ism) if (!smcd) return; smcd->priv = ism; + smcd->parent_pci_dev = ism->dev.parent; ism_set_priv(ism, &smc_ism_client, smcd); if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid)) smc_pnetid_by_table_smcd(smcd); From patchwork Sun Apr 23 12:17:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86712 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2173946vqo; Sun, 23 Apr 2023 05:35:38 -0700 (PDT) X-Google-Smtp-Source: AKy350YtLHMRLTgM7hucVSfnpnfHx60YrOEUzUjeQl/CNecjhUnJzx/gGFFhuftw8VwXcGO9XIfj X-Received: by 2002:a05:6a21:168b:b0:ec:7cc:2da6 with SMTP id np11-20020a056a21168b00b000ec07cc2da6mr11997749pzb.56.1682253338680; Sun, 23 Apr 2023 05:35:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682253338; cv=none; d=google.com; s=arc-20160816; b=uDQ56iRmLj6dMMmyORiOUB4hfPdwPvWQU6jGC0NpVZ0Y50m/fbAHFYOfzF60/iCWD/ NEuUtea4a7P1qnha1LaC/26jfdSVq4YYxXAmDQM36MGw8b6/ffzBjicdP8bevfZSst9j nLI8JKRDcBtQ14LeP6+ITEJkWxZ+gBWHhF+xZsV4cWalEKWVoHsaBxQRWQzarApYnz3I NamXT58NWq4/9Lrp6vGLVnjyHpn8+4+c0al1nerEmIj5hBGfAkDIrPasw35UQBVsfRM8 T1HcW9gbKeA2le2kY9cAOhMCcq6AhZr3xRtblCHDhlv157hltCwC5rc8oECVUvbuDbLX /O/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=Jbm8jBluBW3YzCIuB12UIZvTw55Nh1GRHKfNk9NIK8Y=; b=v26OGLiM7TFDkkWWc5WKJV7HUVe7Med8VFf9iGbJcbL1EInHkdIx0aDkAkEAF5zaXo gLrZ99sdsbhtxKWsD542s6txvgz/hil0r5f6Ov7KVem33827CiTGKBDqBIS64MxzBQK5 tYPTzdx4vO032xAO18EIrHBbOZT96BDlIzXXD0Bfnyt1GNE+5A6ujSP0SXTYKjG+9fF7 s95ZACe9LCcZ3dqT1wPFxWCK5/0MM9LTSkaRbSXZGfaMwfhnMpnzCCV1CWZ1OBBZam8x XhvlSkW1q29D55PEd3hHqO2TaB745vXd0eBmpB+hlQelLoJ3tumCPW4vm1fAxet+W5Zc OBNA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 2-20020a631442000000b004fb95c8f645si8856200pgu.392.2023.04.23.05.35.24; Sun, 23 Apr 2023 05:35:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229976AbjDWMSH (ORCPT + 99 others); Sun, 23 Apr 2023 08:18:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229697AbjDWMSC (ORCPT ); Sun, 23 Apr 2023 08:18:02 -0400 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23D171723; Sun, 23 Apr 2023 05:18:00 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0VgjtUoW_1682252276; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0VgjtUoW_1682252276) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:17:58 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 2/9] net/smc: Decouple ism_dev from SMC-D DMB registration Date: Sun, 23 Apr 2023 20:17:44 +0800 Message-Id: <1682252271-2544-3-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763970477174706937?= X-GMAIL-MSGID: =?utf-8?q?1763970477174706937?= This patch tries to decouple ISM device from SMC-D DMB registration, So that the register_dmb option is not restricted to be used by ISM device. Signed-off-by: Wen Gu --- drivers/s390/net/ism_drv.c | 5 +++-- include/net/smc.h | 4 ++-- net/smc/smc_ism.c | 7 ++----- 3 files changed, 7 insertions(+), 9 deletions(-) diff --git a/drivers/s390/net/ism_drv.c b/drivers/s390/net/ism_drv.c index 8acb9eb..5eeb54d 100644 --- a/drivers/s390/net/ism_drv.c +++ b/drivers/s390/net/ism_drv.c @@ -796,9 +796,10 @@ static int smcd_query_rgid(struct smcd_dev *smcd, u64 rgid, u32 vid_valid, } static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, - struct ism_client *client) + void *client_priv) { - return ism_register_dmb(smcd->priv, (struct ism_dmb *)dmb, client); + return ism_register_dmb(smcd->priv, (struct ism_dmb *)dmb, + (struct ism_client *)client_priv); } static int smcd_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) diff --git a/include/net/smc.h b/include/net/smc.h index 963ce9c..26206d2 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -50,13 +50,12 @@ struct smcd_dmb { #define ISM_ERROR 0xFFFF struct smcd_dev; -struct ism_client; struct smcd_ops { int (*query_remote_gid)(struct smcd_dev *dev, u64 rgid, u32 vid_valid, u32 vid); int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb, - struct ism_client *client); + void *client_priv); int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb); int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id); int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id); @@ -77,6 +76,7 @@ struct smcd_ops { struct smcd_dev { const struct smcd_ops *ops; void *priv; + void *client_priv; struct device *parent_pci_dev; struct list_head list; spinlock_t lock; diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index 4249fb9..d4709ca 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -200,7 +200,6 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc) int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, struct smc_buf_desc *dmb_desc) { -#if IS_ENABLED(CONFIG_ISM) struct smcd_dmb dmb; int rc; @@ -209,7 +208,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, dmb.sba_idx = dmb_desc->sba_idx; dmb.vlan_id = lgr->vlan_id; dmb.rgid = lgr->peer_gid; - rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, &smc_ism_client); + rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, lgr->smcd->client_priv); if (!rc) { dmb_desc->sba_idx = dmb.sba_idx; dmb_desc->token = dmb.dmb_tok; @@ -218,9 +217,6 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, dmb_desc->len = dmb.dmb_len; } return rc; -#else - return 0; -#endif } static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, @@ -419,6 +415,7 @@ static void smcd_register_dev(struct ism_dev *ism) if (!smcd) return; smcd->priv = ism; + smcd->client_priv = &smc_ism_client; smcd->parent_pci_dev = ism->dev.parent; ism_set_priv(ism, &smc_ism_client, smcd); if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid)) From patchwork Sun Apr 23 12:17:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86711 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2173939vqo; Sun, 23 Apr 2023 05:35:38 -0700 (PDT) X-Google-Smtp-Source: AKy350YRRksiw102toboomQbN1ALL7QHzqWuY2SAFOi5deI6Eea8jBrbaQdPNZhT7HG1+D4o8BXk X-Received: by 2002:a17:902:cec9:b0:1a5:1842:f7da with SMTP id d9-20020a170902cec900b001a51842f7damr18020897plg.6.1682253338253; Sun, 23 Apr 2023 05:35:38 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682253338; cv=none; d=google.com; s=arc-20160816; b=Xkkp5OhAUT1GkljkoMX3lPasff3qmXw7CUZ/iN7hC+qv4oNtN10/zggS4dIUyvO8Pg pf0+wMdEwTddVfnWWOIjDK8aACN4Ld53GX42dVo1JFnJ+s6R50ZIdtHYfAENU3O8H1d6 hvsEOO0Kia07cpUu0y5YP28lnPqxpSqEWQLxqrfqcnfHGpuwUGDfkfRmy7E1rlPUjAk8 c9Q4A1TN5XrDQJTyUWsLeDBFbW7RH7VnhL7txlnW5mWwk3F2PbClHd6EsQYFu5uHpyBS w1LOU4i4ASJDckyQFLhP90VfAlZ66wvRueRg5fbaZwFfkhvFZfsbU2a416olAnvZN6Nk cIFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=dwOnnlFrAXrktW1lI0OTmz1+yBrD5vcOOkfiQEPC/14=; b=jQmyT9v8zHJCzNhbTuPrCBe/O9yisWOoTN1I/yuwJihFI/iCbWaTCjbmN8+UVFrlaR Z0rZAzHm1rzTcDtzC7142yXPA3fEBTq9wub7ccn/YJSheCYZDTvy21mVM6FQrdalLM6V 0tGp1SesDWdvAIMSQ41fi2vsbSRoDl/BsirG38et5d5fakOIvCcswt44gKtgQUawhSL0 U8pKMo1LrFhC2pmIoz2peAwrzBbQV+cGHz9nyt8ZgDBotEkMZg+GxW7wWTBlr+PhbOos XmSJzbKSqJuVSMQMrP5tVW0o8pLogAvEwno6L1oxmG/y+7NsgJzimny/0VYZrdUZfQRs tTwg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id go8-20020a17090b03c800b002373125c085si8893037pjb.141.2023.04.23.05.35.23; Sun, 23 Apr 2023 05:35:38 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230185AbjDWMSW (ORCPT + 99 others); Sun, 23 Apr 2023 08:18:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230038AbjDWMSQ (ORCPT ); Sun, 23 Apr 2023 08:18:16 -0400 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F9DD10F0; Sun, 23 Apr 2023 05:18:04 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0VgjzSOT_1682252278; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0VgjzSOT_1682252278) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:17:59 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 3/9] net/smc: Extract v2 check helper from SMC-D device registration Date: Sun, 23 Apr 2023 20:17:45 +0800 Message-Id: <1682252271-2544-4-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763970476528315595?= X-GMAIL-MSGID: =?utf-8?q?1763970476528315595?= This patch extracts v2-capable logic from the process of registering the ISM device as an SMC-D device, so that the registration process of other underlying devices can reuse it. Signed-off-by: Wen Gu --- net/smc/smc_ism.c | 27 +++++++++++++++++---------- net/smc/smc_ism.h | 1 + 2 files changed, 18 insertions(+), 10 deletions(-) diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index d4709ca..8ad4c71 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -69,6 +69,22 @@ bool smc_ism_is_v2_capable(void) return smc_ism_v2_capable; } +/* must be called under smcd_dev_list.mutex lock */ +void smc_ism_check_v2_capable(struct smcd_dev *smcd) +{ + u8 *system_eid = NULL; + + if (!list_empty(&smcd_dev_list.list)) + return; + + system_eid = smcd->ops->get_system_eid(); + if (smcd->ops->supports_v2()) { + smc_ism_v2_capable = true; + memcpy(smc_ism_v2_system_eid, system_eid, + SMC_MAX_EID_LEN); + } +} + /* Set a connection using this DMBE. */ void smc_ism_set_conn(struct smc_connection *conn) { @@ -422,16 +438,7 @@ static void smcd_register_dev(struct ism_dev *ism) smc_pnetid_by_table_smcd(smcd); mutex_lock(&smcd_dev_list.mutex); - if (list_empty(&smcd_dev_list.list)) { - u8 *system_eid = NULL; - - system_eid = smcd->ops->get_system_eid(); - if (smcd->ops->supports_v2()) { - smc_ism_v2_capable = true; - memcpy(smc_ism_v2_system_eid, system_eid, - SMC_MAX_EID_LEN); - } - } + smc_ism_check_v2_capable(smcd); /* sort list: devices without pnetid before devices with pnetid */ if (smcd->pnetid[0]) list_add_tail(&smcd->list, &smcd_dev_list.list); diff --git a/net/smc/smc_ism.h b/net/smc/smc_ism.h index 832b2f4..14d2e77 100644 --- a/net/smc/smc_ism.h +++ b/net/smc/smc_ism.h @@ -42,6 +42,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size, void smc_ism_get_system_eid(u8 **eid); u16 smc_ism_get_chid(struct smcd_dev *dev); bool smc_ism_is_v2_capable(void); +void smc_ism_check_v2_capable(struct smcd_dev *dev); int smc_ism_init(void); void smc_ism_exit(void); int smcd_nl_get_device(struct sk_buff *skb, struct netlink_callback *cb); From patchwork Sun Apr 23 12:17:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86703 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2170399vqo; Sun, 23 Apr 2023 05:27:33 -0700 (PDT) X-Google-Smtp-Source: AKy350ZCy8Fuf5e8I0XN+iZrLyE1iDFNrqPXGxYY9LwYG9UMuGtxYiYvSL6KivPEbYRlGZUDNV1t X-Received: by 2002:a17:903:2409:b0:1a2:89eb:3d1a with SMTP id e9-20020a170903240900b001a289eb3d1amr10008633plo.6.1682252853172; Sun, 23 Apr 2023 05:27:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682252853; cv=none; d=google.com; s=arc-20160816; b=FJwpyH7fvCU0FMHXcGUGoniB45kacXTvDIuGQ/qbialmrrlHyVSH3CZF39rxUTPzaT zeG980H5CEKPJ0a4GLjz476XiKPlHgIN+TAwlMjID8ZVi/3hk5xn1XVuQahY6CvfhHgT lL00gI7wNdSa0+/ZS9+RHKYoT58CC7dLqwEfc2K3GySIg+8JNzTjuEgEOPfMsbQI9FtB DrHvZ+nAY9lZ9+3v5hjt6LEcWkhwTQ5V5VBbai9urLjOLKr5BGCseMr0sQ214TX9R3Tq xsyirNQbe1xI3KIRgvJ5Mmm8e4k5G/YbPQ5qa3ieLSOrpKHlkDhmxn1Y5rPwRnlTB2Uc lGXA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=j+LVhqP+pFRiy+IqizDpnmtpsQwLbxkB1mHkRDo4zzI=; b=H/CQjRWZdjrJM46eZDo1cynxvcZvSkoItgDbPRBywgM3GfZBt31sMQMpKeP+IdUuh8 czB29A9rXRy2g9To/w/LkmNm2SLnIc8JeLggYcgERqnF1E+o1zFzMEsA6xhAYSxT0BBj 9N5pbNRDzpzvSYg3Vde+nETyzNgQtiZt0jw0yz2rmQKAO8ZYPWO9hFoSUbL1HjzW9Veu V1SeS8lmlFuLi706AGrMCqATorlZyiTAsxE0mZXXCu1HokUkDQnwJSqy8y4DpcWri5Q9 jGHA1YDs2AjesalPhJxM7XMoO4YLaEtNJPCDeTgRor4cpZ3/1L1Js/E6EchuZD5GNVr1 kqKA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ij6-20020a170902ab4600b0019cc4426180si8787582plb.240.2023.04.23.05.27.16; Sun, 23 Apr 2023 05:27:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230234AbjDWMSY (ORCPT + 99 others); Sun, 23 Apr 2023 08:18:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229852AbjDWMSU (ORCPT ); Sun, 23 Apr 2023 08:18:20 -0400 Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF1771FE3; Sun, 23 Apr 2023 05:18:04 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vgjux3C_1682252280; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vgjux3C_1682252280) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:01 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 4/9] net/smc: Introduce SMC-D loopback device Date: Sun, 23 Apr 2023 20:17:46 +0800 Message-Id: <1682252271-2544-5-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763969967775779696?= X-GMAIL-MSGID: =?utf-8?q?1763969967775779696?= This patch introduces a kind of loopback device for SMC-D, thus enabling the SMC communication between two local sockets within one OS instance. The loopback device supports basic capabilities defined by SMC-D options, and exposed as an SMC-D v2 device. The GID of loopback device is random generated, CHID is 0xFFFF and SEID is SMCD_DEFAULT_V2_SEID. TODO: - The hierarchy preference of coexistent SMC-D devices. Signed-off-by: Wen Gu --- include/net/smc.h | 6 + net/smc/Makefile | 2 +- net/smc/af_smc.c | 12 +- net/smc/smc_cdc.c | 9 +- net/smc/smc_cdc.h | 1 + net/smc/smc_ism.h | 2 + net/smc/smc_loopback.c | 406 +++++++++++++++++++++++++++++++++++++++++++++++++ net/smc/smc_loopback.h | 51 +++++++ 8 files changed, 486 insertions(+), 3 deletions(-) create mode 100644 net/smc/smc_loopback.c create mode 100644 net/smc/smc_loopback.h diff --git a/include/net/smc.h b/include/net/smc.h index 26206d2..021ca42 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -41,6 +41,12 @@ struct smcd_dmb { dma_addr_t dma_addr; }; +struct smcd_seid { + u8 seid_string[24]; + u8 serial_number[4]; + u8 type[4]; +}; + #define ISM_EVENT_DMB 0 #define ISM_EVENT_GID 1 #define ISM_EVENT_SWR 2 diff --git a/net/smc/Makefile b/net/smc/Makefile index 875efcd..a8c3711 100644 --- a/net/smc/Makefile +++ b/net/smc/Makefile @@ -4,5 +4,5 @@ obj-$(CONFIG_SMC) += smc.o obj-$(CONFIG_SMC_DIAG) += smc_diag.o smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o -smc-y += smc_tracepoint.o +smc-y += smc_tracepoint.o smc_loopback.o smc-$(CONFIG_SYSCTL) += smc_sysctl.o diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 50c38b6..3230309 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -53,6 +53,7 @@ #include "smc_stats.h" #include "smc_tracepoint.h" #include "smc_sysctl.h" +#include "smc_loopback.h" static DEFINE_MUTEX(smc_server_lgr_pending); /* serialize link group * creation on server @@ -3482,15 +3483,23 @@ static int __init smc_init(void) goto out_sock; } + rc = smc_loopback_init(); + if (rc) { + pr_err("%s: smc_loopback_init fails with %d\n", __func__, rc); + goto out_ib; + } + rc = tcp_register_ulp(&smc_ulp_ops); if (rc) { pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc); - goto out_ib; + goto out_lo; } static_branch_enable(&tcp_have_smc); return 0; +out_lo: + smc_loopback_exit(); out_ib: smc_ib_unregister_client(); out_sock: @@ -3528,6 +3537,7 @@ static void __exit smc_exit(void) tcp_unregister_ulp(&smc_ulp_ops); sock_unregister(PF_SMC); smc_core_exit(); + smc_loopback_exit(); smc_ib_unregister_client(); smc_ism_exit(); destroy_workqueue(smc_close_wq); diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index 89105e9..2f79bac 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -410,7 +410,14 @@ static void smc_cdc_msg_recv(struct smc_sock *smc, struct smc_cdc_msg *cdc) */ static void smcd_cdc_rx_tsklet(struct tasklet_struct *t) { - struct smc_connection *conn = from_tasklet(conn, t, rx_tsklet); + struct smc_connection *conn = + from_tasklet(conn, t, rx_tsklet); + + smcd_cdc_rx_handler(conn); +} + +void smcd_cdc_rx_handler(struct smc_connection *conn) +{ struct smcd_cdc_msg *data_cdc; struct smcd_cdc_msg cdc; struct smc_sock *smc; diff --git a/net/smc/smc_cdc.h b/net/smc/smc_cdc.h index 696cc11..11559d4 100644 --- a/net/smc/smc_cdc.h +++ b/net/smc/smc_cdc.h @@ -301,5 +301,6 @@ int smcr_cdc_msg_send_validation(struct smc_connection *conn, struct smc_wr_buf *wr_buf); int smc_cdc_init(void) __init; void smcd_cdc_rx_init(struct smc_connection *conn); +void smcd_cdc_rx_handler(struct smc_connection *conn); #endif /* SMC_CDC_H */ diff --git a/net/smc/smc_ism.h b/net/smc/smc_ism.h index 14d2e77..d18c50a 100644 --- a/net/smc/smc_ism.h +++ b/net/smc/smc_ism.h @@ -15,6 +15,8 @@ #include "smc.h" +#define S390_ISM_IDENT_MASK 0x00FFFF + struct smcd_dev_list { /* List of SMCD devices */ struct list_head list; struct mutex mutex; /* Protects list of devices */ diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c new file mode 100644 index 0000000..4197201 --- /dev/null +++ b/net/smc/smc_loopback.c @@ -0,0 +1,406 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Shared Memory Communications Direct over loopback device. + * + * Provide a SMC-D loopback dummy device. + * + * Copyright (c) 2022, Alibaba Inc. + * + * Author: Wen Gu + * Tony Lu + * + */ + +#include +#include +#include + +#include "smc_cdc.h" +#include "smc_ism.h" +#include "smc_loopback.h" + +#define SMC_LO_SUPPORTS_V2 0x1 /* SMC-D loopback supports SMC-Dv2 */ +#define SMC_LO_DMA_ADDR_INVALID (~(dma_addr_t)0) + +static const char smc_lo_dev_name[] = "smcd_loopback_dev"; +static struct smcd_seid SMC_LO_SEID = { + .seid_string = "IBM-SYSZ-ISMSEID00000000", + .serial_number = "0000", + .type = "0000", +}; + +static struct smc_lo_dev *lo_dev; + +static void smc_lo_create_seid(struct smcd_seid *seid) +{ +#if IS_ENABLED(CONFIG_ISM) + struct cpuid id; + u16 ident_tail; + char tmp[5]; + + get_cpu_id(&id); + ident_tail = (u16)(id.ident & S390_ISM_IDENT_MASK); + snprintf(tmp, 5, "%04X", ident_tail); + memcpy(&seid->serial_number, tmp, 4); + snprintf(tmp, 5, "%04X", id.machine); + memcpy(&seid->type, tmp, 4); +#endif +} + +static void smc_lo_generate_id(struct smc_lo_dev *ldev) +{ + /* Note that local GID of loopback device is random generated, + * so there is a very little possibility of collision. The + * collision may cause a mistaken belief that two sides are on + * the same OS instance and loopback device can be used to + * communicate with each other. + * + * But here is a relief that when choosing loopback device in + * CLC handshake, local sndbuf will be attached to peer RMB (DMB), + * which involves another 64-bits rtoken check. + * + * Therefore, the probability of mistakenly believing loopback + * can be used is equivalent to the collision probability of + * 128-bits random numbers (64-bits GID and 64-bits rtoken). + */ + get_random_bytes(&ldev->local_gid, sizeof(ldev->local_gid)); + ldev->chid = SMC_LO_CHID; + smc_lo_create_seid(&SMC_LO_SEID); +} + +static int smc_lo_query_rgid(struct smcd_dev *smcd, u64 rgid, + u32 vid_valid, u32 vid) +{ + struct smc_lo_dev *ldev = smcd->priv; + + /* rgid should equal to lgid in loopback */ + if (!ldev || rgid != ldev->local_gid) + return -ENETUNREACH; + return 0; +} + +static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, + void *client_priv) +{ + struct smc_lo_dmb_node *dmb_node, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + int sba_idx, rc; + + /* check space for new dmb */ + for_each_clear_bit(sba_idx, ldev->sba_idx_mask, SMC_LODEV_MAX_DMBS) { + if (!test_and_set_bit(sba_idx, ldev->sba_idx_mask)) + break; + } + if (sba_idx == SMC_LODEV_MAX_DMBS) + return -ENOSPC; + + dmb_node = kzalloc(sizeof(*dmb_node), GFP_KERNEL); + if (!dmb_node) { + rc = -ENOMEM; + goto err_bit; + } + + dmb_node->sba_idx = sba_idx; + dmb_node->cpu_addr = kzalloc(dmb->dmb_len, GFP_KERNEL | + __GFP_NOWARN | __GFP_NORETRY | + __GFP_NOMEMALLOC); + if (!dmb_node->cpu_addr) { + rc = -ENOMEM; + goto err_node; + } + dmb_node->len = dmb->dmb_len; + dmb_node->dma_addr = SMC_LO_DMA_ADDR_INVALID; + +again: + /* add new dmb into hash table */ + get_random_bytes(&dmb_node->token, sizeof(dmb_node->token)); + write_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_node->token) { + /* make sure the tokens in the same OS are unique */ + if (tmp_node->token == dmb_node->token) { + write_unlock(&ldev->dmb_ht_lock); + goto again; + } + } + hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token); + write_unlock(&ldev->dmb_ht_lock); + + dmb->sba_idx = dmb_node->sba_idx; + dmb->dmb_tok = dmb_node->token; + dmb->cpu_addr = dmb_node->cpu_addr; + dmb->dma_addr = dmb_node->dma_addr; + dmb->dmb_len = dmb_node->len; + + return 0; + +err_node: + kfree(dmb_node); +err_bit: + clear_bit(sba_idx, ldev->sba_idx_mask); + return rc; +} + +static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) +{ + struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + /* remove dmb from hash table */ + write_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) { + if (tmp_node->token == dmb->dmb_tok) { + dmb_node = tmp_node; + break; + } + } + if (!dmb_node) { + write_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + hash_del(&dmb_node->list); + write_unlock(&ldev->dmb_ht_lock); + + clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask); + kfree(dmb_node->cpu_addr); + kfree(dmb_node); + + return 0; +} + +static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_del_vlan_id(struct smcd_dev *smcd, u64 vlan_id) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_set_vlan_required(struct smcd_dev *smcd) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_reset_vlan_required(struct smcd_dev *smcd) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_signal_event(struct smcd_dev *dev, u64 rgid, u32 trigger_irq, + u32 event_code, u64 info) +{ + return 0; +} + +static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx, + bool sf, unsigned int offset, void *data, + unsigned int size) +{ + struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + read_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) { + if (tmp_node->token == dmb_tok) { + rmb_node = tmp_node; + break; + } + } + if (!rmb_node) { + read_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + read_unlock(&ldev->dmb_ht_lock); + + memcpy((char *)rmb_node->cpu_addr + offset, data, size); + + if (sf) { + struct smc_connection *conn = + smcd->conn[rmb_node->sba_idx]; + + if (conn && !conn->killed) + smcd_cdc_rx_handler(conn); + } + return 0; +} + +static int smc_lo_supports_v2(void) +{ + return SMC_LO_SUPPORTS_V2; +} + +static u8 *smc_lo_get_system_eid(void) +{ + return SMC_LO_SEID.seid_string; +} + +static u64 smc_lo_get_local_gid(struct smcd_dev *smcd) +{ + return ((struct smc_lo_dev *)smcd->priv)->local_gid; +} + +static u16 smc_lo_get_chid(struct smcd_dev *smcd) +{ + return ((struct smc_lo_dev *)smcd->priv)->chid; +} + +static struct device *smc_lo_get_dev(struct smcd_dev *smcd) +{ + return &((struct smc_lo_dev *)smcd->priv)->dev; +} + +static const struct smcd_ops lo_ops = { + .query_remote_gid = smc_lo_query_rgid, + .register_dmb = smc_lo_register_dmb, + .unregister_dmb = smc_lo_unregister_dmb, + .add_vlan_id = smc_lo_add_vlan_id, + .del_vlan_id = smc_lo_del_vlan_id, + .set_vlan_required = smc_lo_set_vlan_required, + .reset_vlan_required = smc_lo_reset_vlan_required, + .signal_event = smc_lo_signal_event, + .move_data = smc_lo_move_data, + .supports_v2 = smc_lo_supports_v2, + .get_system_eid = smc_lo_get_system_eid, + .get_local_gid = smc_lo_get_local_gid, + .get_chid = smc_lo_get_chid, + .get_dev = smc_lo_get_dev, +}; + +static struct smcd_dev *smcd_lo_alloc_dev(const struct smcd_ops *ops, + int max_dmbs) +{ + struct smcd_dev *smcd; + + smcd = kzalloc(sizeof(*smcd), GFP_KERNEL); + if (!smcd) + return NULL; + + smcd->conn = kcalloc(max_dmbs, sizeof(struct smc_connection *), + GFP_KERNEL); + if (!smcd->conn) + goto out_smcd; + + smcd->ops = ops; + + spin_lock_init(&smcd->lock); + spin_lock_init(&smcd->lgr_lock); + INIT_LIST_HEAD(&smcd->vlan); + INIT_LIST_HEAD(&smcd->lgr_list); + init_waitqueue_head(&smcd->lgrs_deleted); + return smcd; + +out_smcd: + kfree(smcd); + return NULL; +} + +static int smcd_lo_register_dev(struct smc_lo_dev *ldev) +{ + struct smcd_dev *smcd; + + smcd = smcd_lo_alloc_dev(&lo_ops, SMC_LODEV_MAX_DMBS); + if (!smcd) + return -ENOMEM; + + ldev->smcd = smcd; + smcd->priv = ldev; + smcd->parent_pci_dev = NULL; + mutex_lock(&smcd_dev_list.mutex); + smc_ism_check_v2_capable(smcd); + list_add(&smcd->list, &smcd_dev_list.list); + mutex_unlock(&smcd_dev_list.mutex); + pr_warn_ratelimited("smc: adding smcd device %s with pnetid %.16s%s\n", + smc_lo_dev_name, smcd->pnetid, + smcd->pnetid_by_user ? " (user defined)" : ""); + return 0; +} + +static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev) +{ + struct smcd_dev *smcd = ldev->smcd; + + pr_warn_ratelimited("smc: removing smcd device %s\n", + smc_lo_dev_name); + smcd->going_away = 1; + smc_smcd_terminate_all(smcd); + mutex_lock(&smcd_dev_list.mutex); + list_del_init(&smcd->list); + mutex_unlock(&smcd_dev_list.mutex); +} + +static void smc_lo_dev_release(struct device *dev) +{ + struct smc_lo_dev *ldev = + container_of(dev, struct smc_lo_dev, dev); + struct smcd_dev *smcd = ldev->smcd; + + kfree(smcd->conn); + kfree(smcd); + kfree(ldev); +} + +static int smc_lo_dev_init(struct smc_lo_dev *ldev) +{ + smc_lo_generate_id(ldev); + rwlock_init(&ldev->dmb_ht_lock); + hash_init(ldev->dmb_ht); + + return smcd_lo_register_dev(ldev); +} + +static int smc_lo_dev_probe(void) +{ + struct smc_lo_dev *ldev; + int ret; + + ldev = kzalloc(sizeof(*ldev), GFP_KERNEL); + if (!ldev) + return -ENOMEM; + + ldev->dev.parent = NULL; + ldev->dev.release = smc_lo_dev_release; + device_initialize(&ldev->dev); + dev_set_name(&ldev->dev, smc_lo_dev_name); + ret = device_add(&ldev->dev); + if (ret) + goto free_dev; + + ret = smc_lo_dev_init(ldev); + if (ret) + goto put_dev; + + lo_dev = ldev; /* global loopback device */ + return 0; + +put_dev: + device_del(&ldev->dev); +free_dev: + kfree(ldev); + return ret; +} + +static void smc_lo_dev_exit(struct smc_lo_dev *ldev) +{ + smcd_lo_unregister_dev(ldev); +} + +static void smc_lo_dev_remove(void) +{ + if (!lo_dev) + return; + + smc_lo_dev_exit(lo_dev); + device_del(&lo_dev->dev); /* device_add in smc_lo_dev_probe */ + put_device(&lo_dev->dev); /* device_initialize in smc_lo_dev_probe */ +} + +int smc_loopback_init(void) +{ + return smc_lo_dev_probe(); +} + +void smc_loopback_exit(void) +{ + smc_lo_dev_remove(); +} diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h new file mode 100644 index 0000000..9d34aba --- /dev/null +++ b/net/smc/smc_loopback.h @@ -0,0 +1,51 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Shared Memory Communications Direct over loopback device. + * + * Provide a SMC-D loopback dummy device. + * + * Copyright (c) 2022, Alibaba Inc. + * + * Author: Wen Gu + * Tony Lu + * + */ + +#ifndef _SMC_LOOPBACK_H +#define _SMC_LOOPBACK_H + +#include +#include +#include +#include +#include + +#include "smc_core.h" + +#define SMC_LO_CHID 0xFFFF +#define SMC_LODEV_MAX_DMBS 5000 +#define SMC_LODEV_MAX_DMBS_BUCKETS 16 + +struct smc_lo_dmb_node { + struct hlist_node list; + u64 token; + u32 len; + u32 sba_idx; + void *cpu_addr; + dma_addr_t dma_addr; +}; + +struct smc_lo_dev { + struct smcd_dev *smcd; + struct device dev; + u16 chid; + u64 local_gid; + DECLARE_BITMAP(sba_idx_mask, SMC_LODEV_MAX_DMBS); + rwlock_t dmb_ht_lock; + DECLARE_HASHTABLE(dmb_ht, SMC_LODEV_MAX_DMBS_BUCKETS); +}; + +int smc_loopback_init(void); +void smc_loopback_exit(void); + +#endif /* _SMC_LOOPBACK_H */ From patchwork Sun Apr 23 12:17:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86708 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2172223vqo; Sun, 23 Apr 2023 05:31:53 -0700 (PDT) X-Google-Smtp-Source: AKy350aPGiagFyPt6oW+vJyj3zB/7NIOfYUWpyR3ievEfHJaKSbIRsscFC3qIgtfrzy7v98WsBLy X-Received: by 2002:a17:90b:4d8a:b0:246:ac0b:9d45 with SMTP id oj10-20020a17090b4d8a00b00246ac0b9d45mr10447072pjb.18.1682253112743; Sun, 23 Apr 2023 05:31:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682253112; cv=none; d=google.com; s=arc-20160816; b=Hot0OeRB2l7/Ef8p5ZSh5cFWsT27TKfA3qYMU3yl2066CdSQ7fWVOwvEHLjlNSAgbD bgYZwZvsJpAV6nDjwvOqAFbyHz2lCpyLzhtIQohZaN3w9PTJMUCvvht6nWLf0L8ov4Oi e35bBPO+fYWtFJve2SgX1nJA0KKZC39VjYImt3oEXwaUzRC5oMVYv4WXT0WNqbN60C0h 8GOqoUfcf16voFCRmUsfo9shzd1YoHktMSIf7MrqJwSDFRP9HUlaGtG9HUASfQOUZxOs bReIA7bfXaKTY6jX0289IQymDgoccfqdG5cy8rz07BIZyDOFdXut6I+dX6BC36HjYeKB WFaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=JwQlTodHIjFmazLlveTPIvjCBIO+GiN3uYOLRGgR6OU=; b=iT0VTlEKMGT5e2yzJbd1HkasYKTQx1zOYgFaLNDplGhgFY2igRj83rAevaa/Ny9f8U 2JMONjZmjL0K0vOuu+CQgqZEpr47vf/zrRMkczGDNaTpdeO1sLNUpXzgfTZai6BiHEOC C3WzrdKYBahE3DGYHVD8Bb8UPyDXANaDZPzKtyTlX9bVE8PpiuIdj0RXFapMizTDXgUo 2eI99KJOQLcqjAjhHdyiyLwNKXo1I3Ib5XmSzhHQdQvhR4OaHYo1pnkVN7hlXTbsQmAf nMQSgcmd6SG/8mfvJKBL+7mNcnYxsp3vHOy1alNGUntpAahf13320zGuWvsk1IaSZM9Y bBCA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s8-20020a17090aad8800b0024489068406si11700119pjq.113.2023.04.23.05.31.38; Sun, 23 Apr 2023 05:31:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230348AbjDWMS6 (ORCPT + 99 others); Sun, 23 Apr 2023 08:18:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47900 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230184AbjDWMSW (ORCPT ); Sun, 23 Apr 2023 08:18:22 -0400 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 909A91FCC; Sun, 23 Apr 2023 05:18:08 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R821e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vgjz4Dz_1682252281; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vgjz4Dz_1682252281) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:03 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 5/9] net/smc: Introduce an interface for getting DMB attribute Date: Sun, 23 Apr 2023 20:17:47 +0800 Message-Id: <1682252271-2544-6-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763970239644228483?= X-GMAIL-MSGID: =?utf-8?q?1763970239644228483?= On s390, since all OSs run on a kind of machine level hypervisor which is a partitioning hypervisor without paging, the sndbufs and DMBs in such case are unable to be mapped to the same physical memory. However, in other scene, such as communication within the same OS instance (loopback) or between guests of a paging hypervisor (eg. KVM), the sndbufs and DMBs can be mapped to the same physical memory to avoid memory copy from sndbufs to DMBs. So this patch introduces an interface to smcd_ops for users to judge whether DMB-map is available. And for reuse, the interface is designed to return DMB attribute, not only mappability. Signed-off-by: Wen Gu --- include/net/smc.h | 5 +++++ net/smc/smc_ism.c | 8 ++++++++ net/smc/smc_ism.h | 1 + 3 files changed, 14 insertions(+) diff --git a/include/net/smc.h b/include/net/smc.h index 021ca42..e39ac41 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -55,6 +55,10 @@ struct smcd_seid { #define ISM_ERROR 0xFFFF +enum { + ISM_DMB_MAPPABLE = 0, +}; + struct smcd_dev; struct smcd_ops { @@ -77,6 +81,7 @@ struct smcd_ops { u64 (*get_local_gid)(struct smcd_dev *dev); u16 (*get_chid)(struct smcd_dev *dev); struct device* (*get_dev)(struct smcd_dev *dev); + int (*get_dev_dmb_attr)(struct smcd_dev *dev); }; struct smcd_dev { diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index 8ad4c71..1d97e77 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -213,6 +213,14 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc) return rc; } +bool smc_ism_dmb_mappable(struct smcd_dev *smcd) +{ + if (smcd->ops->get_dev_dmb_attr && + (smcd->ops->get_dev_dmb_attr(smcd) & (1 << ISM_DMB_MAPPABLE))) + return true; + return false; +} + int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, struct smc_buf_desc *dmb_desc) { diff --git a/net/smc/smc_ism.h b/net/smc/smc_ism.h index d18c50a..0b1913a 100644 --- a/net/smc/smc_ism.h +++ b/net/smc/smc_ism.h @@ -40,6 +40,7 @@ struct smc_ism_vlanid { /* VLAN id set on ISM device */ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size, struct smc_buf_desc *dmb_desc); int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc); +bool smc_ism_dmb_mappable(struct smcd_dev *smcd); int smc_ism_signal_shutdown(struct smc_link_group *lgr); void smc_ism_get_system_eid(u8 **eid); u16 smc_ism_get_chid(struct smcd_dev *dev); From patchwork Sun Apr 23 12:17:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86709 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2172721vqo; Sun, 23 Apr 2023 05:32:58 -0700 (PDT) X-Google-Smtp-Source: AKy350b51NWtjWT6Nj3jPUIVeVMlm/98UzbTP5exDbgeoLd+E+Xj88Dz7OWJRuuhgKrWBpMIGKlh X-Received: by 2002:a17:902:eac4:b0:19e:874e:7275 with SMTP id p4-20020a170902eac400b0019e874e7275mr11425082pld.23.1682253178719; Sun, 23 Apr 2023 05:32:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682253178; cv=none; d=google.com; s=arc-20160816; b=BBUmM6M6ePUHwC2ZztOfFWoeoxkfPFP5GVMrjgK6i2x+1JSuJGqXqBzQzdAOOyC1RS CejgVGzNGEDB8HJmpX8opScLQ4Tg/APhLUITwpOCJpdtcbyYlBVNw1WWhk9B3oQR7t9A EhKROkaRF/YVRlyIxpPFZDpZyCp9rbLQpYDRbX9euUun7QJCLdqUJrvBCfZV1MrAPJXR yYLzYN3biBJ4SvOuJDmW4WO17lJmb+1cVJNvEoXf1beXGeegCpwgeHRpFBImQhUGx/fq /HR7v0ltX89xrwRB/Nr630OAr7ymMSytmqUOQEb25pEvGYkwwunskhH6/Ok6GGnit9Tw N3hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=4YIB9Lhv/hhcQjHZHBGPM1DGmQS1NKXtZjdLXy3tDq0=; b=utF61GV+kwEU5Bk9Ug/UZTvRMtPm8Nic4tnDYGOfNS/B5xKFbf1K/GeouzkETFQyHA 91m6eWE9ZrNLQjBtb+zZFZK478O01vVxZweVPHalApJGhPB/O+Mgz81qJU3eM66sAybD 4jIn2u00tsVz39H3M5FnH9+c/OSuL+mVsRiCZQ+UG0hNkLnQX3csx8Vf65xs0Qb7lSJE mgzDBwATUnPm3vLclpG3OSbNu2cZry5+V8KkAGa+hU4JN9UWoTMBfNT0IBz0WnnOHsEI /kWSWRpLu1o7Z9eaUYag7U22EEzs0fHOE0dvtpINlAnJYLAJ4Iqop0gFkvZrlwpBlZJF 41CQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u7-20020a17090a6a8700b002475bc0a189si8964827pjj.113.2023.04.23.05.32.44; Sun, 23 Apr 2023 05:32:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230178AbjDWMTM (ORCPT + 99 others); Sun, 23 Apr 2023 08:19:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230149AbjDWMSg (ORCPT ); Sun, 23 Apr 2023 08:18:36 -0400 Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A3852D7F; Sun, 23 Apr 2023 05:18:09 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0VgjzSS8_1682252284; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0VgjzSS8_1682252284) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:05 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 6/9] net/smc: Introudce interfaces for DMB attach and detach Date: Sun, 23 Apr 2023 20:17:48 +0800 Message-Id: <1682252271-2544-7-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763970308770691398?= X-GMAIL-MSGID: =?utf-8?q?1763970308770691398?= This patch extends smcd_ops, adding two more semantic for SMC-D DMB: - attach_dmb: Attach an already registered DMB to a specific buf_desc, so that we can refer to the DMB through this buf_desc. - detach_dmb: Reverse operation of attach_dmb. detach the DMB from the buf_desc. This interface extension is to prepare for the avoidance of memory copy from sndbuf to RMB with SMC-D device whose DMBs has ISM_DMB_MAPPABLE attribute. Signed-off-by: Wen Gu --- include/net/smc.h | 2 ++ net/smc/smc_ism.c | 31 +++++++++++++++++++++++++++++++ net/smc/smc_ism.h | 2 ++ 3 files changed, 35 insertions(+) diff --git a/include/net/smc.h b/include/net/smc.h index e39ac41..0132522 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -67,6 +67,8 @@ struct smcd_ops { int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb, void *client_priv); int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb); + int (*attach_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb); + int (*detach_dmb)(struct smcd_dev *dev, u64 token); int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id); int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id); int (*set_vlan_required)(struct smcd_dev *dev); diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index 1d97e77..925af13 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -243,6 +243,37 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, return rc; } +int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token, + struct smc_buf_desc *dmb_desc) +{ + struct smcd_dmb dmb; + int rc = 0; + + memset(&dmb, 0, sizeof(dmb)); + dmb.dmb_tok = token; + + if (!dev->ops->attach_dmb) + return -EINVAL; + + rc = dev->ops->attach_dmb(dev, &dmb); + if (!rc) { + dmb_desc->sba_idx = dmb.sba_idx; + dmb_desc->token = dmb.dmb_tok; + dmb_desc->cpu_addr = dmb.cpu_addr; + dmb_desc->dma_addr = dmb.dma_addr; + dmb_desc->len = dmb.dmb_len; + } + return rc; +} + +int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token) +{ + if (!dev->ops->detach_dmb) + return -EINVAL; + + return dev->ops->detach_dmb(dev, token); +} + static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, struct sk_buff *skb, struct netlink_callback *cb) diff --git a/net/smc/smc_ism.h b/net/smc/smc_ism.h index 0b1913a..31eb010 100644 --- a/net/smc/smc_ism.h +++ b/net/smc/smc_ism.h @@ -41,6 +41,8 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size, struct smc_buf_desc *dmb_desc); int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc); bool smc_ism_dmb_mappable(struct smcd_dev *smcd); +int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token, struct smc_buf_desc *dmb_desc); +int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token); int smc_ism_signal_shutdown(struct smc_link_group *lgr); void smc_ism_get_system_eid(u8 **eid); u16 smc_ism_get_chid(struct smcd_dev *dev); From patchwork Sun Apr 23 12:17:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86702 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2167475vqo; Sun, 23 Apr 2023 05:19:49 -0700 (PDT) X-Google-Smtp-Source: AKy350bNJohBCdhlXznfi8LvtO6MUzAnUaZiU16oVYHNrf9hJ9/L5TC0tlaLfn4W0K0Av3djsZ+y X-Received: by 2002:a05:6a21:670a:b0:f1:c63a:f7ea with SMTP id wh10-20020a056a21670a00b000f1c63af7eamr11548283pzb.37.1682252389666; Sun, 23 Apr 2023 05:19:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682252389; cv=none; d=google.com; s=arc-20160816; b=uoHW7+yNB8tgUbWsDlNKvZD8BrpPaQUQV+dXlx6Ge6KWXUZHLALOzaYXEEEi90rPbw fml1fffv3Y0YQ1KowArEPnMYltYDWdQ66pH05NWC2Zmwk+JJ31VyuSiF4ek0OBIMFNny /Vo1C8vwycPUGm5Z/n7nstbuSouNHfPuSSXvmygbP+N+2iuPw5XgNuIVV3D+8Q3xbREw +j0kxn9GqptfuMl7cJgot9e/DvTrxzA1Cuk8/mJvKU/2YgJcSPdThdnAYMoMiu9tzxJ/ RrEnSFWZ9WeNm94MNgkfo6xfe21vA04Byd0l+JQCBtCbqiuDnBXvADfnobFq+80GDrOS EHbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=gglhA54WP3E/KNLUXrFM7+WQ81IJllTkPQJWTT2TIVk=; b=D2wdGS6Y9+GNiXLijuoRsUBAPABbOwf8JPlIsAJt7RlT2DkZUiQtbT3WDmXiP4H6rb 2uoGnYGphtbPgcJkV/W4bEMv/6liErTbqj9sQ0xRlT/5Vvl7W+VELLQuWZElhiRmkz0t ar2qlx3ObJGQLXzvUaZVpq8/tkmtiGSDzei3S7alIrz3tD9TgeWrHcgCLEZ52YPnSUYs 7/jFHJZSbZm93T3QBp50CFFHFNbhhys5Hxe7yNb4ei+7v087GslFtUMvyY/FgmEHMHF4 x4vzIBibuan0B7djXOXYUwD4pYkz8/g0snyrsUf3vU4/Rt5PLPpC7d4G/t3i9+F7X5ZN K0yQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e13-20020a65688d000000b0050c06b2361dsi8821279pgt.72.2023.04.23.05.19.35; Sun, 23 Apr 2023 05:19:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230263AbjDWMTG (ORCPT + 99 others); Sun, 23 Apr 2023 08:19:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230261AbjDWMSk (ORCPT ); Sun, 23 Apr 2023 08:18:40 -0400 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AD95330E6; Sun, 23 Apr 2023 05:18:12 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045168;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0VgjtUvN_1682252286; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0VgjtUvN_1682252286) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:08 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 7/9] net/smc: Avoid data copy from sndbuf to peer RMB in SMC-D Date: Sun, 23 Apr 2023 20:17:49 +0800 Message-Id: <1682252271-2544-8-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763969481589888947?= X-GMAIL-MSGID: =?utf-8?q?1763969481589888947?= This patch aims to avoid data copy from local sndbuf to peer RMB by attaching local sndbuf to peer RMB when DMBs have ISM_DMB_MAPPABLE attribute. After this, local sndbuf and peer RMB share the same physical memory. +----------+ +----------+ | socket A | | socket B | +----------+ +----------+ | ^ | +---------+ | regard as | | ----------| local sndbuf | B's | regard as | | RMB | local RMB |-------> | | +---------+ 1. Actions on local RMB. a. Create or reuse RMB when connection is created; b. Unuse RMB when connection is freed; c. Free RMB when link group is freed; 2. Actions on local sndbuf. a. Attach local sndbuf to peer RMB by the rtoken exchanged through CLC message. Since then, accessing local sndbuf is equivalent to accessing peer RMB b. sndbuf_desc is exclusive to specific connection and won't be added to lgr buffer pool for reuse. c. Local sndbuf is detached from peer RMB and freed when connection is freed. Therefore, the data written to local sndbuf will directly reach peer RMB. Signed-off-by: Wen Gu --- net/smc/af_smc.c | 14 +++++++++++ net/smc/smc_core.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++- net/smc/smc_core.h | 1 + 3 files changed, 84 insertions(+), 1 deletion(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 3230309..0956cf5 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -1378,6 +1378,12 @@ static int smc_connect_ism(struct smc_sock *smc, } smc_conn_save_peer_info(smc, aclc); + + if (smc_ism_dmb_mappable(smc->conn.lgr->smcd)) { + rc = smcd_buf_attach(smc); + if (rc) + goto connect_abort; + } smc_close_init(smc); smc_rx_init(smc); smc_tx_init(smc); @@ -2436,6 +2442,14 @@ static void smc_listen_work(struct work_struct *work) mutex_unlock(&smc_server_lgr_pending); } smc_conn_save_peer_info(new_smc, cclc); + + if (ini->is_smcd && + smc_ism_dmb_mappable(new_smc->conn.lgr->smcd)) { + rc = smcd_buf_attach(new_smc); + if (rc) + goto out_decl; + } + smc_listen_out_connected(new_smc); SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini); goto out_free; diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c index 4543567..0fa26cc 100644 --- a/net/smc/smc_core.c +++ b/net/smc/smc_core.c @@ -1130,6 +1130,20 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb, } } +static void smcd_buf_detach(struct smc_connection *conn) +{ + struct smcd_dev *smcd = conn->lgr->smcd; + u64 peer_token = conn->peer_token; + + if (!conn->sndbuf_desc) + return; + + smc_ism_detach_dmb(smcd, peer_token); + + kfree(conn->sndbuf_desc); + conn->sndbuf_desc = NULL; +} + static void smc_buf_unuse(struct smc_connection *conn, struct smc_link_group *lgr) { @@ -1174,6 +1188,10 @@ void smc_conn_free(struct smc_connection *conn) if (!list_empty(&lgr->list)) smc_ism_unset_conn(conn); tasklet_kill(&conn->rx_tsklet); + + /* detach sndbuf from peer RMB */ + if (smc_ism_dmb_mappable(lgr->smcd)) + smcd_buf_detach(conn); } else { smc_cdc_wait_pend_tx_wr(conn); if (current_work() != &conn->abort_work) @@ -2425,15 +2443,23 @@ void smc_rmb_sync_sg_for_cpu(struct smc_connection *conn) */ int smc_buf_create(struct smc_sock *smc, bool is_smcd) { + bool sndbuf_created = false; int rc; + if (is_smcd && + smc_ism_dmb_mappable(smc->conn.lgr->smcd)) + goto create_rmb; + /* create send buffer */ rc = __smc_buf_create(smc, is_smcd, false); if (rc) return rc; + sndbuf_created = true; + +create_rmb: /* create rmb */ rc = __smc_buf_create(smc, is_smcd, true); - if (rc) { + if (rc && sndbuf_created) { down_write(&smc->conn.lgr->sndbufs_lock); list_del(&smc->conn.sndbuf_desc->list); up_write(&smc->conn.lgr->sndbufs_lock); @@ -2443,6 +2469,48 @@ int smc_buf_create(struct smc_sock *smc, bool is_smcd) return rc; } +int smcd_buf_attach(struct smc_sock *smc) +{ + struct smc_connection *conn = &smc->conn; + struct smcd_dev *smcd = conn->lgr->smcd; + u64 peer_token = conn->peer_token; + struct smc_buf_desc *buf_desc; + int rc; + + buf_desc = kzalloc(sizeof(*buf_desc), GFP_KERNEL); + if (!buf_desc) + return -ENOMEM; + + /* map local sndbuf desc to peer RMB, so operations on local + * sndbuf are equivalent to operations on peer RMB. + */ + rc = smc_ism_attach_dmb(smcd, peer_token, buf_desc); + if (rc) { + rc = SMC_CLC_DECL_MEM; + goto free; + } + + smc->sk.sk_sndbuf = buf_desc->len; + buf_desc->cpu_addr = (u8 *)buf_desc->cpu_addr + sizeof(struct smcd_cdc_msg); + buf_desc->len -= sizeof(struct smcd_cdc_msg); + conn->sndbuf_desc = buf_desc; + conn->sndbuf_desc->used = 1; + atomic_set(&conn->sndbuf_space, conn->sndbuf_desc->len); + return 0; + +free: + if (conn->rmb_desc) { + /* free local RMB as well */ + down_write(&conn->lgr->rmbs_lock); + list_del(&conn->rmb_desc->list); + up_write(&conn->lgr->rmbs_lock); + smc_buf_free(conn->lgr, true, conn->rmb_desc); + conn->rmb_desc = NULL; + } + kfree(buf_desc); + return rc; +} + static inline int smc_rmb_reserve_rtoken_idx(struct smc_link_group *lgr) { int i; diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h index 1645fba..e52cf70 100644 --- a/net/smc/smc_core.h +++ b/net/smc/smc_core.h @@ -524,6 +524,7 @@ void smc_smcd_terminate(struct smcd_dev *dev, u64 peer_gid, void smc_smcd_terminate_all(struct smcd_dev *dev); void smc_smcr_terminate_all(struct smc_ib_device *smcibdev); int smc_buf_create(struct smc_sock *smc, bool is_smcd); +int smcd_buf_attach(struct smc_sock *smc); int smc_uncompress_bufsize(u8 compressed); int smc_rmb_rtoken_handling(struct smc_connection *conn, struct smc_link *link, struct smc_clc_msg_accept_confirm *clc); From patchwork Sun Apr 23 12:17:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86705 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2171278vqo; Sun, 23 Apr 2023 05:30:05 -0700 (PDT) X-Google-Smtp-Source: AKy350ac6WEeB0eMIqLff3+uhGEmirAfQRduZSR1A8kxGVC8s9uNX755Lm0fWC7y6gnPTwxQ9tFJ X-Received: by 2002:a17:90b:81:b0:247:3895:e416 with SMTP id bb1-20020a17090b008100b002473895e416mr10681839pjb.16.1682253004987; Sun, 23 Apr 2023 05:30:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682253004; cv=none; d=google.com; s=arc-20160816; b=IEgrSBQU8G0f/FTVQdQA5CHmJH/VU9gqyxd3xG8x2j55/U3TLr2WXJP+wrZFdxrJNB JgG3JiCwO5SPojc/Fs2SDAXKocp305ZaKQp4ucYWqYhyWFSaDj8FS5mCb1SCuCWrIHLW xw6jPYG4RV3lVT98iqlAHFLHDTSOH1VXBTsx305+VFe5i1COvHOujnhKuEkGh+9fSWdg KoxAG2gGG+TiLbHeDTo0DMaNwf3DBL0YKM76+2sGCOaTS3pLHICNXCv+OTh82Hv18Px8 YoG58nGN5wa/UjMbD4B3ZCW5WvlIr3o+slM05MQWe2Pud+Uav0/LIlvEQF5Esh8uZWBN VrCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=WspUPY56llsxWqE0J/S3tGFmY3vq3FxldCe7rIFT1jM=; b=o2gJ05Zr7txxZMXL8tTgWHr7EoAcmrOcVQHZZ7v+/r2Gr+uTXJkNIjm9ezzUwzF1TG NTuAM3CG34XqvvSosz8hyJ9ZvS4B/mtt/sFYlAH+GHsY8lOElxrWloU8X9BwEQf7diMy i7T36rYGA4S8cE6cPqxElOKfKlJLpI6AE8zXMjgaV4sQ3Sf4jyvBPyE+k8QAfW44XQg2 8PnWnXoALBQhmjWXj8oXw5HHxZbVFV+hgpnnt1ZOVuJSRqd2MrASzBcnhnjanVZkK0gr t1qTR5OX5oMtQiImcjP6VPGBDdMNh8PZg14w1TOs7uconqYSAZ88PohUmUeDHYtFUzMZ lH6Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h10-20020a63f90a000000b005194ff488d9si8306992pgi.516.2023.04.23.05.29.50; Sun, 23 Apr 2023 05:30:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230237AbjDWMTX (ORCPT + 99 others); Sun, 23 Apr 2023 08:19:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230070AbjDWMSy (ORCPT ); Sun, 23 Apr 2023 08:18:54 -0400 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6C3E3A95; Sun, 23 Apr 2023 05:18:14 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R881e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vgjt3Kt_1682252288; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vgjt3Kt_1682252288) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:10 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 8/9] net/smc: Modify cursor update logic when using mappable DMB Date: Sun, 23 Apr 2023 20:17:50 +0800 Message-Id: <1682252271-2544-9-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763970126845621390?= X-GMAIL-MSGID: =?utf-8?q?1763970126845621390?= Since local sndbuf shares the same physical memory region with peer RMB when using mappable DMBs, the cursor update logic needs to be adapted. The main concern is to ensure that the data written by local to this memory region won't overwrite the data that has not been consumed by the peer. So in this scene, the fin_curs and sndbuf_space that were originally updated when sending out CDC message are not updated until the cons_curs update from the peer is received. Signed-off-by: Wen Gu --- net/smc/smc_cdc.c | 50 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index 2f79bac..915b8e7 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -18,6 +18,7 @@ #include "smc_tx.h" #include "smc_rx.h" #include "smc_close.h" +#include "smc_ism.h" /********************************** send *************************************/ @@ -256,17 +257,24 @@ int smcd_cdc_msg_send(struct smc_connection *conn) return rc; smc_curs_copy(&conn->rx_curs_confirmed, &curs, conn); conn->local_rx_ctrl.prod_flags.cons_curs_upd_req = 0; - /* Calculate transmitted data and increment free send buffer space */ - diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, - &conn->tx_curs_sent); - /* increased by confirmed number of bytes */ - smp_mb__before_atomic(); - atomic_add(diff, &conn->sndbuf_space); - /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ - smp_mb__after_atomic(); - smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + if (!smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * don't update the tx_curs_fin and sndbuf_space until + * peer has consumed the data in RMB. + */ - smc_tx_sndbuf_nonfull(smc); + /* Calculate transmitted data and increment free send buffer space */ + diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->tx_curs_sent); + /* increased by confirmed number of bytes */ + smp_mb__before_atomic(); + atomic_add(diff, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + + smc_tx_sndbuf_nonfull(smc); + } return rc; } @@ -324,7 +332,7 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, { union smc_host_cursor cons_old, prod_old; struct smc_connection *conn = &smc->conn; - int diff_cons, diff_prod; + int diff_cons, diff_prod, diff_tx; smc_curs_copy(&prod_old, &conn->local_rx_ctrl.prod, conn); smc_curs_copy(&cons_old, &conn->local_rx_ctrl.cons, conn); @@ -340,6 +348,26 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, atomic_add(diff_cons, &conn->peer_rmbe_space); /* guarantee 0 <= peer_rmbe_space <= peer_rmbe_size */ smp_mb__after_atomic(); + + if (conn->lgr->is_smcd && + smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * update tx_curs_fin and sndbuf_space when peer has + * consumed the data in it's RMB. + */ + + /* calculate peer rmb consumed data */ + diff_tx = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->local_rx_ctrl.cons); + /* increase local sndbuf space and fin_curs */ + smp_mb__before_atomic(); + atomic_add(diff_tx, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->local_rx_ctrl.cons, conn); + + smc_tx_sndbuf_nonfull(smc); + } } diff_prod = smc_curs_diff(conn->rmb_desc->len, &prod_old, From patchwork Sun Apr 23 12:17:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86717 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2179417vqo; Sun, 23 Apr 2023 05:50:43 -0700 (PDT) X-Google-Smtp-Source: AKy350b4HXcRn+qpb/p/ulQFzV934is7L9PHu8V0mu1Xn9nYj6hrAAbcWPXg2dFl8Wrw/NOvNLkv X-Received: by 2002:a05:6a00:150f:b0:63f:7663:3a3a with SMTP id q15-20020a056a00150f00b0063f76633a3amr1736674pfu.32.1682254242922; Sun, 23 Apr 2023 05:50:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682254242; cv=none; d=google.com; s=arc-20160816; b=HPnm5orM2/6GIbqB1VRjMCAaa5iX6vP9znmkOj2qa/RWna2NiLzZ1aLVHScwfzmV1+ zg7FUjRNfYguLqBsyZvOG1Cdi0aMyNHuluJOFPw9eg4sugZYQXscBjsIAPIc90dxEGmE GYfVsva0y+Es/bJBMO8aRkXrS5wzcnQM5n0xth/H/lEct6OFjAx9qJiqDxSz8Hwn38NE 22sPSg099I1ArJED8UCzKiCclmMBkhYG29bavuP4+GZdun30kA2jyKml15E7Lj8FeAp6 f/zJOyp6s4dW2TIYaJFe3sYiNFFw5Oow46U1gewrumQQKVfNTeh0+6NA6yE0lljnF10m k5xw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=b4qh2qkr3ewy0FdGOBovrLqy2VURCxbDkNBcUSAYecI=; b=oKwcJiVOvcjj0GGDLMfYvO7pDpCiWAYF66WTA0uSJGYW0G5gLDivNA764Vgcd15SCD CwpEloLpM/6Z3adKEpiUtyJvr1KlMP/LXMU/nfraejnmwcgHVs71YGTdYuURbAz+vR+W I+buPSI5hzlr82+qB/ce60junU+gIU2SfgpijPHhVr01pEwiK/cERiU3Ef+BpS90ry5F zGtqVfOkcvVO4m0ObPwldj6nluhL9/3zC5TYQftOfTOrdaYsRhtEUHWtIc1gjIbxFVKg 0qnAlrdoNHupeROzoHjEVXEG0Z5sOOmb4T2CPvDepIvj/XDvhEeOts9CsI1Q0YLveNO8 3BSQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c72-20020a621c4b000000b0063b8af2948bsi8643010pfc.3.2023.04.23.05.50.31; Sun, 23 Apr 2023 05:50:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230342AbjDWMTe (ORCPT + 99 others); Sun, 23 Apr 2023 08:19:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230323AbjDWMS6 (ORCPT ); Sun, 23 Apr 2023 08:18:58 -0400 Received: from out30-98.freemail.mail.aliyun.com (out30-98.freemail.mail.aliyun.com [115.124.30.98]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2FF63AB0; Sun, 23 Apr 2023 05:18:15 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vgjt3La_1682252290; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vgjt3La_1682252290) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:12 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 9/9] net/smc: Add interface implementation of loopback device Date: Sun, 23 Apr 2023 20:17:51 +0800 Message-Id: <1682252271-2544-10-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763971425262603158?= X-GMAIL-MSGID: =?utf-8?q?1763971425262603158?= This patch completes the specific implementation of loopback device for the newly added SMC-D DMB-related interface. The loopback device always provides mappable DMB because the device users are in the same OS instance. Signed-off-by: Wen Gu --- net/smc/smc_loopback.c | 101 +++++++++++++++++++++++++++++++++++++++++++++---- net/smc/smc_loopback.h | 5 +++ 2 files changed, 98 insertions(+), 8 deletions(-) diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 4197201..dc1909b 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -110,6 +110,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, } dmb_node->len = dmb->dmb_len; dmb_node->dma_addr = SMC_LO_DMA_ADDR_INVALID; + refcount_set(&dmb_node->refcnt, 1); again: /* add new dmb into hash table */ @@ -124,6 +125,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, } hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token); write_unlock(&ldev->dmb_ht_lock); + atomic_inc(&ldev->dmb_cnt); dmb->sba_idx = dmb_node->sba_idx; dmb->dmb_tok = dmb_node->token; @@ -145,11 +147,12 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; struct smc_lo_dev *ldev = smcd->priv; - /* remove dmb from hash table */ + /* find dmb from hash table */ write_lock(&ldev->dmb_ht_lock); hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) { if (tmp_node->token == dmb->dmb_tok) { dmb_node = tmp_node; + dmb_node->freeing = 1; break; } } @@ -157,16 +160,85 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) write_unlock(&ldev->dmb_ht_lock); return -EINVAL; } + write_unlock(&ldev->dmb_ht_lock); + + /* wait for dmb refcnt to be 0 */ + if (!refcount_dec_and_test(&dmb_node->refcnt)) + wait_event(ldev->dmbs_release, !refcount_read(&dmb_node->refcnt)); + + /* remove dmb from hash table */ + write_lock(&ldev->dmb_ht_lock); hash_del(&dmb_node->list); write_unlock(&ldev->dmb_ht_lock); clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask); + kfree(dmb_node->cpu_addr); kfree(dmb_node); + if (atomic_dec_and_test(&ldev->dmb_cnt)) + wake_up(&ldev->ldev_release); return 0; } +static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) +{ + struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + /* find dmb_node according to dmb->dmb_tok */ + read_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) { + if (tmp_node->token == dmb->dmb_tok && !tmp_node->freeing) { + dmb_node = tmp_node; + break; + } + } + if (!dmb_node) { + read_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + refcount_inc(&dmb_node->refcnt); + read_unlock(&ldev->dmb_ht_lock); + + /* provide dmb information */ + dmb->sba_idx = dmb_node->sba_idx; + dmb->dmb_tok = dmb_node->token; + dmb->cpu_addr = dmb_node->cpu_addr; + dmb->dma_addr = dmb_node->dma_addr; + dmb->dmb_len = dmb_node->len; + return 0; +} + +static int smc_lo_detach_dmb(struct smcd_dev *smcd, u64 token) +{ + struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + /* find dmb_node according to dmb->dmb_tok */ + read_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, token) { + if (tmp_node->token == token) { + dmb_node = tmp_node; + break; + } + } + if (!dmb_node) { + read_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + read_unlock(&ldev->dmb_ht_lock); + + if (refcount_dec_and_test(&dmb_node->refcnt)) + wake_up_all(&ldev->dmbs_release); + return 0; +} + +static int smc_lo_get_dev_dmb_attr(struct smcd_dev *smcd) +{ + return (1 << ISM_DMB_MAPPABLE); +} + static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id) { return -EOPNOTSUPP; @@ -199,7 +271,15 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx { struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node; struct smc_lo_dev *ldev = smcd->priv; - + struct smc_connection *conn; + + if (!sf) { + /* local sndbuf shares the same physical memory with + * peer RMB, so no need to copy data from local sndbuf + * to peer RMB. + */ + return 0; + } read_lock(&ldev->dmb_ht_lock); hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) { if (tmp_node->token == dmb_tok) { @@ -215,13 +295,10 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, unsigned int idx memcpy((char *)rmb_node->cpu_addr + offset, data, size); - if (sf) { - struct smc_connection *conn = - smcd->conn[rmb_node->sba_idx]; + conn = smcd->conn[rmb_node->sba_idx]; + if (conn && !conn->killed) + smcd_cdc_rx_handler(conn); - if (conn && !conn->killed) - smcd_cdc_rx_handler(conn); - } return 0; } @@ -254,6 +331,8 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd) .query_remote_gid = smc_lo_query_rgid, .register_dmb = smc_lo_register_dmb, .unregister_dmb = smc_lo_unregister_dmb, + .attach_dmb = smc_lo_attach_dmb, + .detach_dmb = smc_lo_detach_dmb, .add_vlan_id = smc_lo_add_vlan_id, .del_vlan_id = smc_lo_del_vlan_id, .set_vlan_required = smc_lo_set_vlan_required, @@ -265,6 +344,7 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd) .get_local_gid = smc_lo_get_local_gid, .get_chid = smc_lo_get_chid, .get_dev = smc_lo_get_dev, + .get_dev_dmb_attr = smc_lo_get_dev_dmb_attr, }; static struct smcd_dev *smcd_lo_alloc_dev(const struct smcd_ops *ops, @@ -345,6 +425,9 @@ static int smc_lo_dev_init(struct smc_lo_dev *ldev) smc_lo_generate_id(ldev); rwlock_init(&ldev->dmb_ht_lock); hash_init(ldev->dmb_ht); + atomic_set(&ldev->dmb_cnt, 0); + init_waitqueue_head(&ldev->dmbs_release); + init_waitqueue_head(&ldev->ldev_release); return smcd_lo_register_dev(ldev); } @@ -383,6 +466,8 @@ static int smc_lo_dev_probe(void) static void smc_lo_dev_exit(struct smc_lo_dev *ldev) { smcd_lo_unregister_dev(ldev); + if (atomic_read(&ldev->dmb_cnt)) + wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt)); } static void smc_lo_dev_remove(void) diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h index 9d34aba..e0bf044 100644 --- a/net/smc/smc_loopback.h +++ b/net/smc/smc_loopback.h @@ -33,6 +33,8 @@ struct smc_lo_dmb_node { u32 sba_idx; void *cpu_addr; dma_addr_t dma_addr; + refcount_t refcnt; + u8 freeing : 1; }; struct smc_lo_dev { @@ -43,6 +45,9 @@ struct smc_lo_dev { DECLARE_BITMAP(sba_idx_mask, SMC_LODEV_MAX_DMBS); rwlock_t dmb_ht_lock; DECLARE_HASHTABLE(dmb_ht, SMC_LODEV_MAX_DMBS_BUCKETS); + atomic_t dmb_cnt; + wait_queue_head_t dmbs_release; + wait_queue_head_t ldev_release; }; int smc_loopback_init(void);