From patchwork Sun Dec 10 13:24:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176334 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514227vqy; Sun, 10 Dec 2023 05:24:45 -0800 (PST) X-Google-Smtp-Source: AGHT+IGTKY8TfpVabT5ZzU78twXRQrf0bhLaj+SkrmCaP0HJRnp1f88THLupkY9s8aLjJoB6KYjv X-Received: by 2002:a17:902:dacc:b0:1d0:7249:a900 with SMTP id q12-20020a170902dacc00b001d07249a900mr1254671plx.103.1702214684805; Sun, 10 Dec 2023 05:24:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214684; cv=none; d=google.com; s=arc-20160816; b=dhl/U2GPyPXbJkgjobOzgbeiYUqzguXB93QuXAAWb4cV1SlDfBJ+80CidxipreltQf k6NieVPRk1WorcR7iN3vUNTVN0XYzDOxjRO99r4BNt8p5NkCm0MtA6TrmL4THaQkNtqs EQd1p0pAo5JSAIpc0+KmAYEJlVfjIPQVmmVrX/gEXfP8aTpBm7A+6+iOpmsKHw3oMtQ/ FvLAvVMCymXCSYARKKupXC9sFBC7MSycM9JA9ClfC3D0ULgSO0zM+T7T5kiyZGNJWnNw P/vdQnr0b51+afaBvP1/oA3pUh//oi9HUziw1Jb0vyN2+RkUFsQoq3+fbWMjpSWRdKon CUOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=T44Kn6Le3Jbl67f0hY61XNCEKWyk+hkXXNuBn0L+6pc=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=SVgiTXPkUJaUsPQY1EVGhsE/rU75AMVMlxFpgUPKb0WnY+DTKN7wH3sOlhoW5gV4Rv sjARNgAhGVIGl4Y68ML3N52dj/YhuWpCxHeUAYzOw7Zk5zPLplw4wJTQ5zDeJeUq7FAJ wOd05v71JuD9eJyN1fkTHA+2abPl7mjuqEEQ+KyWRrg643OZGm2wOSILAErLrKeAOgyU S8xbVueLOs9ffdEODHIIuoqIRJDryXFWFg98SxFIPfoSC+EmPSKQct6mT7+HoqBoj0YW ySdIeHAAfA9SCj8W9pMlrNO31RIAkv8GhobZN/gE8OEvGqksKK2scdqhcVd45Ft1i1GA Stnw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id p18-20020a170902ead200b001cc2ef72ab2si4588900pld.68.2023.12.10.05.24.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:24:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7F39A8066675; Sun, 10 Dec 2023 05:24:43 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232403AbjLJNYR (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232380AbjLJNYQ (ORCPT ); Sun, 10 Dec 2023 08:24:16 -0500 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D71910D; Sun, 10 Dec 2023 05:24:21 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R601e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8H5GJ_1702214657; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8H5GJ_1702214657) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:19 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 01/13] net/smc: improve SMC-D device dump for virtual ISM Date: Sun, 10 Dec 2023 21:24:02 +0800 Message-Id: <1702214654-32069-2-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:24:43 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901465660543244 X-GMAIL-MSGID: 1784901465660543244 The introduction of virtual ISM requires improvement of SMC-D device dump. Software implemented non-PCI device (loopback-ism) should be handled correctly and the CHID reserved for virtual ISM should be got from smcd_ops interface instead of PCI information. Signed-off-by: Wen Gu --- net/smc/smc_ism.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index ac88de2..66bcfdd 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -252,12 +252,11 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, char smc_pnet[SMC_MAX_PNETID_LEN + 1]; struct smc_pci_dev smc_pci_dev; struct nlattr *port_attrs; + struct device *device; struct nlattr *attrs; - struct ism_dev *ism; int use_cnt = 0; void *nlh; - ism = smcd->priv; nlh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, cb->nlh->nlmsg_seq, &smc_gen_nl_family, NLM_F_MULTI, SMC_NETLINK_GET_DEV_SMCD); @@ -272,7 +271,15 @@ static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, if (nla_put_u8(skb, SMC_NLA_DEV_IS_CRIT, use_cnt > 0)) goto errattr; memset(&smc_pci_dev, 0, sizeof(smc_pci_dev)); - smc_set_pci_values(to_pci_dev(ism->dev.parent), &smc_pci_dev); + device = smcd->ops->get_dev(smcd); + if (device->parent) + smc_set_pci_values(to_pci_dev(device->parent), &smc_pci_dev); + if (smc_ism_is_virtual(smcd)) { + smc_pci_dev.pci_pchid = smc_ism_get_chid(smcd); + if (!device->parent) + snprintf(smc_pci_dev.pci_id, sizeof(smc_pci_dev.pci_id), + "%s", dev_name(device)); + } if (nla_put_u32(skb, SMC_NLA_DEV_PCI_FID, smc_pci_dev.pci_fid)) goto errattr; if (nla_put_u16(skb, SMC_NLA_DEV_PCI_CHID, smc_pci_dev.pci_pchid)) From patchwork Sun Dec 10 13:24:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176335 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514261vqy; Sun, 10 Dec 2023 05:24:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IFg6rOVM638Wmg9cf5PrHIotZWlRmdukQmnKabw61wv1sTOUfOubxF2oivWa89GsPyv4jDu X-Received: by 2002:a05:6359:6302:b0:16b:a9f5:cde0 with SMTP id sf2-20020a056359630200b0016ba9f5cde0mr2712025rwb.29.1702214689214; Sun, 10 Dec 2023 05:24:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214689; cv=none; d=google.com; s=arc-20160816; b=ZP3Id6I8MlnA0cA0/8Y/LnfyFg5k8jlXmhAFqs40rgnqXlgs6U6IMik5qswOZ8pXPh a/TV61S8dfOFvrpWXH4eQvMcRGi3CLa0Dm4rgLZ1kvjAyB+fd6VJ/Czzx54c6VOPRQUh S6e3PzZ1E83vG1FfzZtIZl4WOUa72sRdqqY02bO+hTgbBCFr6DieIV4c5nntrzKsdyjp fhSEw0wLrpj+hkf4GezemI2WobHOvGW3bLRaW+zq/qQs7vncdI9c1WzcyAqEAb3FylCx VfVg0HNtKPZSB8qX8vXuTiUF7murqyKj7mXkkgfcIL6wt+57NcqCbmqxgDcq1J4xrfIo Rnqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=9k+n8822s9b6Yyky5gcJhgfcGqkQJa0u0zOidGjAMbA=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=QJY2yLYtKGioHZIQL4+d+EZVfgYyNxtOgbuefL7EA9KuAh7xfZS4q3EUYKrR1WLLVz 11yqNtYPrVDHRVKB2J2YfOWcXz2yPwKzb/mxO/VzEhhCTeMUzrtI4JwM6AAxD9S2cdyR cdWGjnt2FQHaXmLdrw7D8jyObFuHvlT6+Pq+vh/dKPlGy8L1gVJ8KSeYgr5vCe2sb9EO f0+ChE95BR1l1oyfJBnnn2pnC24Y02trfULXyoTxRej4HzdOcxnUQdktAiB1E+vECbQ6 mPmacGMJz8Uzdl7uvXGppmaUv9S1Qy2l/IyQjUqd750vojroFMPF+aAMPk/5udQF81Dk W98A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id v19-20020a056a00149300b006cc23e358bcsi4532891pfu.1.2023.12.10.05.24.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:24:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 0328C8069A28; Sun, 10 Dec 2023 05:24:43 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232447AbjLJNYX (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232445AbjLJNYS (ORCPT ); Sun, 10 Dec 2023 08:24:18 -0500 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 264DF115; Sun, 10 Dec 2023 05:24:23 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R581e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8BEjY_1702214659; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8BEjY_1702214659) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:21 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 02/13] net/smc: decouple specialized struct from SMC-D DMB registration Date: Sun, 10 Dec 2023 21:24:03 +0800 Message-Id: <1702214654-32069-3-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:24:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901469728009488 X-GMAIL-MSGID: 1784901469728009488 The struct 'ism_client' is specialized for s390 platform firmware ISM. So replace it with 'void' to make SMCD DMB registration helper generic for both virtual ISM and existing ISM. Signed-off-by: Wen Gu --- drivers/s390/net/ism_drv.c | 2 +- include/net/smc.h | 4 ++-- net/smc/smc_ism.c | 7 ++----- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/drivers/s390/net/ism_drv.c b/drivers/s390/net/ism_drv.c index 2c8e964..9b2a529 100644 --- a/drivers/s390/net/ism_drv.c +++ b/drivers/s390/net/ism_drv.c @@ -726,7 +726,7 @@ static int smcd_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid, } static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, - struct ism_client *client) + void *client) { return ism_register_dmb(smcd->priv, (struct ism_dmb *)dmb, client); } diff --git a/include/net/smc.h b/include/net/smc.h index c9dcb30..6273c3a 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -50,7 +50,6 @@ struct smcd_dmb { #define ISM_ERROR 0xFFFF struct smcd_dev; -struct ism_client; struct smcd_gid { u64 gid; @@ -61,7 +60,7 @@ struct smcd_ops { int (*query_remote_gid)(struct smcd_dev *dev, struct smcd_gid *rgid, u32 vid_valid, u32 vid); int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb, - struct ism_client *client); + void *client); int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb); int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id); int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id); @@ -81,6 +80,7 @@ struct smcd_ops { struct smcd_dev { const struct smcd_ops *ops; void *priv; + void *client; struct list_head list; spinlock_t lock; struct smc_connection **conn; diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index 66bcfdd..fb1837d 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -222,7 +222,6 @@ int smc_ism_unregister_dmb(struct smcd_dev *smcd, struct smc_buf_desc *dmb_desc) int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, struct smc_buf_desc *dmb_desc) { -#if IS_ENABLED(CONFIG_ISM) struct smcd_dmb dmb; int rc; @@ -231,7 +230,7 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, dmb.sba_idx = dmb_desc->sba_idx; dmb.vlan_id = lgr->vlan_id; dmb.rgid = lgr->peer_gid.gid; - rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, &smc_ism_client); + rc = lgr->smcd->ops->register_dmb(lgr->smcd, &dmb, lgr->smcd->client); if (!rc) { dmb_desc->sba_idx = dmb.sba_idx; dmb_desc->token = dmb.dmb_tok; @@ -240,9 +239,6 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, dmb_desc->len = dmb.dmb_len; } return rc; -#else - return 0; -#endif } static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, @@ -453,6 +449,7 @@ static void smcd_register_dev(struct ism_dev *ism) if (!smcd) return; smcd->priv = ism; + smcd->client = &smc_ism_client; ism_set_priv(ism, &smc_ism_client, smcd); if (smc_pnetid_by_dev_port(&ism->pdev->dev, 0, smcd->pnetid)) smc_pnetid_by_table_smcd(smcd); From patchwork Sun Dec 10 13:24:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176336 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514268vqy; Sun, 10 Dec 2023 05:24:50 -0800 (PST) X-Google-Smtp-Source: AGHT+IFvQQIU++Ws6GLbB/GFPNpq5Qsd3x176X81afDbELksgBqIqC6c+fs6ROyUNFBph15R3kNG X-Received: by 2002:a17:90a:5e09:b0:280:cc73:4c79 with SMTP id w9-20020a17090a5e0900b00280cc734c79mr1084961pjf.7.1702214689794; Sun, 10 Dec 2023 05:24:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214689; cv=none; d=google.com; s=arc-20160816; b=lpBjyVbxMSIBk9OnaZY9wicCr64/VdfmokAOkmksb6uGGQHi02ci7nzWvKFu48F472 VQ8G/Qk5Q3DvDZPHstKQSML41BVwZ+IDqrwa/fcDdnR4rTG2eapBIs3Smz/PFFjuceQ2 wfdL2nWLy0qJPCm2C7w8dELHcyUnnlHcddz+iW2n7hTlmMIygQxzFpR3qPh1qwdLPrMc Wi8VXI6kp/+AyQS0uHawjGjdDJ3E5SU0JDj6riKbqbNJ7chXFf7pSsoNhhxle31kBgaB apawgaoLM0WHTEGXqSWUVkp47H8ptXB8iUDMjTTbIDjP5XOqdvrvVQJJoMnpZs5EbsCQ sDxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=miJUuQw6oZIRSCMVF0ongZUGyFudD3ykytNhrmVs40k=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=Ypqb6OSvm409HyRd6/dGrxBkelXWl/1k77QSXaKypq0AqgzXfG6MMPCWVj5FbEJqfD aXW7yBuaMQZ5eD3R8TLxR0HdW7in5y6rYvkDfJPysRGtpoXimBYFhMcukkmCaIJWp6H6 r64JrvLKPcQ0n3N1OJZxzmPf177/yG1aOr/UoUUThppntwujorGykhR6Nq56XTU2TspI 1wv+RhMSCuVCX9q+4NqJZDFA8nR5cFlHwK5hWEDwXFRRIj3RZWjQsz03ChTLKtqloWCe By7i2r077VUJmhufNHJYQMHnDZUUuOc3D3bSzf2n83hrKw87JrMPVTGZq9+oYsfFSg+Y 0Usw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id x3-20020a17090a6c0300b002887144daffsi4631710pjj.83.2023.12.10.05.24.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:24:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 96A778067B52; Sun, 10 Dec 2023 05:24:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232503AbjLJNYY (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232459AbjLJNYX (ORCPT ); Sun, 10 Dec 2023 08:24:23 -0500 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B4F1E10D; Sun, 10 Dec 2023 05:24:26 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8Bbmi_1702214662; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8Bbmi_1702214662) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:24 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 03/13] net/smc: introduce virtual ISM device loopback-ism Date: Sun, 10 Dec 2023 21:24:04 +0800 Message-Id: <1702214654-32069-4-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:24:44 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901470235590093 X-GMAIL-MSGID: 1784901470235590093 This introduces a kind of virtual ISM device loopback-ism for SMCDv2.1. loopback-ism is implemented by software and serves inter-process or inter-container SMC communication in the same OS instance. It is created during SMC module loading and destroyed upon unloading. The support for loopback-ism can be configured via CONFIG_SMC_LO. Signed-off-by: Wen Gu --- net/smc/Kconfig | 13 ++++ net/smc/Makefile | 2 +- net/smc/af_smc.c | 12 +++- net/smc/smc_loopback.c | 181 +++++++++++++++++++++++++++++++++++++++++++++++++ net/smc/smc_loopback.h | 33 +++++++++ 5 files changed, 239 insertions(+), 2 deletions(-) create mode 100644 net/smc/smc_loopback.c create mode 100644 net/smc/smc_loopback.h diff --git a/net/smc/Kconfig b/net/smc/Kconfig index 746be39..e191f78 100644 --- a/net/smc/Kconfig +++ b/net/smc/Kconfig @@ -20,3 +20,16 @@ config SMC_DIAG smcss. if unsure, say Y. + +config SMC_LO + bool "SMC_LO: virtual ISM loopback-ism for SMC" + depends on SMC + default n + help + SMC_LO provides a kind of virtual ISM device called loopback-ism + for SMCD to upgrade AF_INET TCP connections whose ends share the + same kernel. + loopback-ism is a software implemented device that does not depend + on a specific architecture or hardware. + + if unsure, say N. diff --git a/net/smc/Makefile b/net/smc/Makefile index 875efcd..a8c3711 100644 --- a/net/smc/Makefile +++ b/net/smc/Makefile @@ -4,5 +4,5 @@ obj-$(CONFIG_SMC) += smc.o obj-$(CONFIG_SMC_DIAG) += smc_diag.o smc-y := af_smc.o smc_pnet.o smc_ib.o smc_clc.o smc_core.o smc_wr.o smc_llc.o smc-y += smc_cdc.o smc_tx.o smc_rx.o smc_close.o smc_ism.o smc_netlink.o smc_stats.o -smc-y += smc_tracepoint.o +smc-y += smc_tracepoint.o smc_loopback.o smc-$(CONFIG_SYSCTL) += smc_sysctl.o diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 9cdc974..6c0e381 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -53,6 +53,7 @@ #include "smc_stats.h" #include "smc_tracepoint.h" #include "smc_sysctl.h" +#include "smc_loopback.h" static DEFINE_MUTEX(smc_server_lgr_pending); /* serialize link group * creation on server @@ -3574,15 +3575,23 @@ static int __init smc_init(void) goto out_sock; } + rc = smc_loopback_init(); + if (rc) { + pr_err("%s: smc_loopback_init fails with %d\n", __func__, rc); + goto out_ib; + } + rc = tcp_register_ulp(&smc_ulp_ops); if (rc) { pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc); - goto out_ib; + goto out_lo; } static_branch_enable(&tcp_have_smc); return 0; +out_lo: + smc_loopback_exit(); out_ib: smc_ib_unregister_client(); out_sock: @@ -3620,6 +3629,7 @@ static void __exit smc_exit(void) tcp_unregister_ulp(&smc_ulp_ops); sock_unregister(PF_SMC); smc_core_exit(); + smc_loopback_exit(); smc_ib_unregister_client(); smc_ism_exit(); destroy_workqueue(smc_close_wq); diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c new file mode 100644 index 0000000..149ccaa --- /dev/null +++ b/net/smc/smc_loopback.c @@ -0,0 +1,181 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Shared Memory Communications Direct over loopback-ism device. + * + * Provide a SMC-D loopback-ism device. + * + * Copyright (c) 2023, Alibaba Inc. + * + * Author: Wen Gu + * Tony Lu + * + */ + +#include +#include +#include + +#include "smc_ism.h" +#include "smc_loopback.h" + +#if IS_ENABLED(CONFIG_SMC_LO) +static const char smc_lo_dev_name[] = "loopback-ism"; +static struct smc_lo_dev *lo_dev; +static struct class *smc_class; + +static const struct smcd_ops lo_ops = { + .query_remote_gid = NULL, + .register_dmb = NULL, + .unregister_dmb = NULL, + .add_vlan_id = NULL, + .del_vlan_id = NULL, + .set_vlan_required = NULL, + .reset_vlan_required = NULL, + .signal_event = NULL, + .move_data = NULL, + .supports_v2 = NULL, + .get_local_gid = NULL, + .get_chid = NULL, + .get_dev = NULL, +}; + +static struct smcd_dev *smcd_lo_alloc_dev(const struct smcd_ops *ops, + int max_dmbs) +{ + struct smcd_dev *smcd; + + smcd = kzalloc(sizeof(*smcd), GFP_KERNEL); + if (!smcd) + return NULL; + + smcd->conn = kcalloc(max_dmbs, sizeof(struct smc_connection *), + GFP_KERNEL); + if (!smcd->conn) + goto out_smcd; + + smcd->ops = ops; + + spin_lock_init(&smcd->lock); + spin_lock_init(&smcd->lgr_lock); + INIT_LIST_HEAD(&smcd->vlan); + INIT_LIST_HEAD(&smcd->lgr_list); + init_waitqueue_head(&smcd->lgrs_deleted); + return smcd; + +out_smcd: + kfree(smcd); + return NULL; +} + +static int smcd_lo_register_dev(struct smc_lo_dev *ldev) +{ + struct smcd_dev *smcd; + + smcd = smcd_lo_alloc_dev(&lo_ops, SMC_LO_MAX_DMBS); + if (!smcd) + return -ENOMEM; + ldev->smcd = smcd; + smcd->priv = ldev; + + /* TODO: + * register loopback-ism to smcd_dev list. + */ + return 0; +} + +static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev) +{ + struct smcd_dev *smcd = ldev->smcd; + + /* TODO: + * unregister loopback-ism from smcd_dev list. + */ + kfree(smcd->conn); + kfree(smcd); +} + +static int smc_lo_dev_init(struct smc_lo_dev *ldev) +{ + return smcd_lo_register_dev(ldev); +} + +static void smc_lo_dev_exit(struct smc_lo_dev *ldev) +{ + smcd_lo_unregister_dev(ldev); +} + +static void smc_lo_dev_release(struct device *dev) +{ + struct smc_lo_dev *ldev = + container_of(dev, struct smc_lo_dev, dev); + + kfree(ldev); +} + +static int smc_lo_dev_probe(void) +{ + struct smc_lo_dev *ldev; + int ret; + + smc_class = class_create("smc"); + if (IS_ERR(smc_class)) + return PTR_ERR(smc_class); + + ldev = kzalloc(sizeof(*ldev), GFP_KERNEL); + if (!ldev) { + ret = -ENOMEM; + goto destroy_class; + } + + ldev->dev.parent = NULL; + ldev->dev.class = smc_class; + ldev->dev.release = smc_lo_dev_release; + device_initialize(&ldev->dev); + dev_set_name(&ldev->dev, smc_lo_dev_name); + ret = device_add(&ldev->dev); + if (ret) + goto free_dev; + + ret = smc_lo_dev_init(ldev); + if (ret) + goto del_dev; + + lo_dev = ldev; /* global loopback device */ + return 0; + +del_dev: + device_del(&ldev->dev); +free_dev: + put_device(&ldev->dev); +destroy_class: + class_destroy(smc_class); + return ret; +} + +static void smc_lo_dev_remove(void) +{ + if (!lo_dev) + return; + + smc_lo_dev_exit(lo_dev); + device_del(&lo_dev->dev); /* device_add in smc_lo_dev_probe */ + put_device(&lo_dev->dev); /* device_initialize in smc_lo_dev_probe */ + class_destroy(smc_class); +} +#endif + +int smc_loopback_init(void) +{ +#if IS_ENABLED(CONFIG_SMC_LO) + return smc_lo_dev_probe(); +#else + return 0; +#endif +} + +void smc_loopback_exit(void) +{ +#if IS_ENABLED(CONFIG_SMC_LO) + smc_lo_dev_remove(); +#endif +} diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h new file mode 100644 index 0000000..a7c418b --- /dev/null +++ b/net/smc/smc_loopback.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Shared Memory Communications Direct over loopback-ism device. + * + * Provide a SMC-D loopback-ism device. + * + * Copyright (c) 2023, Alibaba Inc. + * + * Author: Wen Gu + * Tony Lu + * + */ + +#ifndef _SMC_LOOPBACK_H +#define _SMC_LOOPBACK_H + +#include +#include +#include + +#if IS_ENABLED(CONFIG_SMC_LO) +#define SMC_LO_MAX_DMBS 5000 + +struct smc_lo_dev { + struct smcd_dev *smcd; + struct device dev; +}; +#endif + +int smc_loopback_init(void); +void smc_loopback_exit(void); + +#endif /* _SMC_LOOPBACK_H */ From patchwork Sun Dec 10 13:24:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176337 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514274vqy; Sun, 10 Dec 2023 05:24:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IFZwBJj8Zu6Mx15nYmAFsw/xGIoHsedCM+QD7uMcQG2BP/heEoVAKUEegbqLC18wu+f7RO0 X-Received: by 2002:a17:90a:c24d:b0:286:ab89:837d with SMTP id d13-20020a17090ac24d00b00286ab89837dmr1073665pjx.48.1702214690698; Sun, 10 Dec 2023 05:24:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214690; cv=none; d=google.com; s=arc-20160816; b=MFzJVKcOIu5XRZ2XWRmlTIUUmyp6hNgLRgXrjD9Yinzk9KdgM2vRaOtnL7X9qs/3qY M0FoIDMe1anXJvEZtgQ3Kl30bHfcaPywyiP7ihySac4gKcUu1wrS5wZbG65micjTSDe6 hy9uZ+aaDzpHRGFlomKYvmRmwzgqxOjSa17UONXj9YWz6aOliDfPQi6vDTGxnQ9CKCbs h/82wjcyl/DOSaS4N1CQRCIFL7EtxBOITUs8vZQBkjjgqbvVGBlhIxVBfmg6RxkIRoLl kFVB3XmbPibxeoTj04oECd8jarhJz8ZwEIAjrlAb4O4ZgzYC5TEPe7hUDhlrwQBDlDi6 FzAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=z5fqgwh5DQKiSaMqkTuqfwmEEvtpyOUckOw5ZTWea+s=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=jNCPs1BhlGst4FX0Kelx7w1OTLohT2b15hj7Ejr1UKjMsPX1WNzkWuy0ipVp8saPIM FsULWn5R5/N2K0egDYr1tYQaUwMI0Iv1ejR8a1iy2aKP5WmzLDSTXJatB+Ekmnmd5uvK HKrk/LzLb+1Jd9u3k6TLeA6ExA+nxKEkM8oyJrzEVRMzpsfBUhU7TcspbkDWJf/vZDxP 68pzx24gf9LS2vbE0HR/pY6Drl3bHY45yGmSELAUDbJ2rv64ItMi01Pro7dJpGWFJ+db Bd6IzSdiKqT79UNN9nOjPAhf3mhbGvmxk/HZ9JgeFnfngiORMj0ERcyhpugns07wzbsY pjkg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id mn24-20020a17090b189800b00286b29bb241si5927562pjb.7.2023.12.10.05.24.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:24:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id D956C8069A16; Sun, 10 Dec 2023 05:24:46 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232429AbjLJNY0 (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232465AbjLJNYX (ORCPT ); Sun, 10 Dec 2023 08:24:23 -0500 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AB26118; Sun, 10 Dec 2023 05:24:28 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8Bbna_1702214664; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8Bbna_1702214664) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:26 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 04/13] net/smc: implement ID-related operations of loopback-ism Date: Sun, 10 Dec 2023 21:24:05 +0800 Message-Id: <1702214654-32069-5-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:24:47 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901471148110016 X-GMAIL-MSGID: 1784901471148110016 This implements GID and CHID related operations of loopback-ism device. loopback-ism acts as an ISMv2. It's GID is generated randomly by UUIDv4 algorithm and CHID is reserved 0xFFFF. Signed-off-by: Wen Gu --- net/smc/smc_loopback.c | 62 ++++++++++++++++++++++++++++++++++++++++++++++---- net/smc/smc_loopback.h | 3 +++ 2 files changed, 60 insertions(+), 5 deletions(-) diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 149ccaa..39e8c8d 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -19,12 +19,63 @@ #include "smc_loopback.h" #if IS_ENABLED(CONFIG_SMC_LO) +#define SMC_LO_V2_CAPABLE 0x1 /* loopback-ism acts as ISMv2 */ + static const char smc_lo_dev_name[] = "loopback-ism"; static struct smc_lo_dev *lo_dev; static struct class *smc_class; +static void smc_lo_generate_id(struct smc_lo_dev *ldev) +{ + struct smcd_gid *lgid = &ldev->local_gid; + uuid_t uuid; + + uuid_gen(&uuid); + memcpy(&lgid->gid, &uuid, sizeof(lgid->gid)); + memcpy(&lgid->gid_ext, (u8 *)&uuid + sizeof(lgid->gid), + sizeof(lgid->gid_ext)); + + ldev->chid = SMC_LO_CHID; +} + +static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid, + u32 vid_valid, u32 vid) +{ + struct smc_lo_dev *ldev = smcd->priv; + + /* rgid should equal to lgid in loopback situation */ + if (!ldev || rgid->gid != ldev->local_gid.gid || + rgid->gid_ext != ldev->local_gid.gid_ext) + return -ENETUNREACH; + return 0; +} + +static int smc_lo_supports_v2(void) +{ + return SMC_LO_V2_CAPABLE; +} + +static void smc_lo_get_local_gid(struct smcd_dev *smcd, + struct smcd_gid *smcd_gid) +{ + struct smc_lo_dev *ldev = smcd->priv; + + smcd_gid->gid = ldev->local_gid.gid; + smcd_gid->gid_ext = ldev->local_gid.gid_ext; +} + +static u16 smc_lo_get_chid(struct smcd_dev *smcd) +{ + return ((struct smc_lo_dev *)smcd->priv)->chid; +} + +static struct device *smc_lo_get_dev(struct smcd_dev *smcd) +{ + return &((struct smc_lo_dev *)smcd->priv)->dev; +} + static const struct smcd_ops lo_ops = { - .query_remote_gid = NULL, + .query_remote_gid = smc_lo_query_rgid, .register_dmb = NULL, .unregister_dmb = NULL, .add_vlan_id = NULL, @@ -33,10 +84,10 @@ .reset_vlan_required = NULL, .signal_event = NULL, .move_data = NULL, - .supports_v2 = NULL, - .get_local_gid = NULL, - .get_chid = NULL, - .get_dev = NULL, + .supports_v2 = smc_lo_supports_v2, + .get_local_gid = smc_lo_get_local_gid, + .get_chid = smc_lo_get_chid, + .get_dev = smc_lo_get_dev, }; static struct smcd_dev *smcd_lo_alloc_dev(const struct smcd_ops *ops, @@ -96,6 +147,7 @@ static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev) static int smc_lo_dev_init(struct smc_lo_dev *ldev) { + smc_lo_generate_id(ldev); return smcd_lo_register_dev(ldev); } diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h index a7c418b..73eed6a 100644 --- a/net/smc/smc_loopback.h +++ b/net/smc/smc_loopback.h @@ -20,10 +20,13 @@ #if IS_ENABLED(CONFIG_SMC_LO) #define SMC_LO_MAX_DMBS 5000 +#define SMC_LO_CHID 0xFFFF struct smc_lo_dev { struct smcd_dev *smcd; struct device dev; + u16 chid; + struct smcd_gid local_gid; }; #endif From patchwork Sun Dec 10 13:24:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176340 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514422vqy; Sun, 10 Dec 2023 05:25:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IEeVMnmb4tafAOCNTFpw73dio5bi0VdLyjTZEL4mNEjrnwhqdrDySyxLLczAra0rnkShryT X-Received: by 2002:a05:6e02:ef3:b0:35d:7ce5:3a55 with SMTP id j19-20020a056e020ef300b0035d7ce53a55mr3653585ilk.121.1702214710695; Sun, 10 Dec 2023 05:25:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214710; cv=none; d=google.com; s=arc-20160816; b=PavarV40As6ZEo+3s1H7yZSqxM4y7oIhAYEmggw2AU6F+mnU/97BZ+92mwNTMU6q// fKEQrE7gg5mmR8yfBqUCChE9Jm17LglI2DslSUuc1X2vti03Hn8iJv32F0Ig0NVMpkcD 2Wp33qS+SqHAI2kWx8MioMYvJ+pxj8skBX/TMu3lhcLY0kasJEYYhuTPovE2szYnpZFh iTBJlu/kyrsMb5YiRvP+hWVyJ1j/tICUpgZyu95bX0qj7Aw8bkbagy99X3UkXBflnM5n BzOzDqeAeIUb1PGU3CVp93LW0ZrNCwlJekFZbED9B0w5qYlux1MHyNPTM9vTwd8aSMDb esHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=eAhTLEyxtGLN/eUKyibMIk/ctE8yITan3peWN2XYho4=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=iB/ZDFmy6ZAhzNmrp20DHJ5UyA8y+jJnpJllz6HHBNPMSc1dKuoGgKHcElTIeoQgjV 47KLkM3BLFtPltVvPhwY66aJPo85gd+kqyvQY1W8SQ/f4v2g5C8DKCxHAzPA/pn2CjNB TvXqgJs57YCV0pE3SgXHwzEXgNyXQVqvh1O/yyQYZu1WAfbwO7M0nIHg465VOUEGG7k+ AogC/nvFbLroMbVPxxRMtFO+DmJahJfR47nJFOVQt4i2PPJ3b96rG9BKY792N71XthRi A3Qu8MDfAi2oeg44aDetyKaOVzSzwkDnTgCtF4VMGjK41sjVmpkFY57iREGq1g/JgP+0 S0bg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id bk5-20020a056a02028500b005c5e26ed226si4861284pgb.429.2023.12.10.05.25.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:25:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id B8E248069A10; Sun, 10 Dec 2023 05:25:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232579AbjLJNYc (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232544AbjLJNYY (ORCPT ); Sun, 10 Dec 2023 08:24:24 -0500 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E8FD10C; Sun, 10 Dec 2023 05:24:30 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R841e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8BEmd_1702214666; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8BEmd_1702214666) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:28 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 05/13] net/smc: implement some unsupported operations of loopback-ism Date: Sun, 10 Dec 2023 21:24:06 +0800 Message-Id: <1702214654-32069-6-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:25:08 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901492324419896 X-GMAIL-MSGID: 1784901492324419896 vlan operations are not supported currently since the need for vlan in loopback situation does not seem to be strong. signal_event operation is not supported since no event now needs to be processed by loopback-ism device. Signed-off-by: Wen Gu --- net/smc/smc_loopback.c | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 39e8c8d..349a969 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -50,6 +50,32 @@ static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid, return 0; } +static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_del_vlan_id(struct smcd_dev *smcd, u64 vlan_id) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_set_vlan_required(struct smcd_dev *smcd) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_reset_vlan_required(struct smcd_dev *smcd) +{ + return -EOPNOTSUPP; +} + +static int smc_lo_signal_event(struct smcd_dev *dev, struct smcd_gid *rgid, + u32 trigger_irq, u32 event_code, u64 info) +{ + return 0; +} + static int smc_lo_supports_v2(void) { return SMC_LO_V2_CAPABLE; @@ -78,11 +104,11 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd) .query_remote_gid = smc_lo_query_rgid, .register_dmb = NULL, .unregister_dmb = NULL, - .add_vlan_id = NULL, - .del_vlan_id = NULL, - .set_vlan_required = NULL, - .reset_vlan_required = NULL, - .signal_event = NULL, + .add_vlan_id = smc_lo_add_vlan_id, + .del_vlan_id = smc_lo_del_vlan_id, + .set_vlan_required = smc_lo_set_vlan_required, + .reset_vlan_required = smc_lo_reset_vlan_required, + .signal_event = smc_lo_signal_event, .move_data = NULL, .supports_v2 = smc_lo_supports_v2, .get_local_gid = smc_lo_get_local_gid, From patchwork Sun Dec 10 13:24:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176338 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514279vqy; Sun, 10 Dec 2023 05:24:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IEJn2MDLGGyOldQ/VngVQ86YbiV0OD6451XSmnQGRbIbS/9WCmHuWjD5fXMzXTM5nIMoMSm X-Received: by 2002:a05:620a:22a6:b0:77e:fba3:81e0 with SMTP id p6-20020a05620a22a600b0077efba381e0mr3408858qkh.118.1702214691676; Sun, 10 Dec 2023 05:24:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214691; cv=none; d=google.com; s=arc-20160816; b=RR1xUxvaHM4tHqmyrRQDjFie1S9rPEOexXAfavvyYYruBIr3NDWHNGtDOgrtybMzsf UZq3TQSplEPbEDm01xsDLjVgt0hQgz29zXZ4qBXl0T9MEKeIgRyYYxev0fxvABY7bhhz GNZKFLllm6LG911P8WuZX4MWUqZyIl7iXGFwTL0J5DfuXaBk1yxxQvANLu37TxVybDKU WCUv5P0MWOeQQc0K7OXoVJzrx8nlQi79qjrgSiCHG/775qD10aAr+9AWRcnATE+8cTe/ h7D0d2O+reiczKHvyC0Z3in9iEs1CT7oajQLR3ua8DjBCJtGNvebsOz33h74tPl9wCkP sbuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=gsnnGI5s7jCX53dgg2MArU5L1bg14+AZ6y9UggnqB9o=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=KEp4t8WkokSDxhIU8hxcaqjToAMBGxf2fe6+koDQoEerbLOIPD9j6WNGQjzxT4cduE /J7VTyT3fCUHO43VXMkt/cYCRSD7LPpXlDu5kGXFE1I3UyaRIAudzJ1LKP0oS+pje8jC i/zXkrE0232c8JO+v54XEBRctFM7jZrknruyJbwc58tWgJotlPWozpAyHkl2QqM/K+Sk Erbz704DHBSmCZLzOsT0SXY7pwU/39b+Eti2lrFw0A91UWC3SD3DgW52ssK7ruUMzw5X FiySM41x52+KxASLA06gvZW3D5nYrprtJ0WkE5qmfle7p4YsIlJaPJ+3nsxOgIcOHqNE FasA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id fz4-20020a17090b024400b002864f3ce7a4si4680491pjb.39.2023.12.10.05.24.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:24:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 8654980657C0; Sun, 10 Dec 2023 05:24:50 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232558AbjLJNYf (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232594AbjLJNYb (ORCPT ); Sun, 10 Dec 2023 08:24:31 -0500 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39496138; Sun, 10 Dec 2023 05:24:33 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R411e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8BEnK_1702214668; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8BEnK_1702214668) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:30 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 06/13] net/smc: implement DMB-related operations of loopback-ism Date: Sun, 10 Dec 2023 21:24:07 +0800 Message-Id: <1702214654-32069-7-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:24:50 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901472508113684 X-GMAIL-MSGID: 1784901472508113684 This implements DMB (un)registration and data move operations of loopback-ism device. Signed-off-by: Wen Gu --- net/smc/smc_cdc.c | 6 +++ net/smc/smc_cdc.h | 1 + net/smc/smc_loopback.c | 128 +++++++++++++++++++++++++++++++++++++++++++++++-- net/smc/smc_loopback.h | 13 +++++ 4 files changed, 145 insertions(+), 3 deletions(-) diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index 3c06625..c820ef1 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -410,6 +410,12 @@ static void smc_cdc_msg_recv(struct smc_sock *smc, struct smc_cdc_msg *cdc) static void smcd_cdc_rx_tsklet(struct tasklet_struct *t) { struct smc_connection *conn = from_tasklet(conn, t, rx_tsklet); + + smcd_cdc_rx_handler(conn); +} + +void smcd_cdc_rx_handler(struct smc_connection *conn) +{ struct smcd_cdc_msg *data_cdc; struct smcd_cdc_msg cdc; struct smc_sock *smc; diff --git a/net/smc/smc_cdc.h b/net/smc/smc_cdc.h index 696cc11..11559d4 100644 --- a/net/smc/smc_cdc.h +++ b/net/smc/smc_cdc.h @@ -301,5 +301,6 @@ int smcr_cdc_msg_send_validation(struct smc_connection *conn, struct smc_wr_buf *wr_buf); int smc_cdc_init(void) __init; void smcd_cdc_rx_init(struct smc_connection *conn); +void smcd_cdc_rx_handler(struct smc_connection *conn); #endif /* SMC_CDC_H */ diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 349a969..ee4f112 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -13,13 +13,16 @@ #include #include +#include #include +#include "smc_cdc.h" #include "smc_ism.h" #include "smc_loopback.h" #if IS_ENABLED(CONFIG_SMC_LO) #define SMC_LO_V2_CAPABLE 0x1 /* loopback-ism acts as ISMv2 */ +#define SMC_DMA_ADDR_INVALID (~(dma_addr_t)0) static const char smc_lo_dev_name[] = "loopback-ism"; static struct smc_lo_dev *lo_dev; @@ -50,6 +53,91 @@ static int smc_lo_query_rgid(struct smcd_dev *smcd, struct smcd_gid *rgid, return 0; } +static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, + void *client_priv) +{ + struct smc_lo_dmb_node *dmb_node, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + int sba_idx, rc; + + /* check space for new dmb */ + for_each_clear_bit(sba_idx, ldev->sba_idx_mask, SMC_LO_MAX_DMBS) { + if (!test_and_set_bit(sba_idx, ldev->sba_idx_mask)) + break; + } + if (sba_idx == SMC_LO_MAX_DMBS) + return -ENOSPC; + + dmb_node = kzalloc(sizeof(*dmb_node), GFP_KERNEL); + if (!dmb_node) { + rc = -ENOMEM; + goto err_bit; + } + + dmb_node->sba_idx = sba_idx; + dmb_node->cpu_addr = vzalloc(dmb->dmb_len); + if (!dmb_node->cpu_addr) { + rc = -ENOMEM; + goto err_node; + } + dmb_node->len = dmb->dmb_len; + dmb_node->dma_addr = SMC_DMA_ADDR_INVALID; + +again: + /* add new dmb into hash table */ + get_random_bytes(&dmb_node->token, sizeof(dmb_node->token)); + write_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_node->token) { + if (tmp_node->token == dmb_node->token) { + write_unlock(&ldev->dmb_ht_lock); + goto again; + } + } + hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token); + write_unlock(&ldev->dmb_ht_lock); + + dmb->sba_idx = dmb_node->sba_idx; + dmb->dmb_tok = dmb_node->token; + dmb->cpu_addr = dmb_node->cpu_addr; + dmb->dma_addr = dmb_node->dma_addr; + dmb->dmb_len = dmb_node->len; + + return 0; + +err_node: + kfree(dmb_node); +err_bit: + clear_bit(sba_idx, ldev->sba_idx_mask); + return rc; +} + +static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) +{ + struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + /* remove dmb from hash table */ + write_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) { + if (tmp_node->token == dmb->dmb_tok) { + dmb_node = tmp_node; + break; + } + } + if (!dmb_node) { + write_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + hash_del(&dmb_node->list); + write_unlock(&ldev->dmb_ht_lock); + + clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask); + vfree(dmb_node->cpu_addr); + kfree(dmb_node); + + return 0; +} + static int smc_lo_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id) { return -EOPNOTSUPP; @@ -76,6 +164,38 @@ static int smc_lo_signal_event(struct smcd_dev *dev, struct smcd_gid *rgid, return 0; } +static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, + unsigned int idx, bool sf, unsigned int offset, + void *data, unsigned int size) +{ + struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + read_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) { + if (tmp_node->token == dmb_tok) { + rmb_node = tmp_node; + break; + } + } + if (!rmb_node) { + read_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + read_unlock(&ldev->dmb_ht_lock); + + memcpy((char *)rmb_node->cpu_addr + offset, data, size); + + if (sf) { + struct smc_connection *conn = + smcd->conn[rmb_node->sba_idx]; + + if (conn && !conn->killed) + smcd_cdc_rx_handler(conn); + } + return 0; +} + static int smc_lo_supports_v2(void) { return SMC_LO_V2_CAPABLE; @@ -102,14 +222,14 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd) static const struct smcd_ops lo_ops = { .query_remote_gid = smc_lo_query_rgid, - .register_dmb = NULL, - .unregister_dmb = NULL, + .register_dmb = smc_lo_register_dmb, + .unregister_dmb = smc_lo_unregister_dmb, .add_vlan_id = smc_lo_add_vlan_id, .del_vlan_id = smc_lo_del_vlan_id, .set_vlan_required = smc_lo_set_vlan_required, .reset_vlan_required = smc_lo_reset_vlan_required, .signal_event = smc_lo_signal_event, - .move_data = NULL, + .move_data = smc_lo_move_data, .supports_v2 = smc_lo_supports_v2, .get_local_gid = smc_lo_get_local_gid, .get_chid = smc_lo_get_chid, @@ -174,6 +294,8 @@ static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev) static int smc_lo_dev_init(struct smc_lo_dev *ldev) { smc_lo_generate_id(ldev); + rwlock_init(&ldev->dmb_ht_lock); + hash_init(ldev->dmb_ht); return smcd_lo_register_dev(ldev); } diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h index 73eed6a..3d5d2f5 100644 --- a/net/smc/smc_loopback.h +++ b/net/smc/smc_loopback.h @@ -20,13 +20,26 @@ #if IS_ENABLED(CONFIG_SMC_LO) #define SMC_LO_MAX_DMBS 5000 +#define SMC_LO_DMBS_HASH_BITS 12 #define SMC_LO_CHID 0xFFFF +struct smc_lo_dmb_node { + struct hlist_node list; + u64 token; + u32 len; + u32 sba_idx; + void *cpu_addr; + dma_addr_t dma_addr; +}; + struct smc_lo_dev { struct smcd_dev *smcd; struct device dev; u16 chid; struct smcd_gid local_gid; + rwlock_t dmb_ht_lock; + DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS); + DECLARE_HASHTABLE(dmb_ht, SMC_LO_DMBS_HASH_BITS); }; #endif From patchwork Sun Dec 10 13:24:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176341 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514435vqy; Sun, 10 Dec 2023 05:25:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IHhN2HQ1+VMwRkxMC6oLp8oiP0dToCNvbQyxOYlrpojrm9hM+S/mT9vzsL5tBSWQnL5aimJ X-Received: by 2002:a05:6a20:6881:b0:18f:9c4:d34d with SMTP id n1-20020a056a20688100b0018f09c4d34dmr3730473pzj.61.1702214713171; Sun, 10 Dec 2023 05:25:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214713; cv=none; d=google.com; s=arc-20160816; b=G0SU/H9zoZKYSqFjSWyvQtyjigTDzVDV3r2VRdcvRrB3vPKKrRKFW3qrS3r6/kiipd uhFCqoFodncbWu3yzhBPpR1P3AdpVXjMODcRc7LPyERb2YqpxVeIJBSgUD1Ltvm8aUxz 76nl4EnaydFTF1Dk0yVPfXt+iP+y+PA7LtMzeA129FB1+DLwv76qWv27iZGBEFBtAfHE WK0oB/Flm+LI0lTRU32Z0DUR57wnI1bnEyWd9dMKrqI+fAdsHdZ5rEFiwJT+ig1U4Jkp //RgUy1wzXL6C+AKrcu9PeQ3USbNUupOicwLSkCwl5P5fATBLnD/M4QE+kZ2314qRQZ1 Yt3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=Rh+sIbinDTJFJ19uGZr6+KMerHgYDN3D0dVTpzkp6fQ=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=DTCdm3Th8R47Jf4AucxoOud8DkfoZATWAUOZtLO8wcJz841KG4fOeYDplgmKGuPxtl gjsoQdAesCq04NoSJ30YXRA+sJTq/xMd80QFmTt7HjKoe7NWrpf0rsnHBQtBLpqSqVXc WfvgobwrpUFX0pJZv9ABr9g2zdbqWlBOYyx/ciSwNZNN1wXuUAEXfqA6qYh5o+ZMGf93 IYLyxyRM5JQU30ICnpSA3DptLZpqW7McyfO7zRoNQeZmlbUkNPFTlAaJJ94BYVfBfYCr JpB+B39ScXoMxQPwsTSWGeP2YlQlk7TjpToIxExhIPeb3pvQvxJrkhJofSyVv+MqHOlj BiuQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id m5-20020a632605000000b005c688ff3468si4714793pgm.579.2023.12.10.05.25.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:25:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 21682806B9F7; Sun, 10 Dec 2023 05:25:10 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232848AbjLJNYv (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232593AbjLJNYf (ORCPT ); Sun, 10 Dec 2023 08:24:35 -0500 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1FBE10E7; Sun, 10 Dec 2023 05:24:37 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R791e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8Bbrt_1702214673; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8Bbrt_1702214673) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:35 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 08/13] net/smc: introduce loopback-ism runtime switch Date: Sun, 10 Dec 2023 21:24:09 +0800 Message-Id: <1702214654-32069-9-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:25:10 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901494951064439 X-GMAIL-MSGID: 1784901494951064439 This provides a runtime switch to activate or deactivate loopback-ism device by echo {1|0} > /sys/devices/virtual/smc/loopback-ism/active. It will trigger the registration or removal of loopback-ism from the SMC-D device list. Signed-off-by: Wen Gu --- net/smc/smc_loopback.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++ net/smc/smc_loopback.h | 1 + 2 files changed, 56 insertions(+) diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 83bc9a7..04f8612 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -28,6 +28,58 @@ static struct smc_lo_dev *lo_dev; static struct class *smc_class; +static int smcd_lo_register_dev(struct smc_lo_dev *ldev); +static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev); + +static ssize_t active_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct smc_lo_dev *ldev = + container_of(dev, struct smc_lo_dev, dev); + + return sysfs_emit(buf, "%d\n", ldev->active); +} + +static ssize_t active_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct smc_lo_dev *ldev = + container_of(dev, struct smc_lo_dev, dev); + bool active; + int ret; + + ret = kstrtobool(buf, &active); + if (ret) + return ret; + + if (active && !ldev->active) { + /* activate loopback-ism */ + ret = smcd_lo_register_dev(ldev); + if (ret) + return ret; + } else if (!active && ldev->active) { + /* deactivate loopback-ism */ + smcd_lo_unregister_dev(ldev); + } + + return count; +} +static DEVICE_ATTR_RW(active); +static struct attribute *smc_lo_attrs[] = { + &dev_attr_active.attr, + NULL +}; + +static struct attribute_group smc_lo_attr_group = { + .attrs = smc_lo_attrs, +}; + +static const struct attribute_group *smc_lo_attr_groups[] = { + &smc_lo_attr_group, + NULL, +}; + static void smc_lo_generate_id(struct smc_lo_dev *ldev) { struct smcd_gid *lgid = &ldev->local_gid; @@ -277,6 +329,7 @@ static int smcd_lo_register_dev(struct smc_lo_dev *ldev) mutex_lock(&smcd_dev_list.mutex); list_add(&smcd->list, &smcd_dev_list.list); mutex_unlock(&smcd_dev_list.mutex); + ldev->active = 1; pr_warn_ratelimited("smc: adding smcd device %s\n", smc_lo_dev_name); return 0; @@ -288,6 +341,7 @@ static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev) pr_warn_ratelimited("smc: removing smcd device %s\n", smc_lo_dev_name); + ldev->active = 0; smcd->going_away = 1; smc_smcd_terminate_all(smcd); mutex_lock(&smcd_dev_list.mutex); @@ -335,6 +389,7 @@ static int smc_lo_dev_probe(void) ldev->dev.parent = NULL; ldev->dev.class = smc_class; + ldev->dev.groups = smc_lo_attr_groups; ldev->dev.release = smc_lo_dev_release; device_initialize(&ldev->dev); dev_set_name(&ldev->dev, smc_lo_dev_name); diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h index 3d5d2f5..0cc2f83 100644 --- a/net/smc/smc_loopback.h +++ b/net/smc/smc_loopback.h @@ -35,6 +35,7 @@ struct smc_lo_dmb_node { struct smc_lo_dev { struct smcd_dev *smcd; struct device dev; + u8 active; u16 chid; struct smcd_gid local_gid; rwlock_t dmb_ht_lock; From patchwork Sun Dec 10 13:24:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176339 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514343vqy; Sun, 10 Dec 2023 05:25:00 -0800 (PST) X-Google-Smtp-Source: AGHT+IHx39A1YpR8j33nU27C0TBYZ2Ns546izAah+uLs0hDkQm0rm8VKDVyxse4Sd197dZ5dG6bH X-Received: by 2002:a05:6e02:12e4:b0:35d:58ce:2322 with SMTP id l4-20020a056e0212e400b0035d58ce2322mr5692092iln.5.1702214700421; Sun, 10 Dec 2023 05:25:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214700; cv=none; d=google.com; s=arc-20160816; b=Tl608bfZhK5P7/d3bfu4b0HwBt3qx4csKGldp8N7WH+t1M085An8bEel6WXqRl1BAW SRADPmcnkDes+uQG7dVVuV5gS3x1lqeiCQFf5BNht5o27WvaV4UWgHKuGhhfQ4ZNPr9N /Ldv7jeIZiPBk9gPdCZAsVXY8vLYKiNeFOvfUJ8IoD6DEeqZPMl9nVSS6QS9Uau+JLk1 Q2QOmxbon/Y2f/66Ig0q3XZO7i11nuXuot743z+UOk5k24OSb10Cwoz19wLApD4mQjhG EmMVvYCyjVkcr2+DIYkdkkAGYEDGM/x0IEjkM7rCOXvgU844ZirYGR2WeLOlPNILEFA1 T9vQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=cVDAV8hgGPP7oEyGRZEMPYQlYznkdjiXb1ENubks9cg=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=y2D1O3s2HkOaA9gPE69pauD/y8i6NZ2xYNhQl9zrUJSDhmw1sD7fCS7cKgj50XP1kP cV46h224f1HLzG2MVGNX59y5CvJT5JLFxCZteOLnWXXiEnL6SenMB08F4c3jcRyb5ywz E3pxTnkjln0cZ7CD+sWEOruWtB+bSoCwXbQ1ogQEjW8eqNwXJkDjVeRy5Kv0fl7KUiru d7PNDa6nBVysZkX2I1KsLJnMYxnLJriesCbrJQiuIoI4mm6DZHiEh/IM2jWQ39v0Mkg5 t7vqq6iCbG5ab8zSIWadbuedTXZoU0pufQgVfR9ehakw2o65/pNvidcngor36n2xy0pV vGWw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id f1-20020a17090274c100b001cfcc0ca762si4564502plt.108.2023.12.10.05.25.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:25:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 61C1A8095014; Sun, 10 Dec 2023 05:24:59 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232608AbjLJNYt (ORCPT + 99 others); Sun, 10 Dec 2023 08:24:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232537AbjLJNYe (ORCPT ); Sun, 10 Dec 2023 08:24:34 -0500 Received: from out30-98.freemail.mail.aliyun.com (out30-98.freemail.mail.aliyun.com [115.124.30.98]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FDEFFE; Sun, 10 Dec 2023 05:24:39 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R581e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8BErC_1702214675; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8BErC_1702214675) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:37 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 09/13] net/smc: introduce loopback-ism statistics attributes Date: Sun, 10 Dec 2023 21:24:10 +0800 Message-Id: <1702214654-32069-10-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:24:59 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901481371286429 X-GMAIL-MSGID: 1784901481371286429 This introduces some statistics attributes of loopback-ism. They can be read from /sys/devices/virtual/smc/loopback-ism/{{tx|rx}_tytes|dmbs_cnt}. Signed-off-by: Wen Gu --- net/smc/smc_loopback.c | 80 +++++++++++++++++++++++++++++++++++++++++++++++++- net/smc/smc_loopback.h | 26 ++++++++++++++++ 2 files changed, 105 insertions(+), 1 deletion(-) diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 04f8612..057ea6a 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -31,6 +31,67 @@ static int smcd_lo_register_dev(struct smc_lo_dev *ldev); static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev); +static void smc_lo_clear_stats(struct smc_lo_dev *ldev) +{ + struct smc_lo_dev_stats64 *tmp; + int cpu; + + for_each_possible_cpu(cpu) { + tmp = per_cpu_ptr(ldev->stats, cpu); + tmp->tx_bytes = 0; + tmp->rx_bytes = 0; + } +} + +static void smc_lo_get_stats(struct smc_lo_dev *ldev, + struct smc_lo_dev_stats64 *stats) +{ + int size, cpu, i; + u64 *src, *sum; + + memset(stats, 0, sizeof(*stats)); + size = sizeof(*stats) / sizeof(u64); + for_each_possible_cpu(cpu) { + src = (u64 *)per_cpu_ptr(ldev->stats, cpu); + sum = (u64 *)stats; + for (i = 0; i < size; i++) + *(sum++) += *(src++); + } +} + +static ssize_t smc_lo_show_stats(struct device *dev, + struct device_attribute *attr, + char *buf, unsigned long offset) +{ + struct smc_lo_dev *ldev = + container_of(dev, struct smc_lo_dev, dev); + struct smc_lo_dev_stats64 stats; + ssize_t ret = -EINVAL; + + if (WARN_ON(offset > sizeof(struct smc_lo_dev_stats64) || + offset % sizeof(u64) != 0)) + goto out; + + smc_lo_get_stats(ldev, &stats); + ret = sysfs_emit(buf, "%llu\n", *(u64 *)(((u8 *)&stats) + offset)); +out: + return ret; +} + +/* generate a read-only statistics attribute */ +#define SMC_LO_DEVICE_ATTR_RO(name) \ +static ssize_t name##_show(struct device *dev, \ + struct device_attribute *attr, char *buf) \ +{ \ + return smc_lo_show_stats(dev, attr, buf, \ + offsetof(struct smc_lo_dev_stats64, name)); \ +} \ +static DEVICE_ATTR_RO(name) + +SMC_LO_DEVICE_ATTR_RO(rx_bytes); +SMC_LO_DEVICE_ATTR_RO(tx_bytes); +SMC_LO_DEVICE_ATTR_RO(dmbs_cnt); + static ssize_t active_show(struct device *dev, struct device_attribute *attr, char *buf) { @@ -68,6 +129,9 @@ static ssize_t active_store(struct device *dev, static DEVICE_ATTR_RW(active); static struct attribute *smc_lo_attrs[] = { &dev_attr_active.attr, + &dev_attr_rx_bytes.attr, + &dev_attr_tx_bytes.attr, + &dev_attr_dmbs_cnt.attr, NULL }; @@ -147,6 +211,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, } hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token); write_unlock(&ldev->dmb_ht_lock); + SMC_LO_STAT_DMBS_INC(ldev); dmb->sba_idx = dmb_node->sba_idx; dmb->dmb_tok = dmb_node->token; @@ -186,6 +251,7 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask); vfree(dmb_node->cpu_addr); kfree(dmb_node); + SMC_LO_STAT_DMBS_DEC(ldev); return 0; } @@ -237,13 +303,16 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, read_unlock(&ldev->dmb_ht_lock); memcpy((char *)rmb_node->cpu_addr + offset, data, size); + SMC_LO_STAT_TX_BYTES(ldev, size); if (sf) { struct smc_connection *conn = smcd->conn[rmb_node->sba_idx]; - if (conn && !conn->killed) + if (conn && !conn->killed) { + SMC_LO_STAT_RX_BYTES(ldev, size); smcd_cdc_rx_handler(conn); + } } return 0; } @@ -349,6 +418,7 @@ static void smcd_lo_unregister_dev(struct smc_lo_dev *ldev) mutex_unlock(&smcd_dev_list.mutex); kfree(smcd->conn); kfree(smcd); + smc_lo_clear_stats(ldev); } static int smc_lo_dev_init(struct smc_lo_dev *ldev) @@ -369,6 +439,7 @@ static void smc_lo_dev_release(struct device *dev) struct smc_lo_dev *ldev = container_of(dev, struct smc_lo_dev, dev); + free_percpu(ldev->stats); kfree(ldev); } @@ -387,6 +458,13 @@ static int smc_lo_dev_probe(void) goto destroy_class; } + ldev->stats = alloc_percpu(struct smc_lo_dev_stats64); + if (!ldev->stats) { + ret = -ENOMEM; + kfree(ldev); + goto destroy_class; + } + ldev->dev.parent = NULL; ldev->dev.class = smc_class; ldev->dev.groups = smc_lo_attr_groups; diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h index 0cc2f83..ad0feaa 100644 --- a/net/smc/smc_loopback.h +++ b/net/smc/smc_loopback.h @@ -32,16 +32,42 @@ struct smc_lo_dmb_node { dma_addr_t dma_addr; }; +struct smc_lo_dev_stats64 { + __u64 rx_bytes; + __u64 tx_bytes; + __u64 dmbs_cnt; +}; + struct smc_lo_dev { struct smcd_dev *smcd; struct device dev; u8 active; u16 chid; struct smcd_gid local_gid; + struct smc_lo_dev_stats64 __percpu *stats; rwlock_t dmb_ht_lock; DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS); DECLARE_HASHTABLE(dmb_ht, SMC_LO_DMBS_HASH_BITS); }; + +#define SMC_LO_STAT_SUB(ldev, key, val) \ +do { \ + struct smc_lo_dev_stats64 *_stats = (ldev)->stats; \ + this_cpu_add((*(_stats)).key, val); \ +} \ +while (0) + +#define SMC_LO_STAT_RX_BYTES(ldev, val) \ + SMC_LO_STAT_SUB(ldev, rx_bytes, val) + +#define SMC_LO_STAT_TX_BYTES(ldev, val) \ + SMC_LO_STAT_SUB(ldev, tx_bytes, val) + +#define SMC_LO_STAT_DMBS_INC(ldev) \ + SMC_LO_STAT_SUB(ldev, dmbs_cnt, 1) + +#define SMC_LO_STAT_DMBS_DEC(ldev) \ + SMC_LO_STAT_SUB(ldev, dmbs_cnt, -1) #endif int smc_loopback_init(void); From patchwork Sun Dec 10 13:24:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176343 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514501vqy; Sun, 10 Dec 2023 05:25:25 -0800 (PST) X-Google-Smtp-Source: AGHT+IEvRZS+7iQQ/HthjvIbU5yKZVsD0Il1oMckbbVVoXyPeUNhipxvhOAKn+L6Nedx5HRkpFlW X-Received: by 2002:a05:6359:d06:b0:170:a3fb:dfb2 with SMTP id gp6-20020a0563590d0600b00170a3fbdfb2mr3365762rwb.37.1702214724976; Sun, 10 Dec 2023 05:25:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214724; cv=none; d=google.com; s=arc-20160816; b=eqgy3U5+lG1GjUKlGK9s/R1jyQ/X1r0cylhh+0c/0HtThZxvKstzCy/x8YcIEIea9n nNDZMXtK/tdoxQ8QbU4uAALL3VsopeiN3AXkDYXurXrt0wCxcdo3uSJFu/K38TkGHkbc ifGjwKXL+gJ9cU/NoAquZBbSYyENSeain7tHjcHjR4Wt59s91V2LRCKTSkm40NdWBSmE jo98upV9hJU5To9cd2tju7srvh9pYy2oeQ6Ta6WWKpmZHQr3scpkFrxHWOo1AII3AH8L 0sxxRUJNIiQr/V9EPWivWS5S3yI70EFVPYWn/va1iAiJYa2UZZT+290sxFJ9nqYBi8q7 Y0Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=2H/FV5lSKnhRTSPDOPA7eT5ypIGCDPAXJ2JWrzx5cgE=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=t7vu/zEJ/deyysddgAX7jgrfH92WJZJMRI0TDA23BdBIVhvjX4Mqa1xSOIDZrzrJA6 xWqrms3o7R+/IqgbEoI7Z1dRhld3X94oipDE8vMi3OhY0kZQHJbxSe2xaQ9jMTBuZHry NDfYLFVD64AQvMjMjSMv4H+GKEIZExV77Szn820+JClce8p8T6DtokjCfskSnMOYnLf5 GByK3BPT43KT/QYJ0I3OFqKzz96X3lfG1VYakbN7GKZn0t8fnVAdk56W2MrEaC03ok+z sBpTndJhEISJgb167LV3uNSIm13q6lJXRUI7v6JAGGKV7IzgS1C6myrrWju8kCVPY3lr VL2w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from agentk.vger.email (agentk.vger.email. [23.128.96.32]) by mx.google.com with ESMTPS id x1-20020a631701000000b005b887ff580asi4416982pgl.878.2023.12.10.05.25.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:25:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 041A4806290B; Sun, 10 Dec 2023 05:25:22 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232766AbjLJNZA (ORCPT + 99 others); Sun, 10 Dec 2023 08:25:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232479AbjLJNYp (ORCPT ); Sun, 10 Dec 2023 08:24:45 -0500 Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B56D171F; Sun, 10 Dec 2023 05:24:42 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R401e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8Bbu0_1702214677; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8Bbu0_1702214677) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:39 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 10/13] net/smc: introduce operations to {at|de}tach ghost sndbuf to peer DMB Date: Sun, 10 Dec 2023 21:24:11 +0800 Message-Id: <1702214654-32069-11-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:25:22 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901507402957110 X-GMAIL-MSGID: 1784901507402957110 In some scenarios using virtual ISM device, sndbuf can share the same physical memory region with peer DMB to avoid data copy from one side to the other. In such case the sndbuf is only a descriptor that describes the shared memory and does not actually occupy memory, it's more like a ghost buffer. +----------+ +----------+ | socket A | | socket B | +----------+ +----------+ | | +--------+ +--------+ | sndbuf | | DMB | | desc | | desc | +--------+ +--------+ | | | +----v-----+ +--------------------------> memory | +----------+ So here introduces two new SMC-D device operations for attaching or detaching ghost sndbuf to peer DMB. For now only intra-VM communication with loopback-ism device supports this. Signed-off-by: Wen Gu --- include/net/smc.h | 2 ++ net/smc/smc_ism.c | 38 ++++++++++++++++++++++++++++++++++++++ net/smc/smc_ism.h | 4 ++++ 3 files changed, 44 insertions(+) diff --git a/include/net/smc.h b/include/net/smc.h index 6273c3a..648c2f8 100644 --- a/include/net/smc.h +++ b/include/net/smc.h @@ -62,6 +62,8 @@ struct smcd_ops { int (*register_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb, void *client); int (*unregister_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb); + int (*attach_dmb)(struct smcd_dev *dev, struct smcd_dmb *dmb); + int (*detach_dmb)(struct smcd_dev *dev, u64 token); int (*add_vlan_id)(struct smcd_dev *dev, u64 vlan_id); int (*del_vlan_id)(struct smcd_dev *dev, u64 vlan_id); int (*set_vlan_required)(struct smcd_dev *dev); diff --git a/net/smc/smc_ism.c b/net/smc/smc_ism.c index 4065ebd..24a51ed 100644 --- a/net/smc/smc_ism.c +++ b/net/smc/smc_ism.c @@ -246,6 +246,44 @@ int smc_ism_register_dmb(struct smc_link_group *lgr, int dmb_len, return rc; } +bool smc_ism_dmb_mappable(struct smcd_dev *smcd) +{ + /* for now only loopback-ism supports mapping + * ghost sndbuf to peer DMB. + */ + return (smcd->ops->get_chid(smcd) == 0xFFFF); +} + +int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token, + struct smc_buf_desc *dmb_desc) +{ + struct smcd_dmb dmb; + int rc = 0; + + if (!dev->ops->attach_dmb) + return -EINVAL; + + memset(&dmb, 0, sizeof(dmb)); + dmb.dmb_tok = token; + rc = dev->ops->attach_dmb(dev, &dmb); + if (!rc) { + dmb_desc->sba_idx = dmb.sba_idx; + dmb_desc->token = dmb.dmb_tok; + dmb_desc->cpu_addr = dmb.cpu_addr; + dmb_desc->dma_addr = dmb.dma_addr; + dmb_desc->len = dmb.dmb_len; + } + return rc; +} + +int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token) +{ + if (!dev->ops->detach_dmb) + return -EINVAL; + + return dev->ops->detach_dmb(dev, token); +} + static int smc_nl_handle_smcd_dev(struct smcd_dev *smcd, struct sk_buff *skb, struct netlink_callback *cb) diff --git a/net/smc/smc_ism.h b/net/smc/smc_ism.h index 6903cd5..8ed1f76 100644 --- a/net/smc/smc_ism.h +++ b/net/smc/smc_ism.h @@ -48,6 +48,10 @@ int smc_ism_cantalk(struct smcd_gid *peer_gid, unsigned short vlan_id, int smc_ism_register_dmb(struct smc_link_group *lgr, int buf_size, struct smc_buf_desc *dmb_desc); int smc_ism_unregister_dmb(struct smcd_dev *dev, struct smc_buf_desc *dmb_desc); +bool smc_ism_dmb_mappable(struct smcd_dev *smcd); +int smc_ism_attach_dmb(struct smcd_dev *dev, u64 token, + struct smc_buf_desc *dmb_desc); +int smc_ism_detach_dmb(struct smcd_dev *dev, u64 token); int smc_ism_signal_shutdown(struct smc_link_group *lgr); void smc_ism_get_system_eid(u8 **eid); u16 smc_ism_get_chid(struct smcd_dev *dev); From patchwork Sun Dec 10 13:24:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176342 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514498vqy; Sun, 10 Dec 2023 05:25:24 -0800 (PST) X-Google-Smtp-Source: AGHT+IH0UVtXKqBkmunMKeWrh9Urj8CPg4/K4jJ1e2gA+v3ftnvaknsxZqv9AuCmEYsBRi8bbvBL X-Received: by 2002:a17:90a:d255:b0:286:6cc0:cada with SMTP id o21-20020a17090ad25500b002866cc0cadamr1976106pjw.81.1702214724685; Sun, 10 Dec 2023 05:25:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214724; cv=none; d=google.com; s=arc-20160816; b=Yr7QkL5vrdq384ZySQv4EWQYaUHHg/B3DqrAR2US+ylV9WY8WF1LP/YMyTl6dCIwvD 1EbGBejCoBHTGJogjDKiifDfimddhD5hG+DQBc1msZ0clFgZGl2zPLuLlrK/7LWpVEsT dLwbl2nbTClapsBdeT+Wbu2ooud9y3yoeO03x1QpmVst5VTLEIIQbJ36Vwo65ri6se6T qqduQiVzyiypAeBDrCUt7lbxMjBG1vsBrvnLl3JpA+6yqzV4A6PhyZvWlRWF13BKlFIC I3126vfnFtUxdpQYlQtZlKOLWBQNkWdRJ96ixa9umxtcJqnaFMBlbTUpADvMvPlhFj9/ XOlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=50768w5tMk7Jv5TMgXNN97MO59A0QHfuKx4D863PZOI=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=yGu3YBPDSYjKViiw8H02RzWmbt4OC6X0uHvzgdPDC2Gf3Q7nWBAsH5ZhVWiOOooog1 C42rlFmkeDFQ0k6Kx3GM+cHnzvNK0NNENRct3jl2QqMMP8pZQoEU6qb8EzyDtK2Fz3vU mpq3nyPglfIeXYM/un+c3uaXHwUQMBfrjEebNU0/z4PLPQUiKbPdEQou+41Ui2Ro97lk I8AM477fsU8tzPQOOpbzlkBjiPcfjzCzc9gb8+xfz9DKIf+6lT1QuVymPYrdMSPnFpff Jxq0hFbqM19hLu6i/uFpqYXuW5rhLFUP0POKeFHcQFuNf3n9pbxHfqHKkMF1yPmdPrNq VCvg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id jw17-20020a170903279100b001cfde84f92esi4512052plb.82.2023.12.10.05.25.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:25:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id ABBEB806B9F5; Sun, 10 Dec 2023 05:25:20 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232890AbjLJNZL (ORCPT + 99 others); Sun, 10 Dec 2023 08:25:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232895AbjLJNY5 (ORCPT ); Sun, 10 Dec 2023 08:24:57 -0500 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F17FC19AB; Sun, 10 Dec 2023 05:24:45 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8H5PM_1702214680; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8H5PM_1702214680) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:42 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 11/13] net/smc: attach or detach ghost sndbuf to peer DMB. Date: Sun, 10 Dec 2023 21:24:12 +0800 Message-Id: <1702214654-32069-12-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:25:20 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901507237596318 X-GMAIL-MSGID: 1784901507237596318 The ghost sndbuf descriptor will be created and attached to peer DMB once peer token is obtained and it will be detach and freed when the connection is freed. Signed-off-by: Wen Gu --- net/smc/af_smc.c | 21 ++++++++++++++++ net/smc/smc_core.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++- net/smc/smc_core.h | 1 + 3 files changed, 92 insertions(+), 1 deletion(-) diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index 6c0e381..838cf9c 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -1449,6 +1449,14 @@ static int smc_connect_ism(struct smc_sock *smc, } smc_conn_save_peer_info(smc, aclc); + + if (smc_ism_dmb_mappable(smc->conn.lgr->smcd)) { + rc = smcd_buf_attach(smc); + if (rc) { + rc = SMC_CLC_DECL_MEM; /* try to fallback */ + goto connect_abort; + } + } smc_close_init(smc); smc_rx_init(smc); smc_tx_init(smc); @@ -2559,6 +2567,19 @@ static void smc_listen_work(struct work_struct *work) mutex_unlock(&smc_server_lgr_pending); } smc_conn_save_peer_info(new_smc, cclc); + + if (ini->is_smcd && + smc_ism_dmb_mappable(new_smc->conn.lgr->smcd)) { + rc = smcd_buf_attach(new_smc); + if (rc) + /* SMC-D client has completed the CLC process, + * so server can not send decline any more. + * This requires the rc returned here no larger + * than 0. + */ + goto out_decl; + } + smc_listen_out_connected(new_smc); SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini); goto out_free; diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c index 95cc954..4a76e38 100644 --- a/net/smc/smc_core.c +++ b/net/smc/smc_core.c @@ -1149,6 +1149,20 @@ static void smcr_buf_unuse(struct smc_buf_desc *buf_desc, bool is_rmb, } } +static void smcd_buf_detach(struct smc_connection *conn) +{ + struct smcd_dev *smcd = conn->lgr->smcd; + u64 peer_token = conn->peer_token; + + if (!conn->sndbuf_desc) + return; + + smc_ism_detach_dmb(smcd, peer_token); + + kfree(conn->sndbuf_desc); + conn->sndbuf_desc = NULL; +} + static void smc_buf_unuse(struct smc_connection *conn, struct smc_link_group *lgr) { @@ -1192,6 +1206,8 @@ void smc_conn_free(struct smc_connection *conn) if (lgr->is_smcd) { if (!list_empty(&lgr->list)) smc_ism_unset_conn(conn); + if (smc_ism_dmb_mappable(lgr->smcd)) + smcd_buf_detach(conn); tasklet_kill(&conn->rx_tsklet); } else { smc_cdc_wait_pend_tx_wr(conn); @@ -1445,6 +1461,8 @@ static void smc_conn_kill(struct smc_connection *conn, bool soft) smc_sk_wake_ups(smc); if (conn->lgr->is_smcd) { smc_ism_unset_conn(conn); + if (smc_ism_dmb_mappable(conn->lgr->smcd)) + smcd_buf_detach(conn); if (soft) tasklet_kill(&conn->rx_tsklet); else @@ -2455,15 +2473,23 @@ void smc_rmb_sync_sg_for_cpu(struct smc_connection *conn) */ int smc_buf_create(struct smc_sock *smc, bool is_smcd) { + bool sndbuf_created = false; int rc; /* create send buffer */ + if (is_smcd && + smc_ism_dmb_mappable(smc->conn.lgr->smcd)) + goto create_rmb; + rc = __smc_buf_create(smc, is_smcd, false); if (rc) return rc; + sndbuf_created = true; + +create_rmb: /* create rmb */ rc = __smc_buf_create(smc, is_smcd, true); - if (rc) { + if (rc && sndbuf_created) { down_write(&smc->conn.lgr->sndbufs_lock); list_del(&smc->conn.sndbuf_desc->list); up_write(&smc->conn.lgr->sndbufs_lock); @@ -2473,6 +2499,49 @@ int smc_buf_create(struct smc_sock *smc, bool is_smcd) return rc; } +int smcd_buf_attach(struct smc_sock *smc) +{ + struct smc_connection *conn = &smc->conn; + struct smcd_dev *smcd = conn->lgr->smcd; + u64 peer_token = conn->peer_token; + struct smc_buf_desc *buf_desc; + int rc; + + buf_desc = kzalloc(sizeof(*buf_desc), GFP_KERNEL); + if (!buf_desc) + return -ENOMEM; + + /* The ghost sndbuf_desc describes the same memory region as + * peer RMB. Its lifecycle is consistent with the connection's + * and it will be freed with the connections instead of the + * link group. + */ + rc = smc_ism_attach_dmb(smcd, peer_token, buf_desc); + if (rc) + goto free; + + smc->sk.sk_sndbuf = buf_desc->len; + buf_desc->cpu_addr = + (u8 *)buf_desc->cpu_addr + sizeof(struct smcd_cdc_msg); + buf_desc->len -= sizeof(struct smcd_cdc_msg); + conn->sndbuf_desc = buf_desc; + conn->sndbuf_desc->used = 1; + atomic_set(&conn->sndbuf_space, conn->sndbuf_desc->len); + return 0; + +free: + if (conn->rmb_desc) { + /* free local RMB as well */ + down_write(&conn->lgr->rmbs_lock); + list_del(&conn->rmb_desc->list); + up_write(&conn->lgr->rmbs_lock); + smc_buf_free(conn->lgr, true, conn->rmb_desc); + conn->rmb_desc = NULL; + } + kfree(buf_desc); + return rc; +} + static inline int smc_rmb_reserve_rtoken_idx(struct smc_link_group *lgr) { int i; diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h index 1f17537..d93cf51 100644 --- a/net/smc/smc_core.h +++ b/net/smc/smc_core.h @@ -557,6 +557,7 @@ void smc_smcd_terminate(struct smcd_dev *dev, struct smcd_gid *peer_gid, void smc_smcd_terminate_all(struct smcd_dev *dev); void smc_smcr_terminate_all(struct smc_ib_device *smcibdev); int smc_buf_create(struct smc_sock *smc, bool is_smcd); +int smcd_buf_attach(struct smc_sock *smc); int smc_uncompress_bufsize(u8 compressed); int smc_rmb_rtoken_handling(struct smc_connection *conn, struct smc_link *link, struct smc_clc_msg_accept_confirm *clc); From patchwork Sun Dec 10 13:24:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176344 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514845vqy; Sun, 10 Dec 2023 05:26:10 -0800 (PST) X-Google-Smtp-Source: AGHT+IHxaOUckHUT/gHmrXSSjJ/0LQWwxmx2pum+KP/HQ51fmEmWeig2fDBqdyVyZlJYd2DQDpwV X-Received: by 2002:a05:6358:784:b0:16e:12c:16b8 with SMTP id n4-20020a056358078400b0016e012c16b8mr3060348rwj.7.1702214770682; Sun, 10 Dec 2023 05:26:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214770; cv=none; d=google.com; s=arc-20160816; b=JCHlCr8RD7IfELmUFJliatKzpmy/bRz4BI4gHb5tt6IVcTOLIW/OV8HfzyCo+FHbib +twZQ9E3xwSfSRTXhOTfCivCHzhLoGFdZwp5p2lO0Jz9QEx3mJSRJXZ4ayOjOaPq6pNh d1+q3Ojm3qI6MWSqgQlRqyRmb5OtV5EDodYHiMXa19E2tjP7bsU7dSpnYUi4G0TY7LQq D5u8hyYlacsPNefDvoiYqSUQr20pMxd7ghe0MBh4mnC8WM17mhGoxOS+fCROfTfRUr1A nJ4onzXJHUQcJCzv5nMxPag+V0RkCVTTdFyNblC0OkPqgFoSuNNUQXqh9zVIwDilQo6o LLZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=XvDoOBDuZAQ6onbGD6I0aXl4aFD+GGQINmDbQxyRYL4=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=Vb7UPd3CaTWbTRkK8AsYSE1pngBW+6P5KbPXv9ANgd3uSM6DpGiDcqsCthX677Fs4O 0XvfBeVY2f1H3nfTer15NQHrbJSIyWRUi+vF2jsa20LE2dLj9226odrqNleLpsd+q4sF vyYArwUop6Latz7V+8wWY4d3XpoRq/hcfRyCQjJ4mmUYSmB5UmHiMgVTBQaAnhLGx4rZ H6UOXlFyw5Bp4Mo4sQaMOMrfnYp+pD1t6pPGSJ8uRR6iQLUWCckoAhSrsTz6xzUI2vDJ +P9ktX+kuQGBLNhKSMyheWFJ1VK+bcMkE0nIpJLUDLeIRhALxp6S2WaemhggqXdkxTig TLHg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id i10-20020a63540a000000b005be03f0da65si4509268pgb.720.2023.12.10.05.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:26:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id 9E694807BEE8; Sun, 10 Dec 2023 05:25:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232582AbjLJNZP (ORCPT + 99 others); Sun, 10 Dec 2023 08:25:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54630 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232665AbjLJNY7 (ORCPT ); Sun, 10 Dec 2023 08:24:59 -0500 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49EBD1BC5; Sun, 10 Dec 2023 05:24:46 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R261e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8EUSe_1702214682; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8EUSe_1702214682) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:44 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 12/13] net/smc: adapt cursor update when sndbuf is mapped to peer DMB Date: Sun, 10 Dec 2023 21:24:13 +0800 Message-Id: <1702214654-32069-13-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:25:39 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901555187861467 X-GMAIL-MSGID: 1784901555187861467 Since ghost sndbuf shares the same physical memory with peer DMB, the cursor update processing needs to be adapted to ensure that the data to be consumed won't be overwritten. So in this case, the fin_curs and sndbuf_space that were originally updated after sending the CDC message should be modified to not be update until the peer updates cons_curs. Signed-off-by: Wen Gu --- net/smc/smc_cdc.c | 52 +++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index c820ef1..f2b0bcc 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -18,6 +18,7 @@ #include "smc_tx.h" #include "smc_rx.h" #include "smc_close.h" +#include "smc_ism.h" /********************************** send *************************************/ @@ -255,17 +256,25 @@ int smcd_cdc_msg_send(struct smc_connection *conn) return rc; smc_curs_copy(&conn->rx_curs_confirmed, &curs, conn); conn->local_rx_ctrl.prod_flags.cons_curs_upd_req = 0; - /* Calculate transmitted data and increment free send buffer space */ - diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, - &conn->tx_curs_sent); - /* increased by confirmed number of bytes */ - smp_mb__before_atomic(); - atomic_add(diff, &conn->sndbuf_space); - /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ - smp_mb__after_atomic(); - smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + if (!smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* Ghost sndbuf shares the same memory region with + * peer DMB, so don't update the tx_curs_fin and + * sndbuf_space until peer has consumed the data. + */ + /* Calculate transmitted data and increment free + * send buffer space + */ + diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->tx_curs_sent); + /* increased by confirmed number of bytes */ + smp_mb__before_atomic(); + atomic_add(diff, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); - smc_tx_sndbuf_nonfull(smc); + smc_tx_sndbuf_nonfull(smc); + } return rc; } @@ -323,7 +332,7 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, { union smc_host_cursor cons_old, prod_old; struct smc_connection *conn = &smc->conn; - int diff_cons, diff_prod; + int diff_cons, diff_prod, diff_tx; smc_curs_copy(&prod_old, &conn->local_rx_ctrl.prod, conn); smc_curs_copy(&cons_old, &conn->local_rx_ctrl.cons, conn); @@ -339,6 +348,27 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, atomic_add(diff_cons, &conn->peer_rmbe_space); /* guarantee 0 <= peer_rmbe_space <= peer_rmbe_size */ smp_mb__after_atomic(); + + if (conn->lgr->is_smcd && + smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* Ghost sndbuf shares the same memory region with + * peer RMB, so update tx_curs_fin and sndbuf_space + * when peer has consumed the data. + */ + /* calculate peer rmb consumed data */ + diff_tx = smc_curs_diff(conn->sndbuf_desc->len, + &conn->tx_curs_fin, + &conn->local_rx_ctrl.cons); + /* increase local sndbuf space and fin_curs */ + smp_mb__before_atomic(); + atomic_add(diff_tx, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, + &conn->local_rx_ctrl.cons, conn); + + smc_tx_sndbuf_nonfull(smc); + } } diff_prod = smc_curs_diff(conn->rmb_desc->len, &prod_old, From patchwork Sun Dec 10 13:24:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 176345 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp6514937vqy; Sun, 10 Dec 2023 05:26:21 -0800 (PST) X-Google-Smtp-Source: AGHT+IE8kQ1Vwz1mQdrY/Jif9M0XKsBoS33HXy9c9IawlzDq51SJvAKwfOD1ld9isBs1I8fWd+NE X-Received: by 2002:a05:6a00:2e84:b0:6ce:2732:573 with SMTP id fd4-20020a056a002e8400b006ce27320573mr3119404pfb.36.1702214781463; Sun, 10 Dec 2023 05:26:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702214781; cv=none; d=google.com; s=arc-20160816; b=o1MC0eN1BjkDYm+7ThRmadf353M9/Eq83os49p1MDiFPAnfTssTfFm4PemP8nXp9S3 yN/UGJxlRNs6GBA9fDxn1cv3YcqxsBI5Yf7LG+KU6Rczsc3SSE6evv62Lfpx7NCY0B86 SKVHbf5M732OEZHQPKYLO/sN9ILgz/ZfgDi2UY+uw+wMjR10PR8Qu8uwAtjCaUXfj1ki tYq/iIa8Gc9nofPbjqu9StcHtpIp7ds5e3pEykJtvkCHj17785Af8iph/dFUwWx0f8DL Yp0QkDSjSIqnTkcUqj0R6uxD8Y/ebzuXpOyHgCiv2XHTUFQKOqxk+Xwzv67q9Ei7w0uW oYjw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=Cre+1nXrsTR7FdFu4bMcwv2JC0KdPxE1Jz2K6EvhJf8=; fh=OseD/KU9IozxwwuPl+YAZpOMu/WojE/9fn/xPSMdB/8=; b=jwYkpVTzwsmueJ0eUmU9eiVWcnVTfMns3Cup/lomyopNhmwex74xkPVON4FwIBexDG VQNGnMLyTvkY5XJzblEoYcFi0p+n6pLCnSy6+T8z/ncTxSgjWS9c/E4RSCSlZqRX0QBC qlKy5RMhYap3VQDx9hvmmILA3ydB3rLUIW66mx+Hm1AC2RtXGGKrV1rMX3rRurJkWS59 p+wP2rTn/8kJ1jVTQe0AkpPEUUksYtLNYI/hLXk66me9beEVShynRLxn2vxY9yttHbPd l920ysL3uoxpHpwiiWYfaTMbW9lVangmZ52sYp5u9m8+yxyhjgdr3ZN5MFQmrmZyT26R 01ug== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from pete.vger.email (pete.vger.email. [23.128.96.36]) by mx.google.com with ESMTPS id l3-20020a056a00140300b006cd989cfe42si4551063pfu.7.2023.12.10.05.26.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Dec 2023 05:26:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by pete.vger.email (Postfix) with ESMTP id 16FDE80688B1; Sun, 10 Dec 2023 05:26:14 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at pete.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233042AbjLJNZ0 (ORCPT + 99 others); Sun, 10 Dec 2023 08:25:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232597AbjLJNZD (ORCPT ); Sun, 10 Dec 2023 08:25:03 -0500 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6F591BEF; Sun, 10 Dec 2023 05:24:48 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R371e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0Vy8DodD_1702214684; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vy8DodD_1702214684) by smtp.aliyun-inc.com; Sun, 10 Dec 2023 21:24:46 +0800 From: Wen Gu To: wintera@linux.ibm.com, wenjia@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, kgraul@linux.ibm.com, jaka@linux.ibm.com Cc: borntraeger@linux.ibm.com, svens@linux.ibm.com, alibuda@linux.alibaba.com, tonylu@linux.alibaba.com, guwen@linux.alibaba.com, linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next 13/13] net/smc: implement {at|de}tach_dmb interfaces of loopback-ism Date: Sun, 10 Dec 2023 21:24:14 +0800 Message-Id: <1702214654-32069-14-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> References: <1702214654-32069-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Sun, 10 Dec 2023 05:26:14 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784901566845826576 X-GMAIL-MSGID: 1784901566845826576 This implements attach_dmb and detach_dmb interfaces of loopback-ism. The DMB won't be unregistered by loopback-ism until no sndbuf is attached to it. Signed-off-by: Wen Gu --- net/smc/smc_loopback.c | 107 +++++++++++++++++++++++++++++++++++++++++++------ net/smc/smc_loopback.h | 4 ++ 2 files changed, 99 insertions(+), 12 deletions(-) diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c index 057ea6a..909c069 100644 --- a/net/smc/smc_loopback.c +++ b/net/smc/smc_loopback.c @@ -198,6 +198,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, } dmb_node->len = dmb->dmb_len; dmb_node->dma_addr = SMC_DMA_ADDR_INVALID; + refcount_set(&dmb_node->refcnt, 1); again: /* add new dmb into hash table */ @@ -211,6 +212,7 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb, } hash_add(ldev->dmb_ht, &dmb_node->list, dmb_node->token); write_unlock(&ldev->dmb_ht_lock); + atomic_inc(&ldev->dmb_cnt); SMC_LO_STAT_DMBS_INC(ldev); dmb->sba_idx = dmb_node->sba_idx; @@ -233,8 +235,8 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; struct smc_lo_dev *ldev = smcd->priv; - /* remove dmb from hash table */ - write_lock(&ldev->dmb_ht_lock); + /* find dmb from hash table */ + read_lock(&ldev->dmb_ht_lock); hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) { if (tmp_node->token == dmb->dmb_tok) { dmb_node = tmp_node; @@ -242,9 +244,18 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) } } if (!dmb_node) { - write_unlock(&ldev->dmb_ht_lock); + read_unlock(&ldev->dmb_ht_lock); return -EINVAL; } + read_unlock(&ldev->dmb_ht_lock); + + /* wait for peer sndbuf to detach from this dmb */ + if (!refcount_dec_and_test(&dmb_node->refcnt)) + wait_event(ldev->dmbs_release, + !refcount_read(&dmb_node->refcnt)); + + /* remove dmb from hash table */ + write_lock(&ldev->dmb_ht_lock); hash_del(&dmb_node->list); write_unlock(&ldev->dmb_ht_lock); @@ -253,6 +264,66 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) kfree(dmb_node); SMC_LO_STAT_DMBS_DEC(ldev); + if (atomic_dec_and_test(&ldev->dmb_cnt)) + wake_up(&ldev->ldev_release); + return 0; +} + +static int smc_lo_attach_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb) +{ + struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + /* find dmb_node according to dmb->dmb_tok */ + read_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb->dmb_tok) { + if (tmp_node->token == dmb->dmb_tok) { + dmb_node = tmp_node; + break; + } + } + if (!dmb_node) { + read_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + read_unlock(&ldev->dmb_ht_lock); + + if (!refcount_inc_not_zero(&dmb_node->refcnt)) + /* the dmb is being unregistered, but has + * not been removed from the hash table. + */ + return -EINVAL; + + /* provide dmb information */ + dmb->sba_idx = dmb_node->sba_idx; + dmb->dmb_tok = dmb_node->token; + dmb->cpu_addr = dmb_node->cpu_addr; + dmb->dma_addr = dmb_node->dma_addr; + dmb->dmb_len = dmb_node->len; + return 0; +} + +static int smc_lo_detach_dmb(struct smcd_dev *smcd, u64 token) +{ + struct smc_lo_dmb_node *dmb_node = NULL, *tmp_node; + struct smc_lo_dev *ldev = smcd->priv; + + /* find dmb_node according to dmb->dmb_tok */ + read_lock(&ldev->dmb_ht_lock); + hash_for_each_possible(ldev->dmb_ht, tmp_node, list, token) { + if (tmp_node->token == token) { + dmb_node = tmp_node; + break; + } + } + if (!dmb_node) { + read_unlock(&ldev->dmb_ht_lock); + return -EINVAL; + } + read_unlock(&ldev->dmb_ht_lock); + + if (refcount_dec_and_test(&dmb_node->refcnt)) + wake_up_all(&ldev->dmbs_release); return 0; } @@ -288,7 +359,15 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, { struct smc_lo_dmb_node *rmb_node = NULL, *tmp_node; struct smc_lo_dev *ldev = smcd->priv; - + struct smc_connection *conn; + + if (!sf) { + /* Since ghost sndbuf shares the same memory region + * with peer DMB, there is no need to 'transfer' data + * from sndbuf to peer DMB. + */ + return 0; + } read_lock(&ldev->dmb_ht_lock); hash_for_each_possible(ldev->dmb_ht, tmp_node, list, dmb_tok) { if (tmp_node->token == dmb_tok) { @@ -305,14 +384,10 @@ static int smc_lo_move_data(struct smcd_dev *smcd, u64 dmb_tok, memcpy((char *)rmb_node->cpu_addr + offset, data, size); SMC_LO_STAT_TX_BYTES(ldev, size); - if (sf) { - struct smc_connection *conn = - smcd->conn[rmb_node->sba_idx]; - - if (conn && !conn->killed) { - SMC_LO_STAT_RX_BYTES(ldev, size); - smcd_cdc_rx_handler(conn); - } + conn = smcd->conn[rmb_node->sba_idx]; + if (conn && !conn->killed) { + SMC_LO_STAT_RX_BYTES(ldev, size); + smcd_cdc_rx_handler(conn); } return 0; } @@ -345,6 +420,8 @@ static struct device *smc_lo_get_dev(struct smcd_dev *smcd) .query_remote_gid = smc_lo_query_rgid, .register_dmb = smc_lo_register_dmb, .unregister_dmb = smc_lo_unregister_dmb, + .attach_dmb = smc_lo_attach_dmb, + .detach_dmb = smc_lo_detach_dmb, .add_vlan_id = smc_lo_add_vlan_id, .del_vlan_id = smc_lo_del_vlan_id, .set_vlan_required = smc_lo_set_vlan_required, @@ -426,12 +503,18 @@ static int smc_lo_dev_init(struct smc_lo_dev *ldev) smc_lo_generate_id(ldev); rwlock_init(&ldev->dmb_ht_lock); hash_init(ldev->dmb_ht); + atomic_set(&ldev->dmb_cnt, 0); + init_waitqueue_head(&ldev->dmbs_release); + init_waitqueue_head(&ldev->ldev_release); + return smcd_lo_register_dev(ldev); } static void smc_lo_dev_exit(struct smc_lo_dev *ldev) { smcd_lo_unregister_dev(ldev); + if (atomic_read(&ldev->dmb_cnt)) + wait_event(ldev->ldev_release, !atomic_read(&ldev->dmb_cnt)); } static void smc_lo_dev_release(struct device *dev) diff --git a/net/smc/smc_loopback.h b/net/smc/smc_loopback.h index ad0feaa..df6c252 100644 --- a/net/smc/smc_loopback.h +++ b/net/smc/smc_loopback.h @@ -30,6 +30,7 @@ struct smc_lo_dmb_node { u32 sba_idx; void *cpu_addr; dma_addr_t dma_addr; + refcount_t refcnt; }; struct smc_lo_dev_stats64 { @@ -45,9 +46,12 @@ struct smc_lo_dev { u16 chid; struct smcd_gid local_gid; struct smc_lo_dev_stats64 __percpu *stats; + atomic_t dmb_cnt; rwlock_t dmb_ht_lock; DECLARE_BITMAP(sba_idx_mask, SMC_LO_MAX_DMBS); DECLARE_HASHTABLE(dmb_ht, SMC_LO_DMBS_HASH_BITS); + wait_queue_head_t dmbs_release; + wait_queue_head_t ldev_release; }; #define SMC_LO_STAT_SUB(ldev, key, val) \