From patchwork Wed Feb 15 16:18:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 57620 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp284701wrn; Wed, 15 Feb 2023 08:24:37 -0800 (PST) X-Google-Smtp-Source: AK7set+I36rnP9p/ceFjxFUODG/NMsixWpZu2PCH8DBnIGlwct5eUePB+3w6mmkcWeJDb6DU8rN3 X-Received: by 2002:a17:906:a41a:b0:877:ef84:c7de with SMTP id l26-20020a170906a41a00b00877ef84c7demr2408143ejz.61.1676478277323; Wed, 15 Feb 2023 08:24:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1676478277; cv=none; d=google.com; s=arc-20160816; b=sl7dzF0KifQ9p9cWwrAmt3GFFQ3wrPFM2stWMhOAW5VNnGHf6jD9vc7X2zySvjlT88 2aoPYrDsYkLdo0uu5GQqfmkCEO/S7E0mTPL9HlTy9m+TZTMTLJYtBrsgQ/0WecHXjWQs ZxOwsNLSbt+u+eZFa/dg6ZTR7cZvJMqtczbnHaTcAkAFu/oAXSYyh6HpjWGCplLc5aXN yGj8006Pb8F96QspLPX/5er6kpuHfJJGsb3yHulhwlzqPcDoeG3XfeBxj7X+ZzWY0H4a rdt1dy7nGcu0C/74QLrREM0+C85QtsTYjenX3KDtWf9Wwu+4baHJRm2WklMhaUcAUhOO lnqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=RE5yQDtIYXcVGRdSptSw8SumkpVF5dzzzQlZyO6QjA0=; b=h9kMVhlGS4LVOFDz4Rj4wxLyzAJU4HX5yWU8H20xO8S7U3Yjz08uRu22Cgt79O6J4p qH56ijR0Fn8UkISK4VSunmaqmQmmB8BHxACJrgu4fkYdkIK9zCy3kGMHPGPl9ORmoZsk dITYJT9V+SHBW5wXt8faA/7Sjxtj76gc2nsQm45wdQgMS/cUkYmC71nhAlzb6AIsuzeD yk5VNTkDlyRc0aw0wd8UuPLGkPRw74vSQ/m/gWgoHezu3n2FpsA/51RI4iUEaEIbc3or hPy+hEUWKxHdLVoEOS4Aj2UZ0u4A6FnPmKN5TNOrwmkLjUJH9Zjf13heaCVvzGT9MilF wgBg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id uo37-20020a170907cc2500b008aa6e271052si20954288ejc.540.2023.02.15.08.24.13; Wed, 15 Feb 2023 08:24:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229795AbjBOQT2 (ORCPT + 99 others); Wed, 15 Feb 2023 11:19:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230048AbjBOQS7 (ORCPT ); Wed, 15 Feb 2023 11:18:59 -0500 Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33F6F3B0CE; Wed, 15 Feb 2023 08:18:52 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R811e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045168;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vbl04yG_1676477927; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vbl04yG_1676477927) by smtp.aliyun-inc.com; Thu, 16 Feb 2023 00:18:49 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v3 8/9] net/smc: Modify cursor update logic when using mappable DMB Date: Thu, 16 Feb 2023 00:18:24 +0800 Message-Id: <1676477905-88043-9-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1676477905-88043-1-git-send-email-guwen@linux.alibaba.com> References: <1676477905-88043-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY, USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757914886159295086?= X-GMAIL-MSGID: =?utf-8?q?1757914886159295086?= Since local sndbuf shares the same physical memory region with peer RMB when using mappable DMBs, the cursor update logic needs to be adapted. The main concern is to ensure that the data written by local to this memory region won't overwrite the data that has not been consumed by the peer. So in this scene, the fin_curs and sndbuf_space that were originally updated when sending out CDC message are not updated until the cons_curs update from the peer is received. Signed-off-by: Wen Gu --- net/smc/smc_cdc.c | 50 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index 9023684..74fcd2f 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -18,6 +18,7 @@ #include "smc_tx.h" #include "smc_rx.h" #include "smc_close.h" +#include "smc_ism.h" /********************************** send *************************************/ @@ -253,17 +254,24 @@ int smcd_cdc_msg_send(struct smc_connection *conn) return rc; smc_curs_copy(&conn->rx_curs_confirmed, &curs, conn); conn->local_rx_ctrl.prod_flags.cons_curs_upd_req = 0; - /* Calculate transmitted data and increment free send buffer space */ - diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, - &conn->tx_curs_sent); - /* increased by confirmed number of bytes */ - smp_mb__before_atomic(); - atomic_add(diff, &conn->sndbuf_space); - /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ - smp_mb__after_atomic(); - smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + if (!smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * don't update the tx_curs_fin and sndbuf_space until + * peer has consumed the data in RMB. + */ - smc_tx_sndbuf_nonfull(smc); + /* Calculate transmitted data and increment free send buffer space */ + diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->tx_curs_sent); + /* increased by confirmed number of bytes */ + smp_mb__before_atomic(); + atomic_add(diff, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + + smc_tx_sndbuf_nonfull(smc); + } return rc; } @@ -321,7 +329,7 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, { union smc_host_cursor cons_old, prod_old; struct smc_connection *conn = &smc->conn; - int diff_cons, diff_prod; + int diff_cons, diff_prod, diff_tx; smc_curs_copy(&prod_old, &conn->local_rx_ctrl.prod, conn); smc_curs_copy(&cons_old, &conn->local_rx_ctrl.cons, conn); @@ -337,6 +345,26 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, atomic_add(diff_cons, &conn->peer_rmbe_space); /* guarantee 0 <= peer_rmbe_space <= peer_rmbe_size */ smp_mb__after_atomic(); + + if (conn->lgr->is_smcd && + smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * update tx_curs_fin and sndbuf_space when peer has + * consumed the data in it's RMB. + */ + + /* calculate peer rmb consumed data */ + diff_tx = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->local_rx_ctrl.cons); + /* increase local sndbuf space and fin_curs */ + smp_mb__before_atomic(); + atomic_add(diff_tx, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->local_rx_ctrl.cons, conn); + + smc_tx_sndbuf_nonfull(smc); + } } diff_prod = smc_curs_diff(conn->rmb_desc->len, &prod_old,