From patchwork Mon Mar 27 03:28:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 75192 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1248782vqo; Sun, 26 Mar 2023 20:33:37 -0700 (PDT) X-Google-Smtp-Source: AK7set/aFLtafIybJXrNFRGP1EteCVRvNQ7HreXM1XGiCRvZI0ieCh6iMPWdzBe7IgSB+iNspPMv X-Received: by 2002:a17:907:6d91:b0:8af:2f5e:93e3 with SMTP id sb17-20020a1709076d9100b008af2f5e93e3mr20320388ejc.29.1679888017760; Sun, 26 Mar 2023 20:33:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679888017; cv=none; d=google.com; s=arc-20160816; b=NGFBwqKPUb0IteOVB0nOjMICybkjSym4nHZnaUXY4Qs+dJWCFXNo1qDXtB6Pz4cNbP /bnDVa2ZvHugM7P0D01JQ/5fn4qrUuTaYD8N/dcBGp5HgonKBuXpYQkvOgLwPldwKdB/ twUceGHfBbPm0swlCUA/BtOcBwA9Ctfm+Yl3YJLDFx2lbB32kHJ20kv1UyxZI7aX71Pt Qms9wthYRprn+oHACE3+3YFe2jj81GeQqYh3hgzH8xB7FHXnoPnKH7/2AH22JscTtZfs ULWk5p+TgRXmf+bAz+smB6HM3eHbJ0ayeeJ5oyTnh8xTuavJAQwoaFMGvbOTxQsNjouB fvAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=WspUPY56llsxWqE0J/S3tGFmY3vq3FxldCe7rIFT1jM=; b=UFgMnLfvmsYtUALRWH6edmhQREg0Bomc2P3HzNHPr7orLoHWnT0G0Rk1Pn/zvb9VUD FnDowKp5NjQGOKRDAZyUPxBpXiDVCLR4xoS3sHt8qTpefb/x4+cLRFiUzmsVK4nmk6ye OgTWDH+nd1NRxZvEiU2ENdLxNPYBdWUDyJ0IP1kNSc/HNHv7LrpGn4cVnOQ6DAFAO5t0 GCAL40QgSDW/Dqy9qrajOKUAN1rwGZbaH53TdYGv7CnP9QSI+zmisZc6jVRzmlfmYaxh fI29gM1j+FC1I5kaMEzv4vlydusLQORPeDjj+X2ZSu3Z50/hr4wmjZ4yblO/fGyrbQiu 3dVg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h24-20020aa7c958000000b005023ea472fdsi2513280edt.588.2023.03.26.20.33.12; Sun, 26 Mar 2023 20:33:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232362AbjC0D3k (ORCPT + 99 others); Sun, 26 Mar 2023 23:29:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232019AbjC0D3E (ORCPT ); Sun, 26 Mar 2023 23:29:04 -0400 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72FC655A4; Sun, 26 Mar 2023 20:28:40 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R261e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0Vef4Ary_1679887715; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vef4Ary_1679887715) by smtp.aliyun-inc.com; Mon, 27 Mar 2023 11:28:37 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, wintera@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v4 8/9] net/smc: Modify cursor update logic when using mappable DMB Date: Mon, 27 Mar 2023 11:28:18 +0800 Message-Id: <1679887699-54797-9-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1679887699-54797-1-git-send-email-guwen@linux.alibaba.com> References: <1679887699-54797-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-8.0 required=5.0 tests=ENV_AND_HDR_SPF_MATCH, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS, UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761490257749511636?= X-GMAIL-MSGID: =?utf-8?q?1761490257749511636?= Since local sndbuf shares the same physical memory region with peer RMB when using mappable DMBs, the cursor update logic needs to be adapted. The main concern is to ensure that the data written by local to this memory region won't overwrite the data that has not been consumed by the peer. So in this scene, the fin_curs and sndbuf_space that were originally updated when sending out CDC message are not updated until the cons_curs update from the peer is received. Signed-off-by: Wen Gu --- net/smc/smc_cdc.c | 50 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index 2f79bac..915b8e7 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -18,6 +18,7 @@ #include "smc_tx.h" #include "smc_rx.h" #include "smc_close.h" +#include "smc_ism.h" /********************************** send *************************************/ @@ -256,17 +257,24 @@ int smcd_cdc_msg_send(struct smc_connection *conn) return rc; smc_curs_copy(&conn->rx_curs_confirmed, &curs, conn); conn->local_rx_ctrl.prod_flags.cons_curs_upd_req = 0; - /* Calculate transmitted data and increment free send buffer space */ - diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, - &conn->tx_curs_sent); - /* increased by confirmed number of bytes */ - smp_mb__before_atomic(); - atomic_add(diff, &conn->sndbuf_space); - /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ - smp_mb__after_atomic(); - smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + if (!smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * don't update the tx_curs_fin and sndbuf_space until + * peer has consumed the data in RMB. + */ - smc_tx_sndbuf_nonfull(smc); + /* Calculate transmitted data and increment free send buffer space */ + diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->tx_curs_sent); + /* increased by confirmed number of bytes */ + smp_mb__before_atomic(); + atomic_add(diff, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + + smc_tx_sndbuf_nonfull(smc); + } return rc; } @@ -324,7 +332,7 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, { union smc_host_cursor cons_old, prod_old; struct smc_connection *conn = &smc->conn; - int diff_cons, diff_prod; + int diff_cons, diff_prod, diff_tx; smc_curs_copy(&prod_old, &conn->local_rx_ctrl.prod, conn); smc_curs_copy(&cons_old, &conn->local_rx_ctrl.cons, conn); @@ -340,6 +348,26 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, atomic_add(diff_cons, &conn->peer_rmbe_space); /* guarantee 0 <= peer_rmbe_space <= peer_rmbe_size */ smp_mb__after_atomic(); + + if (conn->lgr->is_smcd && + smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * update tx_curs_fin and sndbuf_space when peer has + * consumed the data in it's RMB. + */ + + /* calculate peer rmb consumed data */ + diff_tx = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->local_rx_ctrl.cons); + /* increase local sndbuf space and fin_curs */ + smp_mb__before_atomic(); + atomic_add(diff_tx, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->local_rx_ctrl.cons, conn); + + smc_tx_sndbuf_nonfull(smc); + } } diff_prod = smc_curs_diff(conn->rmb_desc->len, &prod_old,