From patchwork Sun Apr 23 12:17:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wen Gu X-Patchwork-Id: 86705 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2171278vqo; Sun, 23 Apr 2023 05:30:05 -0700 (PDT) X-Google-Smtp-Source: AKy350ac6WEeB0eMIqLff3+uhGEmirAfQRduZSR1A8kxGVC8s9uNX755Lm0fWC7y6gnPTwxQ9tFJ X-Received: by 2002:a17:90b:81:b0:247:3895:e416 with SMTP id bb1-20020a17090b008100b002473895e416mr10681839pjb.16.1682253004987; Sun, 23 Apr 2023 05:30:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1682253004; cv=none; d=google.com; s=arc-20160816; b=IEgrSBQU8G0f/FTVQdQA5CHmJH/VU9gqyxd3xG8x2j55/U3TLr2WXJP+wrZFdxrJNB JgG3JiCwO5SPojc/Fs2SDAXKocp305ZaKQp4ucYWqYhyWFSaDj8FS5mCb1SCuCWrIHLW xw6jPYG4RV3lVT98iqlAHFLHDTSOH1VXBTsx305+VFe5i1COvHOujnhKuEkGh+9fSWdg KoxAG2gGG+TiLbHeDTo0DMaNwf3DBL0YKM76+2sGCOaTS3pLHICNXCv+OTh82Hv18Px8 YoG58nGN5wa/UjMbD4B3ZCW5WvlIr3o+slM05MQWe2Pud+Uav0/LIlvEQF5Esh8uZWBN VrCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:references:in-reply-to:message-id:date:subject :cc:to:from; bh=WspUPY56llsxWqE0J/S3tGFmY3vq3FxldCe7rIFT1jM=; b=o2gJ05Zr7txxZMXL8tTgWHr7EoAcmrOcVQHZZ7v+/r2Gr+uTXJkNIjm9ezzUwzF1TG NTuAM3CG34XqvvSosz8hyJ9ZvS4B/mtt/sFYlAH+GHsY8lOElxrWloU8X9BwEQf7diMy i7T36rYGA4S8cE6cPqxElOKfKlJLpI6AE8zXMjgaV4sQ3Sf4jyvBPyE+k8QAfW44XQg2 8PnWnXoALBQhmjWXj8oXw5HHxZbVFV+hgpnnt1ZOVuJSRqd2MrASzBcnhnjanVZkK0gr t1qTR5OX5oMtQiImcjP6VPGBDdMNh8PZg14w1TOs7uconqYSAZ88PohUmUeDHYtFUzMZ lH6Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h10-20020a63f90a000000b005194ff488d9si8306992pgi.516.2023.04.23.05.29.50; Sun, 23 Apr 2023 05:30:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230237AbjDWMTX (ORCPT + 99 others); Sun, 23 Apr 2023 08:19:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230070AbjDWMSy (ORCPT ); Sun, 23 Apr 2023 08:18:54 -0400 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6C3E3A95; Sun, 23 Apr 2023 05:18:14 -0700 (PDT) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R881e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=guwen@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0Vgjt3Kt_1682252288; Received: from localhost(mailfrom:guwen@linux.alibaba.com fp:SMTPD_---0Vgjt3Kt_1682252288) by smtp.aliyun-inc.com; Sun, 23 Apr 2023 20:18:10 +0800 From: Wen Gu To: kgraul@linux.ibm.com, wenjia@linux.ibm.com, jaka@linux.ibm.com, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com Cc: linux-s390@vger.kernel.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH net-next v5 8/9] net/smc: Modify cursor update logic when using mappable DMB Date: Sun, 23 Apr 2023 20:17:50 +0800 Message-Id: <1682252271-2544-9-git-send-email-guwen@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> References: <1682252271-2544-1-git-send-email-guwen@linux.alibaba.com> X-Spam-Status: No, score=-9.9 required=5.0 tests=BAYES_00, ENV_AND_HDR_SPF_MATCH,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, UNPARSEABLE_RELAY,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763970126845621390?= X-GMAIL-MSGID: =?utf-8?q?1763970126845621390?= Since local sndbuf shares the same physical memory region with peer RMB when using mappable DMBs, the cursor update logic needs to be adapted. The main concern is to ensure that the data written by local to this memory region won't overwrite the data that has not been consumed by the peer. So in this scene, the fin_curs and sndbuf_space that were originally updated when sending out CDC message are not updated until the cons_curs update from the peer is received. Signed-off-by: Wen Gu --- net/smc/smc_cdc.c | 50 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/net/smc/smc_cdc.c b/net/smc/smc_cdc.c index 2f79bac..915b8e7 100644 --- a/net/smc/smc_cdc.c +++ b/net/smc/smc_cdc.c @@ -18,6 +18,7 @@ #include "smc_tx.h" #include "smc_rx.h" #include "smc_close.h" +#include "smc_ism.h" /********************************** send *************************************/ @@ -256,17 +257,24 @@ int smcd_cdc_msg_send(struct smc_connection *conn) return rc; smc_curs_copy(&conn->rx_curs_confirmed, &curs, conn); conn->local_rx_ctrl.prod_flags.cons_curs_upd_req = 0; - /* Calculate transmitted data and increment free send buffer space */ - diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, - &conn->tx_curs_sent); - /* increased by confirmed number of bytes */ - smp_mb__before_atomic(); - atomic_add(diff, &conn->sndbuf_space); - /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ - smp_mb__after_atomic(); - smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + if (!smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * don't update the tx_curs_fin and sndbuf_space until + * peer has consumed the data in RMB. + */ - smc_tx_sndbuf_nonfull(smc); + /* Calculate transmitted data and increment free send buffer space */ + diff = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->tx_curs_sent); + /* increased by confirmed number of bytes */ + smp_mb__before_atomic(); + atomic_add(diff, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->tx_curs_sent, conn); + + smc_tx_sndbuf_nonfull(smc); + } return rc; } @@ -324,7 +332,7 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, { union smc_host_cursor cons_old, prod_old; struct smc_connection *conn = &smc->conn; - int diff_cons, diff_prod; + int diff_cons, diff_prod, diff_tx; smc_curs_copy(&prod_old, &conn->local_rx_ctrl.prod, conn); smc_curs_copy(&cons_old, &conn->local_rx_ctrl.cons, conn); @@ -340,6 +348,26 @@ static void smc_cdc_msg_recv_action(struct smc_sock *smc, atomic_add(diff_cons, &conn->peer_rmbe_space); /* guarantee 0 <= peer_rmbe_space <= peer_rmbe_size */ smp_mb__after_atomic(); + + if (conn->lgr->is_smcd && + smc_ism_dmb_mappable(conn->lgr->smcd)) { + /* If local sndbuf has been mapped to peer RMB, then + * update tx_curs_fin and sndbuf_space when peer has + * consumed the data in it's RMB. + */ + + /* calculate peer rmb consumed data */ + diff_tx = smc_curs_diff(conn->sndbuf_desc->len, &conn->tx_curs_fin, + &conn->local_rx_ctrl.cons); + /* increase local sndbuf space and fin_curs */ + smp_mb__before_atomic(); + atomic_add(diff_tx, &conn->sndbuf_space); + /* guarantee 0 <= sndbuf_space <= sndbuf_desc->len */ + smp_mb__after_atomic(); + smc_curs_copy(&conn->tx_curs_fin, &conn->local_rx_ctrl.cons, conn); + + smc_tx_sndbuf_nonfull(smc); + } } diff_prod = smc_curs_diff(conn->rmb_desc->len, &prod_old,