From patchwork Wed Apr 19 05:52:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Daisuke Matsuda (Fujitsu)" X-Patchwork-Id: 85187 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp133430vqo; Tue, 18 Apr 2023 22:56:23 -0700 (PDT) X-Google-Smtp-Source: AKy350Zy+6CycSKV1Najg8izc1fuLIBr+MKih4QDaIga7qRht33bscuEIUj+0OMqnaH0g2GtLM3/ X-Received: by 2002:a05:6a00:2e0f:b0:627:f0e1:4fbf with SMTP id fc15-20020a056a002e0f00b00627f0e14fbfmr2483117pfb.33.1681883783635; Tue, 18 Apr 2023 22:56:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681883783; cv=none; d=google.com; s=arc-20160816; b=Lfoj8ZrLzITRU2Hp92mXvQlGS8guXzwcrfEVAyf19jefypQ43jG/4JP0H4AT2C/f3C Ir5Zt/IOoDNO3kMy4RimfHMTmWEFp0zHh3FKajNyEqnwwBsg1U2OsDvkl7IZ5fsimjNf Pk9tSIVr7YI7xK9AAtMvekUe/tnrCFA8VT5Dx33Hxp3bZLybTVpjswRXx/IwAgbA4kg+ B6a7aaoHDgBqkXO4TX76b/8En7brJPwAhI91KSRPPXxEGht8Y5ALQ8n6JI2GZcJef4hj /VKX0RTB0jghizzir734I33i9Yf+SVOfYokFFyqiTyZ+8ohFGqEUC8Q4ps5WI92Ue+kf aWwg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=tpech6R/69PJln8zKjlKffgw5PmQ15sXtvndCws9PF0=; b=AvNm87Y+rvDRxYe0cLAy73XUXPF4Chnd+ZVrBGPDmgjqfknC3i/GkLd/JP+qjGcxeR T7E9OGVQJbo9xCLAd1eWm1uhY7cKcUYqJtu5HfvOAY7hRdwrhqgazB77jmvM/SFsX5gN HvVxVV0dvRxvqvhZO43qnhm6YQuSEgYguZiOG+Z8Z6GZqq3Ee/78rhJ0eC9yeJanrUj5 jcrC67firGkkwd4uIZxhnPcN1Ky+1rWKAClC+ApEur30WiCpt2PFogGcN824mxAf6hVr JggxxE7GP4wMmWnNTcZKP//tdiuL8gDz7/OnVwoXguO2OCQlB2be1MGQjm09msD3f8V8 Gc9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m68-20020a632647000000b00513f16584d1si16630635pgm.465.2023.04.18.22.56.11; Tue, 18 Apr 2023 22:56:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231925AbjDSFx0 (ORCPT + 99 others); Wed, 19 Apr 2023 01:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231710AbjDSFxI (ORCPT ); Wed, 19 Apr 2023 01:53:08 -0400 Received: from esa8.hc1455-7.c3s2.iphmx.com (esa8.hc1455-7.c3s2.iphmx.com [139.138.61.253]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A80C5FEC; Tue, 18 Apr 2023 22:53:05 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10684"; a="102053531" X-IronPort-AV: E=Sophos;i="5.99,208,1677510000"; d="scan'208";a="102053531" Received: from unknown (HELO yto-r2.gw.nic.fujitsu.com) ([218.44.52.218]) by esa8.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Apr 2023 14:53:02 +0900 Received: from yto-m4.gw.nic.fujitsu.com (yto-nat-yto-m4.gw.nic.fujitsu.com [192.168.83.67]) by yto-r2.gw.nic.fujitsu.com (Postfix) with ESMTP id 869B1DE527; Wed, 19 Apr 2023 14:53:00 +0900 (JST) Received: from m3003.s.css.fujitsu.com (m3003.s.css.fujitsu.com [10.128.233.114]) by yto-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id A99B8D3F23; Wed, 19 Apr 2023 14:52:59 +0900 (JST) Received: from localhost.localdomain (unknown [10.118.237.106]) by m3003.s.css.fujitsu.com (Postfix) with ESMTP id 79A962043C64; Wed, 19 Apr 2023 14:52:59 +0900 (JST) From: Daisuke Matsuda To: linux-rdma@vger.kernel.org, leonro@nvidia.com, jgg@nvidia.com, zyjzyj2000@gmail.com Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, yangx.jy@fujitsu.com, lizhijian@fujitsu.com, Daisuke Matsuda Subject: [PATCH for-next v4 8/8] RDMA/rxe: Add support for the traditional Atomic operations with ODP Date: Wed, 19 Apr 2023 14:52:00 +0900 Message-Id: X-Mailer: git-send-email 2.39.1 In-Reply-To: References: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763582969895703361?= X-GMAIL-MSGID: =?utf-8?q?1763582969895703361?= Enable 'fetch and add' and 'compare and swap' operations to manipulate data in an ODP-enabled MR. This is comprised of the following steps: 1. Check the driver page table(umem_odp->dma_list) to see if the target page is both readable and writable. 2. If not, then trigger page fault to map the page. 3. Update the entry in the MR xarray. 4. Execute the operation. umem_mutex is used to ensure that dma_list (an array of addresses of an MR) is not changed while it is being checked and that the target page is not invalidated before data access completes. Signed-off-by: Daisuke Matsuda --- drivers/infiniband/sw/rxe/rxe.c | 1 + drivers/infiniband/sw/rxe/rxe_loc.h | 9 +++++++++ drivers/infiniband/sw/rxe/rxe_odp.c | 26 ++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_resp.c | 5 ++++- 4 files changed, 40 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 207a022156f0..abd3267c2873 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -88,6 +88,7 @@ static void rxe_init_device_param(struct rxe_dev *rxe) rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_RECV; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_WRITE; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_READ; + rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_ATOMIC; rxe->attr.odp_caps.per_transport_caps.rc_odp_caps |= IB_ODP_SUPPORT_SRQ_RECV; } } diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 35c2ccb2fdd9..6ff10b15fa32 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -208,6 +208,9 @@ int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, int access_flags, struct rxe_mr *mr); int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, enum rxe_mr_copy_dir dir); +int rxe_odp_mr_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, + u64 compare, u64 swap_add, u64 *orig_val); + #else /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ static inline int rxe_odp_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, @@ -221,6 +224,12 @@ rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, { return -EOPNOTSUPP; } +static inline int +rxe_odp_mr_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, + u64 compare, u64 swap_add, u64 *orig_val) +{ + return RESPST_ERR_UNSUPPORTED_OPCODE; +} #endif /* CONFIG_INFINIBAND_ON_DEMAND_PAGING */ diff --git a/drivers/infiniband/sw/rxe/rxe_odp.c b/drivers/infiniband/sw/rxe/rxe_odp.c index cbe5d0c3fcc4..194b1fab98b7 100644 --- a/drivers/infiniband/sw/rxe/rxe_odp.c +++ b/drivers/infiniband/sw/rxe/rxe_odp.c @@ -283,3 +283,29 @@ int rxe_odp_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, int length, return err; } + +int rxe_odp_mr_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, + u64 compare, u64 swap_add, u64 *orig_val) +{ + int err; + struct ib_umem_odp *umem_odp = to_ib_umem_odp(mr->umem); + + /* If pagefault is not required, umem mutex will be held until the + * atomic operation completes. Otherwise, it is released and locked + * again in rxe_odp_map_range() to let invalidation handler do its + * work meanwhile. + */ + mutex_lock(&umem_odp->umem_mutex); + + /* Atomic operations manipulate a single char. */ + err = rxe_odp_map_range(mr, iova, sizeof(char), 0); + if (err) + return err; + + err = rxe_mr_do_atomic_op(mr, iova, opcode, compare, + swap_add, orig_val); + + mutex_unlock(&umem_odp->umem_mutex); + + return err; +} diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index b40c47477be3..99ad1dec10c7 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -699,7 +699,10 @@ static enum resp_states atomic_reply(struct rxe_qp *qp, u64 iova = qp->resp.va + qp->resp.offset; if (mr->odp_enabled) - err = RESPST_ERR_UNSUPPORTED_OPCODE; + err = rxe_odp_mr_atomic_op(mr, iova, pkt->opcode, + atmeth_comp(pkt), + atmeth_swap_add(pkt), + &res->atomic.orig_val); else err = rxe_mr_do_atomic_op(mr, iova, pkt->opcode, atmeth_comp(pkt),