From patchwork Fri Nov 3 09:55:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 161213 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp911740vqu; Fri, 3 Nov 2023 02:56:58 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFwPGhqffJpWGqVoL2rFool8AAuPz/PSOKETZ+nASD8wKHesK0V518iRwVaZTxTS9nqv6D/ X-Received: by 2002:a05:6870:11cc:b0:1e9:ba42:76bb with SMTP id 12-20020a05687011cc00b001e9ba4276bbmr19504544oav.39.1699005418284; Fri, 03 Nov 2023 02:56:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699005418; cv=none; d=google.com; s=arc-20160816; b=cI/vVdL2892La7EyxqTN80Fd0fNq7DDdlMCNL7xrCDTLXucFBRw0LsdNPzrUZh8Wkc nFeN9eJ+x5+x6ra/cFXebUSoFYx59ihUtx2jH+fd1a3COukY0w5U6fuZl/o+U0yvBLLz L8ugv7EoXqN2eSxCAvrMs/lGRlAQK+QvF9qTr4wbHVFbaMbZ/rJDOS0i0zEnrI3CZBL2 MCsIIHj9N0yhbGbEs0U2fC7xime6Ga5l1nkjwRel+CQG76DKwkwCc2FDDfI8h+ssCvjS 8xXfK6nlMTOKQg/rfVicq2pNp/BrvtbcR2xWIHVJRCbVUSGYtk0Fzp9TETX9yB7UjmQI DxAw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=+o1EmGeuQJoIYJSMqX00q90yttG1zxKROUc/5RMpfnU=; fh=1j3IhSzqKnT9rnQtXyubbbV4ZVXdy+100vlzKZWpXbA=; b=Yz5NyJQMNA0epgo14us/gGQqfrTmKkU13WA5zCivCcSTRbvWEhysF/ZsuvgmD8NsaT sJsnogj0IV4lhP1uHr6ZO9CdX7mPCm3RlvpGim60a2dm/V/JviiZJxRLNYpNtMRcrZxV ZfKikIUL093+vsYr9JLLjkWEHWv6MYAwVZChR5RKkhCwinRYFNdgLRzsk0vPOxHBhed+ MlqgKSuXcXTJj8/0gIP85uMLsq+Hi39daSBW3SfEp4+36wnUcoh2Ff2MTSocHqYN2mXV VGv6CebFSH3P1IGsuz21e3x/trtFA2pZgk7XIxp6EcvDmscT40RNtsnS6IDLnhQrFczZ XdvQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id bz9-20020a056a02060900b005ac61b76914si1331362pgb.93.2023.11.03.02.56.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 02:56:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 3BDD882E4816; Fri, 3 Nov 2023 02:56:44 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347613AbjKCJ4P (ORCPT + 36 others); Fri, 3 Nov 2023 05:56:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347502AbjKCJ4K (ORCPT ); Fri, 3 Nov 2023 05:56:10 -0400 Received: from esa10.hc1455-7.c3s2.iphmx.com (esa10.hc1455-7.c3s2.iphmx.com [139.138.36.225]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B3FAD6D; Fri, 3 Nov 2023 02:56:02 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10882"; a="126169479" X-IronPort-AV: E=Sophos;i="6.03,273,1694703600"; d="scan'208";a="126169479" Received: from unknown (HELO oym-r3.gw.nic.fujitsu.com) ([210.162.30.91]) by esa10.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2023 18:56:00 +0900 Received: from oym-m4.gw.nic.fujitsu.com (oym-nat-oym-m4.gw.nic.fujitsu.com [192.168.87.61]) by oym-r3.gw.nic.fujitsu.com (Postfix) with ESMTP id 44DB1D6475; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from kws-ab4.gw.nic.fujitsu.com (kws-ab4.gw.nic.fujitsu.com [192.51.206.22]) by oym-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id 35AB1D52C1; Fri, 3 Nov 2023 18:55:57 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab4.gw.nic.fujitsu.com (Postfix) with ESMTP id BA5C51EB1D5; Fri, 3 Nov 2023 18:55:56 +0900 (JST) Received: from FNSTPC.g08.fujitsu.local (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id 3779B1A0072; Fri, 3 Nov 2023 17:55:56 +0800 (CST) From: Li Zhijian To: zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, matsuda-daisuke@fujitsu.com, bvanassche@acm.org, yi.zhang@redhat.com, Li Zhijian Subject: [PATCH RFC V2 1/6] RDMA/rxe: RDMA/rxe: don't allow registering !PAGE_SIZE mr Date: Fri, 3 Nov 2023 17:55:44 +0800 Message-ID: <20231103095549.490744-2-lizhijian@fujitsu.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com> References: <20231103095549.490744-1-lizhijian@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-27974.006 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-27974.006 X-TMASE-Result: 10--4.672600-10.000000 X-TMASE-MatchedRID: rDfjNDBkNcOMCPk/lvFUOt9JA2lmQRNU2FA7wK9mP9cRt1EvyOXA0dyQ dWCH/YgKQZCd2iT0MflsOacVUHP+r+W5YZxMb7aKGqSG/c50XgOaDC/RGGCVV5LFwf1lbHnIgK6 qCGa1Z9fDzSde0I1kYIAy6p60ZV62G2i4y8P2xXndB/CxWTRRu+rAZ8KTspSzxtaprLqF3ntFDM BLjwF3v4FTBTx0LGJ0Spt5K+Tfxbm4lv7CWjd+M51xKvhXp79eP+XouMjD7/vMOuUN/hZxXilzw 1xveMciIcmnZRhVxyrE4HwnSlEuHInEpJmLAFfpC1FNc6oqYVV+3BndfXUhXQ== X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 03 Nov 2023 02:56:44 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781536305180417774 X-GMAIL-MSGID: 1781536305180417774 rxe_set_page() only store one PAGE_SIZE page by the step of page_size. when page_size != PAGE_SIZE, we cannot restore the address with wrong index and page_offset. Let's take a look how current the xarray is being used. 0. offset = iova & (page_size -1); // offset is less than page_size but may not PAGE_SIZE 1. index = (iova - mr.iova) >> page_shift; 2. page = xa_load(&mr->page_list, index); 3. page_va = kmap_local_page(page) // map one page only, that means only memory [page_va, page_va + PAGE_SIZE) is valid for this mapping. 4. memcpy(addr, page_va + offset, bytes); - when page_size > PAGE_SIZE, the offset could be beyond PAGE_SIZE, then page_va + offset may be invalid. - when page_size < PAGE_SIZE, the offset may get lost. Note that this patch will break some ULPs that try to register 4K MR when PAGE_SIZE is not 4K. SRP and nvme over RXE is known to be impacted. Signed-off-by: Li Zhijian --- --- drivers/infiniband/sw/rxe/rxe_mr.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index f54042e9aeb2..3755e530e6dc 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -234,6 +234,12 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, struct rxe_mr *mr = to_rmr(ibmr); unsigned int page_size = mr_page_size(mr); + if (page_size != PAGE_SIZE) { + rxe_err_mr(mr, "Unsupport mr page size %x, expect PAGE_SIZE(%lx)\n", + page_size, PAGE_SIZE); + return -EINVAL; + } + mr->nbuf = 0; mr->page_shift = ilog2(page_size); mr->page_mask = ~((u64)page_size - 1); From patchwork Fri Nov 3 09:55:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 161214 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp911765vqu; Fri, 3 Nov 2023 02:57:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGvbB20L6oZFfIWyMgwouesWS/D/pz1W19LQoKoQNyRV6+o48/PmpD12jnwqQ9b/z3UCjwx X-Received: by 2002:a05:6a21:27a4:b0:170:3e5b:bccd with SMTP id rn36-20020a056a2127a400b001703e5bbccdmr18131034pzb.30.1699005421538; Fri, 03 Nov 2023 02:57:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699005421; cv=none; d=google.com; s=arc-20160816; b=P4Mpab27LsoQNBlYyb5lsFlcAt4Q5IqlaesZHXnTAKrcwFto3IVNXcpPnTyFtMtBHS ytM3pRVeqLSZGeI7i8qyswNNhS61r+7YWlR8ijZIqx0wGruC8WNAvzSP6dCMvEWcCdKY FvD1Ep68HnyzfX49qRJu1Y+j9YJ9G09xE8JL6Ux44YDwIr2PWqAFXg74+gMoAQuHGll7 N0I+KxZk1W3lo55a/uaB7LlSz1VkCC0ena23Yiwrdmz7pv04/ctWhQqGWrZHL9b+6IX8 Cs4482qjFyI5KEVmavTs0LkAgA89CpqKvTP1ruySVToUYgfbJp4Y0scOPVheJH7RGtoU NcDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=DesUwXHy2FodJUmZT7rDuTfQa/IDEVBvWzqXRbHjKk8=; fh=1j3IhSzqKnT9rnQtXyubbbV4ZVXdy+100vlzKZWpXbA=; b=zkkBaYsPxicKkIJdMtb3/2na2YhwuHi1J53x4pQb3S67Tl3UJsZ21xNcAjQoecnWYc dB3uVFzv5Ahwahm9E9U/wR7zengo+HU5JXA0HVE7Es29AppOcTM0aQ2oo6p5Tfzl2YQd K33V9QNtrvh+QJ96vSeiT6x7BaRRDc5fHKjBFO+R7vxkgRSCXq3bqI6v0MbO1KbfXr5y NnF5L8Gb58j65BkKk9ilj9ZLyjMKFY+2b3JOwlvQTY/YxHVY21KWXYp1liFhrAP3iyTH e0vC7FKCK8Iz0FSW5Ltmg0zYNisVALdLPkHlQnqUnsvbHWcZdsS3BMct2sACCxlsuURF cieQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id i19-20020a635413000000b005af44ed3148si1248037pgb.367.2023.11.03.02.57.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 02:57:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 7E8BB811F920; Fri, 3 Nov 2023 02:56:46 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347672AbjKCJ4V (ORCPT + 36 others); Fri, 3 Nov 2023 05:56:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347605AbjKCJ4P (ORCPT ); Fri, 3 Nov 2023 05:56:15 -0400 Received: from esa2.hc1455-7.c3s2.iphmx.com (esa2.hc1455-7.c3s2.iphmx.com [207.54.90.48]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FE40D70; Fri, 3 Nov 2023 02:56:02 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10882"; a="138469456" X-IronPort-AV: E=Sophos;i="6.03,273,1694703600"; d="scan'208";a="138469456" Received: from unknown (HELO yto-r4.gw.nic.fujitsu.com) ([218.44.52.220]) by esa2.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2023 18:56:01 +0900 Received: from yto-m4.gw.nic.fujitsu.com (yto-nat-yto-m4.gw.nic.fujitsu.com [192.168.83.67]) by yto-r4.gw.nic.fujitsu.com (Postfix) with ESMTP id 762B7D3EAC; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from kws-ab4.gw.nic.fujitsu.com (kws-ab4.gw.nic.fujitsu.com [192.51.206.22]) by yto-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id A9C5AC4A16; Fri, 3 Nov 2023 18:55:57 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab4.gw.nic.fujitsu.com (Postfix) with ESMTP id 348F9E36BA; Fri, 3 Nov 2023 18:55:57 +0900 (JST) Received: from FNSTPC.g08.fujitsu.local (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id 9ABBE1A006F; Fri, 3 Nov 2023 17:55:56 +0800 (CST) From: Li Zhijian To: zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, matsuda-daisuke@fujitsu.com, bvanassche@acm.org, yi.zhang@redhat.com, Li Zhijian Subject: [PATCH RFC V2 2/6] RDMA/rxe: set RXE_PAGE_SIZE_CAP to PAGE_SIZE Date: Fri, 3 Nov 2023 17:55:45 +0800 Message-ID: <20231103095549.490744-3-lizhijian@fujitsu.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com> References: <20231103095549.490744-1-lizhijian@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-27974.006 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-27974.006 X-TMASE-Result: 10--5.753700-10.000000 X-TMASE-MatchedRID: S8AxUE03d7BTorztX3wKRAmyVrMCuJ9SwTlc9CcHMZerwqxtE531VCzy bVqWyY2ND+LZhHM9RwrRpRbL7Yjxh7QIlEvYJcRNfa4B0jQGGeoogaQhRNNEvpsoi2XrUn/J8m+ hzBStanvIM9mETCO70yAHAopEd76veDyIM8J5kE2z4CsjXTAQJnU5kupDVrVP2Pzvl4G8qgDM1X RxVNdnKQ== X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Fri, 03 Nov 2023 02:56:46 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781536308787332415 X-GMAIL-MSGID: 1781536308787332415 RXE_PAGE_SIZE_CAP means the MR page size supported by RXE. However in current RXE implementation, only PAGE_SIZE MR works well. So change it to PAGE_SIZE only. ULPs such as SRP calculating the page size according to this attribute get worked again with this change. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h index d2f57ead78ad..b1cf1e1c0ce1 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -38,7 +38,7 @@ static inline enum ib_mtu eth_mtu_int_to_enum(int mtu) /* default/initial rxe device parameter settings */ enum rxe_device_param { RXE_MAX_MR_SIZE = -1ull, - RXE_PAGE_SIZE_CAP = 0xfffff000, + RXE_PAGE_SIZE_CAP = PAGE_SIZE, RXE_MAX_QP_WR = DEFAULT_MAX_VALUE, RXE_DEVICE_CAP_FLAGS = IB_DEVICE_BAD_PKEY_CNTR | IB_DEVICE_BAD_QKEY_CNTR From patchwork Fri Nov 3 09:55:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 161215 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp911862vqu; Fri, 3 Nov 2023 02:57:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGFMJMv+GrbqURvI+PWcCHQIzLqK2M8ESHG/JTe7+4wsug0ksNRjT62M240PBh6MUSA5zDs X-Received: by 2002:aa7:d5d2:0:b0:53f:a526:779 with SMTP id d18-20020aa7d5d2000000b0053fa5260779mr16968111eds.12.1699005445081; Fri, 03 Nov 2023 02:57:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699005444; cv=none; d=google.com; s=arc-20160816; b=MRoG1spfOYC4lHuPxsiF+Gk9tEWgT+XieoTNN08VrKR7pL+Oxa9pPaiD8LFoPrfdJC Afin2wC4wQHsZXbH2taqli9t0+FHfF7VDsBFXGk53Je9IAyZISK+hsPCUeyvgSIpI//b ZN0kwbub/ZUZfWd/yeLcrBEcj7h/0SEC/rMh/BG4tNgEPvvHPYIwE3ChO7nFw3DJfoM7 ezykPbZgUv+NYKacG0KUtyLCBjQ+EVKv7SoJDu94HQyIve4n+JnuKRZ/WSFmKhpLF01d p6QvRry/fxb1T46fZ85feNcUDjp0K8BauFmH3LcYnZFhDgWYLROEt3G4DoIUQx5feJjD tjjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=kEITQmZS0u5NIEwKZnSaUN+/Z+pRhtH7cVG6b1gNYOI=; fh=1j3IhSzqKnT9rnQtXyubbbV4ZVXdy+100vlzKZWpXbA=; b=GIbTZmnpbXYve43yvO758gJD1PZaPqh+C2XoNfyv0JI3rsJnbf8SeaEgEsOJKB/XpT K6n1KVwB5MMEeaQOT0AXpHpeYc+O2gLz86o+i3qGBEMP1yNrTB5CWPDj3mTuVyzH8WYq PP7gXEHdBG+Ht7nlvDVadEtY5XgPBYPpo6cLF+qs1wnfIRRprhZSoLBaQsnO376qbvww GZj9TBnLUUpuyjP4UUUWkp6mDDcAk+n+oKkCi6WV/EnNqMocO8guDpSsnH1NeCGJGDlk iyByUE9AJ5qyqCwWZY/0fFyusGM8NYUDVAozo3fMfrcJJMhao0ya+AcaBFeQmK1kBXmF xc9w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id t22-20020a508d56000000b0053eef7c1a6asi772880edt.482.2023.11.03.02.57.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 02:57:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 34E9F82E4827; Fri, 3 Nov 2023 02:57:10 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347656AbjKCJ4S (ORCPT + 36 others); Fri, 3 Nov 2023 05:56:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347526AbjKCJ4M (ORCPT ); Fri, 3 Nov 2023 05:56:12 -0400 Received: from esa12.hc1455-7.c3s2.iphmx.com (esa12.hc1455-7.c3s2.iphmx.com [139.138.37.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7485D77; Fri, 3 Nov 2023 02:56:03 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10882"; a="117864670" X-IronPort-AV: E=Sophos;i="6.03,273,1694703600"; d="scan'208";a="117864670" Received: from unknown (HELO oym-r3.gw.nic.fujitsu.com) ([210.162.30.91]) by esa12.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2023 18:56:00 +0900 Received: from oym-m4.gw.nic.fujitsu.com (oym-nat-oym-m4.gw.nic.fujitsu.com [192.168.87.61]) by oym-r3.gw.nic.fujitsu.com (Postfix) with ESMTP id D783ED6475; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from kws-ab3.gw.nic.fujitsu.com (kws-ab3.gw.nic.fujitsu.com [192.51.206.21]) by oym-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id 0EFF8D5D15; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab3.gw.nic.fujitsu.com (Postfix) with ESMTP id A4D3B200501B1; Fri, 3 Nov 2023 18:55:57 +0900 (JST) Received: from FNSTPC.g08.fujitsu.local (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id 15FD81A0072; Fri, 3 Nov 2023 17:55:57 +0800 (CST) From: Li Zhijian To: zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, matsuda-daisuke@fujitsu.com, bvanassche@acm.org, yi.zhang@redhat.com, Li Zhijian Subject: [PATCH RFC V2 3/6] RDMA/rxe: remove unused rxe_mr.page_shift Date: Fri, 3 Nov 2023 17:55:46 +0800 Message-ID: <20231103095549.490744-4-lizhijian@fujitsu.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com> References: <20231103095549.490744-1-lizhijian@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-27974.006 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-27974.006 X-TMASE-Result: 10--2.244700-10.000000 X-TMASE-MatchedRID: 6q73Zcfo6Jq1UOlz1sLXchF4zyLyne+ATJDl9FKHbrlCw17cr3HRB+K5 fhRxt1aA92kNXgH07FbsbC/Vw6Dhkx8TzIzimOwP/hxxPCwzUoPEQdG7H66TyHEqm8QYBtMOZsY Fy96KWWKZqgH/izpwuaKHvyV1Zkh/LCZafsjE5s7ah7Q7AITJYLpgs6aeGU9hk8OVAzuAhGHiC9 thELD2+yGkZFGhX/A+W4wbpXTb5DJKKve1kh3RY37qSWrndbmQn0bOriG5BVc= X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 03 Nov 2023 02:57:10 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781536333265084931 X-GMAIL-MSGID: 1781536333265084931 it's assigned but never used. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_mr.c | 1 - drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - 2 files changed, 2 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index 3755e530e6dc..bbfedcd8d2cb 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -243,7 +243,6 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, mr->nbuf = 0; mr->page_shift = ilog2(page_size); mr->page_mask = ~((u64)page_size - 1); - mr->page_offset = mr->ibmr.iova & (page_size - 1); return ib_sg_to_pages(ibmr, sgl, sg_nents, sg_offset, rxe_set_page); } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index ccb9d19ffe8a..11647e976282 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -309,7 +309,6 @@ struct rxe_mr { int access; atomic_t num_mw; - unsigned int page_offset; unsigned int page_shift; u64 page_mask; From patchwork Fri Nov 3 09:55:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 161217 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp912105vqu; Fri, 3 Nov 2023 02:58:19 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGq3h52a+PoRIvP/Q43SX1AuN4Q9LLSWdeT5+tnOiT7NiGrLA1zpeQw0QlIyGVaIT+jG02l X-Received: by 2002:a17:90a:17ca:b0:280:cc28:9d2b with SMTP id q68-20020a17090a17ca00b00280cc289d2bmr2775538pja.0.1699005499184; Fri, 03 Nov 2023 02:58:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699005499; cv=none; d=google.com; s=arc-20160816; b=THp3HVpsKF44j5qiejOWQEiy0PYrH5a7ryo+epH5lnTiWn38s8tv6sZB08UmIjcB4x X4jKHYGE0QeV7voNC78NnHVeRBfjPYz41ka/B6uB4d38joPt/8WcvyCeutFZeCaJ6sbj LGbEoVZtfJ760nZyIHyPLWVt53JhFN7wWPHnx83EXrfNu6/UKVIz2YOdDYdg534LiQnQ Oe1CMFWM89IDeZZUOyYqv5myTXuYNjL5ShPS1pu6krIlK1SHATzuF5zTc2oOoC+WdntR /zi6KKMYu542KVs1aTVgThrkE0l8+dEznXLzT5tV0WmGmj9js/+XPWo6sYidlulO9TBK pYng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=q05Jl7Tpyp8qBSq3So87+kzHamMARMFtKj5EWo2Pzl0=; fh=1j3IhSzqKnT9rnQtXyubbbV4ZVXdy+100vlzKZWpXbA=; b=y+G40JlVSgmfDQn4d0yT2HwA0OmYJQpEZTs5g57VbOO+dS1nkNxmGlJTWpGflG0kYh 0B+O8dXROUNG/G4/i2BP/XZne6H6H4mImglKU1boUNWK8YgP38Lg4HYtqcduVpF9Pmyn gFQqBZkWPZRkQbtZOkkJdj/F5/izSEP93+bdLgp43PyAjnUOeX4oVjh+z5M0dBwZaJSx KsxkJa59hJ46MrUL5O37eCmdQmufXCUsjxdDb77JAp8VxVCw2/ZOyx414ci+D7C9klM4 uAFVfyTLigRt15Ef2fQzt+u1z7UhItlgECcL5MfBtm/RjzC88xRV9s7e32TOoH9wBpmr WIkw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id pi11-20020a17090b1e4b00b00268293118efsi1307733pjb.47.2023.11.03.02.58.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 02:58:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id E935682E4818; Fri, 3 Nov 2023 02:58:13 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347390AbjKCJ55 (ORCPT + 36 others); Fri, 3 Nov 2023 05:57:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347425AbjKCJ54 (ORCPT ); Fri, 3 Nov 2023 05:57:56 -0400 X-Greylist: delayed 64 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Fri, 03 Nov 2023 02:57:09 PDT Received: from esa5.hc1455-7.c3s2.iphmx.com (esa5.hc1455-7.c3s2.iphmx.com [68.232.139.130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38BE719BC; Fri, 3 Nov 2023 02:57:08 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10882"; a="137953602" X-IronPort-AV: E=Sophos;i="6.03,273,1694703600"; d="scan'208";a="137953602" Received: from unknown (HELO yto-r4.gw.nic.fujitsu.com) ([218.44.52.220]) by esa5.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2023 18:56:02 +0900 Received: from yto-m4.gw.nic.fujitsu.com (yto-nat-yto-m4.gw.nic.fujitsu.com [192.168.83.67]) by yto-r4.gw.nic.fujitsu.com (Postfix) with ESMTP id 5147ED3EAC; Fri, 3 Nov 2023 18:55:59 +0900 (JST) Received: from kws-ab4.gw.nic.fujitsu.com (kws-ab4.gw.nic.fujitsu.com [192.51.206.22]) by yto-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id 817E7C8BFE; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab4.gw.nic.fujitsu.com (Postfix) with ESMTP id 15438E5E53; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from FNSTPC.g08.fujitsu.local (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id 83B731A006F; Fri, 3 Nov 2023 17:55:57 +0800 (CST) From: Li Zhijian To: zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, matsuda-daisuke@fujitsu.com, bvanassche@acm.org, yi.zhang@redhat.com, Li Zhijian Subject: [PATCH RFC V2 4/6] RDMA/rxe: Use PAGE_SIZE and PAGE_SHIFT to extract address from page_list Date: Fri, 3 Nov 2023 17:55:47 +0800 Message-ID: <20231103095549.490744-5-lizhijian@fujitsu.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com> References: <20231103095549.490744-1-lizhijian@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-27974.006 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-27974.006 X-TMASE-Result: 10--5.509000-10.000000 X-TMASE-MatchedRID: VULvqRy0JHoHl9LdO3B7p4Olbll4OMtkyiKgKtIyB4qe38zXnNg9Q2ZY /RdXrUKNkAkq19PK0vC12HagvbwDji/7QU2czuUNA9lly13c/gFb4dXBMpIkhqoDeu6wu7bqE0o 8W+GU3zCqyptwK2nPv7Qj8USEuwVGZSqIGAn5t/jknMSTG9lH+MZiegX8JN5lmyiLZetSf8mfop 0ytGwvXiq2rl3dzGQ1dQgZJSoREhTm9DV/P8jWkbzOOaYWQHGCZ3g1SwoJbflW8/kkz9GZ0g== X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 03 Nov 2023 02:58:14 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781536390115298419 X-GMAIL-MSGID: 1781536390115298419 As we said in previous commit, page_list only stores PAGE_SIZE page, so when we extract an address from the page_list, we should use PAGE_SIZE and PAGE_SHIFT instead of the ibmr.page_size. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_mr.c | 42 +++++++++------------------ drivers/infiniband/sw/rxe/rxe_verbs.h | 5 ---- 2 files changed, 14 insertions(+), 33 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index bbfedcd8d2cb..d39c02f0c51e 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -72,16 +72,6 @@ void rxe_mr_init_dma(int access, struct rxe_mr *mr) mr->ibmr.type = IB_MR_TYPE_DMA; } -static unsigned long rxe_mr_iova_to_index(struct rxe_mr *mr, u64 iova) -{ - return (iova >> mr->page_shift) - (mr->ibmr.iova >> mr->page_shift); -} - -static unsigned long rxe_mr_iova_to_page_offset(struct rxe_mr *mr, u64 iova) -{ - return iova & (mr_page_size(mr) - 1); -} - static bool is_pmem_page(struct page *pg) { unsigned long paddr = page_to_phys(pg); @@ -232,17 +222,16 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, int sg_nents, unsigned int *sg_offset) { struct rxe_mr *mr = to_rmr(ibmr); - unsigned int page_size = mr_page_size(mr); - if (page_size != PAGE_SIZE) { + if (ibmr->page_size != PAGE_SIZE) { rxe_err_mr(mr, "Unsupport mr page size %x, expect PAGE_SIZE(%lx)\n", - page_size, PAGE_SIZE); + ibmr->page_size, PAGE_SIZE); return -EINVAL; } mr->nbuf = 0; - mr->page_shift = ilog2(page_size); - mr->page_mask = ~((u64)page_size - 1); + mr->page_shift = PAGE_SHIFT; + mr->page_mask = PAGE_MASK; return ib_sg_to_pages(ibmr, sgl, sg_nents, sg_offset, rxe_set_page); } @@ -250,8 +239,8 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, static int rxe_mr_copy_xarray(struct rxe_mr *mr, u64 iova, void *addr, unsigned int length, enum rxe_mr_copy_dir dir) { - unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova); - unsigned long index = rxe_mr_iova_to_index(mr, iova); + unsigned int page_offset = iova & (PAGE_SIZE - 1); + unsigned long index = (iova - mr->ibmr.iova) >> PAGE_SHIFT; unsigned int bytes; struct page *page; void *va; @@ -261,8 +250,7 @@ static int rxe_mr_copy_xarray(struct rxe_mr *mr, u64 iova, void *addr, if (!page) return -EFAULT; - bytes = min_t(unsigned int, length, - mr_page_size(mr) - page_offset); + bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset); va = kmap_local_page(page); if (dir == RXE_FROM_MR_OBJ) memcpy(addr, va + page_offset, bytes); @@ -450,14 +438,12 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) return err; while (length > 0) { - index = rxe_mr_iova_to_index(mr, iova); + index = (iova - mr->ibmr.iova) >> PAGE_SHIFT; page = xa_load(&mr->page_list, index); - page_offset = rxe_mr_iova_to_page_offset(mr, iova); + page_offset = iova & (PAGE_SIZE - 1); if (!page) return -EFAULT; - bytes = min_t(unsigned int, length, - mr_page_size(mr) - page_offset); - + bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset); va = kmap_local_page(page); arch_wb_cache_pmem(va + page_offset, bytes); kunmap_local(va); @@ -498,8 +484,8 @@ int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, rxe_dbg_mr(mr, "iova out of range"); return RESPST_ERR_RKEY_VIOLATION; } - page_offset = rxe_mr_iova_to_page_offset(mr, iova); - index = rxe_mr_iova_to_index(mr, iova); + page_offset = iova & (PAGE_SIZE - 1); + index = (iova - mr->ibmr.iova) >> PAGE_SHIFT; page = xa_load(&mr->page_list, index); if (!page) return RESPST_ERR_RKEY_VIOLATION; @@ -556,8 +542,8 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) rxe_dbg_mr(mr, "iova out of range"); return RESPST_ERR_RKEY_VIOLATION; } - page_offset = rxe_mr_iova_to_page_offset(mr, iova); - index = rxe_mr_iova_to_index(mr, iova); + page_offset = iova & (PAGE_SIZE - 1); + index = (iova - mr->ibmr.iova) >> PAGE_SHIFT; page = xa_load(&mr->page_list, index); if (!page) return RESPST_ERR_RKEY_VIOLATION; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 11647e976282..ccc75f8c0985 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -318,11 +318,6 @@ struct rxe_mr { struct xarray page_list; }; -static inline unsigned int mr_page_size(struct rxe_mr *mr) -{ - return mr ? mr->ibmr.page_size : PAGE_SIZE; -} - enum rxe_mw_state { RXE_MW_STATE_INVALID = RXE_MR_STATE_INVALID, RXE_MW_STATE_FREE = RXE_MR_STATE_FREE, From patchwork Fri Nov 3 09:55:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 161216 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp911992vqu; Fri, 3 Nov 2023 02:57:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEXLuvN4I/wv8IF/VgEZjiK5oIbcDiHNmdguT2nznRwUiavhhK8K+7yTMBtjc1CEBv/4860 X-Received: by 2002:a17:907:e8b:b0:9ba:65e:7529 with SMTP id ho11-20020a1709070e8b00b009ba065e7529mr7123045ejc.68.1699005476682; Fri, 03 Nov 2023 02:57:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699005476; cv=none; d=google.com; s=arc-20160816; b=kAgV18oGI6PF7BzXaTX/CLycLcYcK/UJfpHC/7EKc51H1AAw1K/vYvI/h79fQb37Si YBcNQWEQ3shF1T8gI7hxP/b60vxQfccmqi1cJXxz/fsbsLnrpTw5fO7a4AUx57Ql9leY sTOd+i3HGQUQcDuMPVNTaXxjYE29JtmgcNmNbtsVeHVu4pOy2Ehq06d7MZjyjZ6CokhQ xXsBb46yby4aIU//P5nZ1GnJzSrjr0iHjuFTgN9496zNWrrWNsyQ06qA6YeIny5yvKrN RoZpWkQvxnU0WBRv5I9BSarqGAULWy8STkAA+xf02YPwuMYJvRxz1Rm7WVOOVFcXW7Jp HERQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=OV1HCcRZygBGuDGZ1rsiAu5WuFXrenWq2Pq48t0SPso=; fh=1j3IhSzqKnT9rnQtXyubbbV4ZVXdy+100vlzKZWpXbA=; b=rAiR3KzoA3R/MovUu37uY74dxDTwfVP4YsIS0FY7hCnAfsX4g2BB6iQSqrSNii/dc9 8yUxlvrpbiJTVyzhAtlI7IG2XDH1RvOEeXUqeX481aqjPka06OfZAxiGdLcV9j+fE0Ap VbafBq5G6avHI6OOiXpGXRInnjUvxC40Q6kEqfg1N+VXvoGR1SPo76niMd3crzljEeJZ pKxR9VsOuporVooOp/jgltEqVI30gwaYHwRh4G+xc7rRRvD1RvQLNLnb8648qoqghbcb Df9fI1FNaazH6KBeHKE14mlxuTZJWQhVCpqNx4SXWH+JnDpIJ6JMcdtIhIxBZeUcyxx8 u7fQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id hz6-20020a1709072ce600b009a5bbaad78bsi730266ejc.985.2023.11.03.02.57.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 02:57:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 1C99D82E4813; Fri, 3 Nov 2023 02:57:53 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347526AbjKCJ43 (ORCPT + 36 others); Fri, 3 Nov 2023 05:56:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347655AbjKCJ4S (ORCPT ); Fri, 3 Nov 2023 05:56:18 -0400 Received: from esa1.hc1455-7.c3s2.iphmx.com (esa1.hc1455-7.c3s2.iphmx.com [207.54.90.47]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47B6D10C4; Fri, 3 Nov 2023 02:56:04 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10882"; a="138283629" X-IronPort-AV: E=Sophos;i="6.03,273,1694703600"; d="scan'208";a="138283629" Received: from unknown (HELO yto-r2.gw.nic.fujitsu.com) ([218.44.52.218]) by esa1.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2023 18:56:02 +0900 Received: from yto-m4.gw.nic.fujitsu.com (yto-nat-yto-m4.gw.nic.fujitsu.com [192.168.83.67]) by yto-r2.gw.nic.fujitsu.com (Postfix) with ESMTP id C0533D6186; Fri, 3 Nov 2023 18:55:59 +0900 (JST) Received: from kws-ab3.gw.nic.fujitsu.com (kws-ab3.gw.nic.fujitsu.com [192.51.206.21]) by yto-m4.gw.nic.fujitsu.com (Postfix) with ESMTP id 0C0FEC4A00; Fri, 3 Nov 2023 18:55:59 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab3.gw.nic.fujitsu.com (Postfix) with ESMTP id 8905F2005019A; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from FNSTPC.g08.fujitsu.local (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id E97941A0072; Fri, 3 Nov 2023 17:55:57 +0800 (CST) From: Li Zhijian To: zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, matsuda-daisuke@fujitsu.com, bvanassche@acm.org, yi.zhang@redhat.com, Li Zhijian Subject: [PATCH RFC V2 5/6] RDMA/rxe: cleanup rxe_mr.{page_size,page_shift} Date: Fri, 3 Nov 2023 17:55:48 +0800 Message-ID: <20231103095549.490744-6-lizhijian@fujitsu.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com> References: <20231103095549.490744-1-lizhijian@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-27974.006 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-27974.006 X-TMASE-Result: 10--2.085100-10.000000 X-TMASE-MatchedRID: 4vuDYoAhWgkEAXVIFoPmpMfIK2dgWYWGCZa9cSpBObnAuQ0xDMaXkH4q tYI9sRE/lTJXKqh1ne2XNWQa4uM6kJcFdomgH0lnFEUknJ/kEl5lVdRvgpNpe/oLR4+zsDTtuVf c976pNypHlescvaZnOWnt3jHkT+4bhPISvaLQ542U/6gnt9otdY9p61D8mgeNW27Gp3VFEiCpMY 9k1M7dHstkdSYjBYPP5H+vESxOZUGGk+xUaqdMDwHEKwHwYevbwUSxXh+jiUgkww/gwY7hMA== X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Fri, 03 Nov 2023 02:57:53 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781536366464344356 X-GMAIL-MSGID: 1781536366464344356 This 2 elements were believed to be designed for extracting address from the page_list before. But now we use PAGE_SIZE and PAGE_SHIFT directly, so we can drop it. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_mr.c | 4 ---- drivers/infiniband/sw/rxe/rxe_verbs.h | 3 --- 2 files changed, 7 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index d39c02f0c51e..a038133e1322 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -59,8 +59,6 @@ static void rxe_mr_init(int access, struct rxe_mr *mr) mr->access = access; mr->ibmr.page_size = PAGE_SIZE; - mr->page_mask = PAGE_MASK; - mr->page_shift = PAGE_SHIFT; mr->state = RXE_MR_STATE_INVALID; } @@ -230,8 +228,6 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, } mr->nbuf = 0; - mr->page_shift = PAGE_SHIFT; - mr->page_mask = PAGE_MASK; return ib_sg_to_pages(ibmr, sgl, sg_nents, sg_offset, rxe_set_page); } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index ccc75f8c0985..ef813560b0ab 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -309,9 +309,6 @@ struct rxe_mr { int access; atomic_t num_mw; - unsigned int page_shift; - u64 page_mask; - u32 num_buf; u32 nbuf; From patchwork Fri Nov 3 09:55:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhijian Li (Fujitsu)" X-Patchwork-Id: 161218 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:8f47:0:b0:403:3b70:6f57 with SMTP id j7csp912219vqu; Fri, 3 Nov 2023 02:58:42 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEifYCXqcPzk+UBg2jSz61zCvte3MXkAw3HedU41KR6nqZs8CZDtJ0kIjGudAAptTFvoILq X-Received: by 2002:a17:906:9597:b0:9d1:92bb:ce74 with SMTP id r23-20020a170906959700b009d192bbce74mr4451148ejx.38.1699005522504; Fri, 03 Nov 2023 02:58:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1699005522; cv=none; d=google.com; s=arc-20160816; b=fuUD7NCrs7ZAZZQ4GniG+lgD29pwDn0oYvlhM64r3Oe97W0J7loXQQVfOzUfHeF2li qDK8/Jdcj/y4l8NQiNrCk/i6if0tGKqh8E6tOYftVXK4xQRT5PzgYmPEGHK+Tusij6lR JUR+A2kkgKg+PJgDOzKcFR5Nk/wtl0pUBDCn/yuz5Ov8scp/Y+mELvrB2aWFo9Iq/sxP ynKJnV2pvbYFIdmdeNEK55PP4M+JMwbY1yJLOtEU0ZRhpWntol8FA4f2jdghRHEH3QIF Bxtx5yFf6DrRJGUfn61i4iCpO3Iz5TrmydRHy2OcjV+ahebx/xem45pN2nq5dk7/+eFq qbsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=6mX7YA6Ye1EXBTra+AVxF/fMCUUojB3RI0NAq3JH/Ig=; fh=1j3IhSzqKnT9rnQtXyubbbV4ZVXdy+100vlzKZWpXbA=; b=Z7bh77R0jQaa0ikREvgGh1oui9+oNBA24Y4H7ik78eGnTpTmGNSFIc6kiyowqnIBeQ 518pkI3QPom6cIs2+Do76e7T2RbFO5TGpkstYKL4DQJSM5tbvzfUxe2Z8HftI7FHHvkD XxyfwLM/3YS6PWQ7awVknWIBIN6ezbBNmoHUCC08RGsi8xxMpNBPb9uFwYBN+U2dj0+a CbUOLPbv5OL/VEtsgUH+gHY1QQIvg14TfuofcklRBxyPd8+4TiygYLfavDAUQNGtl1YE UzrpvuNAvD8yV3eYui3n4l0Fy6fQTY8bFSPl4hmwj/CEbsbyYi5wXBkHRhaLFjhIj6dd TowQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id 25-20020a170906025900b009945658c5b9si747619ejl.811.2023.11.03.02.58.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Nov 2023 02:58:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=fujitsu.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 1519C807E66E; Fri, 3 Nov 2023 02:58:39 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1347613AbjKCJ63 (ORCPT + 36 others); Fri, 3 Nov 2023 05:58:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347612AbjKCJ61 (ORCPT ); Fri, 3 Nov 2023 05:58:27 -0400 Received: from esa5.hc1455-7.c3s2.iphmx.com (esa5.hc1455-7.c3s2.iphmx.com [68.232.139.130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4119D71; Fri, 3 Nov 2023 02:57:55 -0700 (PDT) X-IronPort-AV: E=McAfee;i="6600,9927,10882"; a="137953603" X-IronPort-AV: E=Sophos;i="6.03,273,1694703600"; d="scan'208";a="137953603" Received: from unknown (HELO oym-r4.gw.nic.fujitsu.com) ([210.162.30.92]) by esa5.hc1455-7.c3s2.iphmx.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2023 18:56:03 +0900 Received: from oym-m1.gw.nic.fujitsu.com (oym-nat-oym-m1.gw.nic.fujitsu.com [192.168.87.58]) by oym-r4.gw.nic.fujitsu.com (Postfix) with ESMTP id 31184D9D8F; Fri, 3 Nov 2023 18:56:00 +0900 (JST) Received: from kws-ab3.gw.nic.fujitsu.com (kws-ab3.gw.nic.fujitsu.com [192.51.206.21]) by oym-m1.gw.nic.fujitsu.com (Postfix) with ESMTP id 5CBB5D88D6; Fri, 3 Nov 2023 18:55:59 +0900 (JST) Received: from edo.cn.fujitsu.com (edo.cn.fujitsu.com [10.167.33.5]) by kws-ab3.gw.nic.fujitsu.com (Postfix) with ESMTP id DB3F4200501B0; Fri, 3 Nov 2023 18:55:58 +0900 (JST) Received: from FNSTPC.g08.fujitsu.local (unknown [10.167.226.45]) by edo.cn.fujitsu.com (Postfix) with ESMTP id 597851A006F; Fri, 3 Nov 2023 17:55:58 +0800 (CST) From: Li Zhijian To: zyjzyj2000@gmail.com, jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org, rpearsonhpe@gmail.com, matsuda-daisuke@fujitsu.com, bvanassche@acm.org, yi.zhang@redhat.com, Li Zhijian Subject: [PATCH RFC V2 6/6] RDMA/rxe: Support PAGE_SIZE aligned MR Date: Fri, 3 Nov 2023 17:55:49 +0800 Message-ID: <20231103095549.490744-7-lizhijian@fujitsu.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231103095549.490744-1-lizhijian@fujitsu.com> References: <20231103095549.490744-1-lizhijian@fujitsu.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-TM-AS-Product-Ver: IMSS-9.1.0.1417-9.0.0.1002-27974.006 X-TM-AS-User-Approved-Sender: Yes X-TMASE-Version: IMSS-9.1.0.1417-9.0.1002-27974.006 X-TMASE-Result: 10--3.626700-10.000000 X-TMASE-MatchedRID: uf8KE2NGjd9Bt63ZJXZ1oU7nLUqYrlslFIuBIWrdOePfUZT83lbkEFI6 c0LB6zRTgnKDgRN2gHJJNhitn2ZCFKimPM07Al9SnXdphQTSK/IO4jfa+nI3P5soi2XrUn/Jn6K dMrRsL14qtq5d3cxkNUZzh5+urigLQuzJLcC9/jgLiXnACqUA9wm53Cjx6YN5yPyw704COky5J1 x/8WkKqRoeB7ARQTeWvE7sDZ3+mHhXOEE2otkM4RiKAP5Y7TkhFcUQf3Yp/ridO0/GUi4gFb0fO PzpgdcEKeJ/HkAZ8Is= X-TMASE-SNAP-Result: 1.821001.0001-0-1-22:0,33:0,34:0-0 X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 03 Nov 2023 02:58:39 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1781536415077911983 X-GMAIL-MSGID: 1781536415077911983 In order to support PAGE_SIZE aligned MR, rxe_map_mr_sg() should be able to split a large buffer to N * page entry into the xarray page_list. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_mr.c | 39 +++++++++++++++++++++++++----- 1 file changed, 33 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index a038133e1322..3761740af986 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -193,9 +193,8 @@ int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr) return err; } -static int rxe_set_page(struct ib_mr *ibmr, u64 dma_addr) +static int rxe_store_page(struct rxe_mr *mr, u64 dma_addr) { - struct rxe_mr *mr = to_rmr(ibmr); struct page *page = ib_virt_dma_to_page(dma_addr); bool persistent = !!(mr->access & IB_ACCESS_FLUSH_PERSISTENT); int err; @@ -216,20 +215,48 @@ static int rxe_set_page(struct ib_mr *ibmr, u64 dma_addr) return 0; } +static int rxe_set_page(struct ib_mr *base_mr, u64 buf_addr) +{ + return 0; +} + int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, - int sg_nents, unsigned int *sg_offset) + int sg_nents, unsigned int *sg_offset_p) { struct rxe_mr *mr = to_rmr(ibmr); + struct scatterlist *sg; + unsigned int sg_offset = sg_offset_p ? *sg_offset_p : 0; + int i; - if (ibmr->page_size != PAGE_SIZE) { - rxe_err_mr(mr, "Unsupport mr page size %x, expect PAGE_SIZE(%lx)\n", + if (!IS_ALIGNED(ibmr->page_size, PAGE_SIZE)) { + rxe_err_mr(mr, "Misaligned page size %x, expect PAGE_SIZE(%lx) aligned\n", ibmr->page_size, PAGE_SIZE); return -EINVAL; } mr->nbuf = 0; - return ib_sg_to_pages(ibmr, sgl, sg_nents, sg_offset, rxe_set_page); + for_each_sg(sgl, sg, sg_nents, i) { + u64 dma_addr = sg_dma_address(sg) + sg_offset; + unsigned int dma_len = sg_dma_len(sg) - sg_offset; + u64 end_dma_addr = dma_addr + dma_len; + u64 page_addr = dma_addr & PAGE_MASK; + + if (sg_dma_len(sg) == 0) { + rxe_dbg_mr(mr, "empty SGE\n"); + return -EINVAL; + } + do { + int ret = rxe_store_page(mr, page_addr); + if (ret) + return ret; + + page_addr += PAGE_SIZE; + } while (page_addr < end_dma_addr); + sg_offset = 0; + } + + return ib_sg_to_pages(ibmr, sgl, sg_nents, sg_offset_p, rxe_set_page); } static int rxe_mr_copy_xarray(struct rxe_mr *mr, u64 iova, void *addr,