From patchwork Fri Dec 8 00:52:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 175505 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp5163505vqy; Thu, 7 Dec 2023 16:54:09 -0800 (PST) X-Google-Smtp-Source: AGHT+IFmuw1Dss3lmBvXN9FW3sD+s/kKPJEbkkzMJf3CxTFCGrpw+KNwKusM9f5CqLq0Vu/68cOx X-Received: by 2002:a17:902:c943:b0:1d0:7535:8b94 with SMTP id i3-20020a170902c94300b001d075358b94mr4065874pla.97.1701996849342; Thu, 07 Dec 2023 16:54:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701996849; cv=none; d=google.com; s=arc-20160816; b=kN0wIvNGqz8kjzDJQt9ODnVjOdU8NwiK0JFjY0wJrHnwoHxIZLGQ6+TB273OBRkslA StskB7OlQDj3zqgEYG2U9hu9IP9E4EkPDww42QKJw6C2jTlgrV0KLKgdE9JJ/5S38PgL 8hDeJKUP9u/Zl6KMGGpFdPrKh9vC1P18pib5zwj5hutCBDaeV315inM7HafQMkavFtiv cZqzrnPvWsiOrRl7ivYm97Vv6kq1I6/IrH2UO26JFgblGqoTs/Fjk/wUC5msdYgNYyVY tv59ehQEyFG6RlsihztO4b6kzBW5Ysjoy1yLtMscwzVN28dVbmWH+aD4weugtjHFXeaT z8Ug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=MQeyU/Au+VHoLC/CemSxB4TP3xyLB5qWh0DnEWyEst8=; fh=spYIC3Yl69sWtCLqpfRLMGcxeD5ncKplqDHTlWXfkQk=; b=DG/lnECqv2VNc8nFmC6Bul9upujMtEgUOT8jD86bXxEFOmvHVOfl8TIdPpXhRZvU3t dcqs42fvSDCK/3Deug61iRusm0oRI8heonIumTMz1Gki8lt1nlMtFJrfkZrUYoUD6QCs f5qOhSsAbtOatepgqPmp8+8hPCX2Snw0EnL2XFNYfkjLgXR8F9f3y4u0te0/wuW7H+LP HC3IHLA8v1npzlLU3Pj2dQvPJcyrhuIxEYykX8HU7EN44Cp+KDyPdbA2E/BJZsfULydT EKHuP/Vb3RA/lUSd+BfaqJYJYR2gkLso3DLi4MFAkeHWb8Ywz44MdzbIkAKxQQrhTXt8 ZJpQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mKG3LIYS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id u8-20020a170902e5c800b001d2e6c3f136si643214plf.96.2023.12.07.16.54.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 16:54:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=mKG3LIYS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B391280DECEA; Thu, 7 Dec 2023 16:54:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1444267AbjLHAx6 (ORCPT + 99 others); Thu, 7 Dec 2023 19:53:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1444192AbjLHAxY (ORCPT ); Thu, 7 Dec 2023 19:53:24 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C08DF19A5 for ; Thu, 7 Dec 2023 16:53:26 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5d10f5bf5d9so15543227b3.3 for ; Thu, 07 Dec 2023 16:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701996806; x=1702601606; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MQeyU/Au+VHoLC/CemSxB4TP3xyLB5qWh0DnEWyEst8=; b=mKG3LIYSvgUaZXWqG3b0c+K1M73nhULqOdoFoCOgyTKk66VBWzbX96bgjj6q2coEaB +alO1hqSD/lN3OsaBmALW4aN/tPw1MVkUDbOCh4T2mdmCu1uNA90oIUd9rQSly0q8ySC 8F3adQtYkoDGvOFufA+3Ig4bABNF5tDVnew6MHT3sdge26xEWlNaMlGfYhzXcLSBKgLe ur51g6f3iPQT8ses4LqLU3GkMuxWErC6QeP/+h+Nqk+DxPq5JR0KvlUq2R0WuIwOggHy GfqR22DwgU5QoDj18nEnB80rCnzSG39OeKXJby62ESV6fqAgVpSL8isnufEvZn5hEtay nmUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701996806; x=1702601606; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MQeyU/Au+VHoLC/CemSxB4TP3xyLB5qWh0DnEWyEst8=; b=fYIcgWk32xucZIDloOTnxgSWCG7DJdGWjCu9nI0gLvWVjcQu8QJ8xdzY6I1tVKLdso LdTVjNzQ2RF9BwuJzBxrga5J53N2YUa5OH2zAN/M40LOsY4J2/O8ctu9DGUMxbfiAfAn fDMF3Xc2AM+NxRm3UKWS/Oakk8XiL1U/lBDA/irGt7AieT8Yh9SmMO6zaP7JMzVRzA95 ncTZtf3eonw7nnysgx3EVuxDD/6ypWn+yxu3qndAy7fvI56loVU/QBEXnwMBGs9ZQAQt nuPwZEcz8nOLN7orkxtuguhYaFw09Lns3i7cx47+bmCxaJ3TJAC4wkVUhVeUnViyBUN5 /k1A== X-Gm-Message-State: AOJu0YzXZX2vVq3AMahzaT5+ZiYtcZSJcGfOCQ2vr+EsEoLsIJTlDw9a JRSBiw7m055al+NckYhVAmQro7tFMPKtLOgfBg== X-Received: from almasrymina.svl.corp.google.com ([2620:15c:2c4:200:f1cf:c733:235b:9fff]) (user=almasrymina job=sendgmr) by 2002:a05:690c:844:b0:5d6:cb62:4793 with SMTP id bz4-20020a05690c084400b005d6cb624793mr51347ywb.0.1701996806041; Thu, 07 Dec 2023 16:53:26 -0800 (PST) Date: Thu, 7 Dec 2023 16:52:46 -0800 In-Reply-To: <20231208005250.2910004-1-almasrymina@google.com> Mime-Version: 1.0 References: <20231208005250.2910004-1-almasrymina@google.com> X-Mailer: git-send-email 2.43.0.472.g3155946c3a-goog Message-ID: <20231208005250.2910004-16-almasrymina@google.com> Subject: [net-next v1 15/16] net: add devmem TCP documentation From: Mina Almasry To: Shailend Chand , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-arch@vger.kernel.org, linux-kselftest@vger.kernel.org, bpf@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Mina Almasry , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Jeroen de Borst , Praveen Kaligineedi , Jesper Dangaard Brouer , Ilias Apalodimas , Arnd Bergmann , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , " =?utf-8?q?Christian_K=C3=B6nig?= " , Yunsheng Lin , Harshitha Ramamurthy , Shakeel Butt X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 07 Dec 2023 16:54:08 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784673048401873348 X-GMAIL-MSGID: 1784673048401873348 Signed-off-by: Mina Almasry --- Documentation/networking/devmem.rst | 270 ++++++++++++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 Documentation/networking/devmem.rst diff --git a/Documentation/networking/devmem.rst b/Documentation/networking/devmem.rst new file mode 100644 index 000000000000..ed0d9c88b708 --- /dev/null +++ b/Documentation/networking/devmem.rst @@ -0,0 +1,270 @@ + +================= +Device Memory TCP +================= + + +Intro +===== + +Device memory TCP (devmem TCP) enables receiving data directly into device +memory (dmabuf). The feature is currently implemented for TCP sockets. + + +Opportunity +----------- + +A large amount of data transfers have device memory as the source and/or +destination. Accelerators drastically increased the volume of such transfers. +Some examples include: + +- Distributed training, where ML accelerators, such as GPUs on different hosts, + exchange data among them. + +- Distributed raw block storage applications transfer large amounts of data with + remote SSDs, much of this data does not require host processing. + +Today, the majority of the Device-to-Device data transfers the network are +implemented as the following low level operations: Device-to-Host copy, +Host-to-Host network transfer, and Host-to-Device copy. + +The implementation is suboptimal, especially for bulk data transfers, and can +put significant strains on system resources such as host memory bandwidth and +PCIe bandwidth. + +Devmem TCP optimizes this use case by implementing socket APIs that enable +the user to receive incoming network packets directly into device memory. + +Packet payloads go directly from the NIC to device memory. + +Packet headers go to host memory and are processed by the TCP/IP stack +normally. The NIC must support header split to achieve this. + +Advantages: + +- Alleviate host memory bandwidth pressure, compared to existing + network-transfer + device-copy semantics. + +- Alleviate PCIe bandwidth pressure, by limiting data transfer to the lowest + level of the PCIe tree, compared to traditional path which sends data through + the root complex. + + +More Info +--------- + + slides, video + https://netdevconf.org/0x17/sessions/talk/device-memory-tcp.html + + patchset + [RFC PATCH v3 00/12] Device Memory TCP + https://lore.kernel.org/lkml/20231106024413.2801438-1-almasrymina@google.com/T/ + + +Interface +========= + +Example +------- + +tools/testing/selftests/net/ncdevmem.c:do_server shows an example of setting up +the RX path of this API. + +NIC Setup +--------- + +Header split, flow steering, & RSS are required features for devmem TCP. + +Header split is used to split incoming packets into a header buffer in host +memory, and a payload buffer in device memory. + +Flow steering & RSS are used to ensure that only flows targeting devmem land on +RX queue bound to devmem. + +Enable header split & flow steering: + +:: + + # enable header split (assuming priv-flag) + ethtool --set-priv-flags eth1 enable-header-split on + + # enable flow steering + ethtool -K eth1 ntuple on + +Configure RSS to steer all traffic away from the target RX queue (queue 15 in +this example): + +:: + + ethtool --set-rxfh-indir eth1 equal 15 + + +The user must bind a dmabuf to any number of RX queues on a given NIC using +netlink API: + +:: + + /* Bind dmabuf to NIC RX queue 15 */ + struct netdev_queue *queues; + queues = malloc(sizeof(*queues) * 1); + + queues[0]._present.type = 1; + queues[0]._present.idx = 1; + queues[0].type = NETDEV_RX_QUEUE_TYPE_RX; + queues[0].idx = 15; + + *ys = ynl_sock_create(&ynl_netdev_family, &yerr); + + req = netdev_bind_rx_req_alloc(); + netdev_bind_rx_req_set_ifindex(req, 1 /* ifindex */); + netdev_bind_rx_req_set_dmabuf_fd(req, dmabuf_fd); + __netdev_bind_rx_req_set_queues(req, queues, n_queue_index); + + rsp = netdev_bind_rx(*ys, req); + + dmabuf_id = rsp->dmabuf_id; + + +The netlink API returns a dmabuf_id: a unique ID that refers to this dmabuf +that has been bound. + +Socket Setup +------------ + +The socket must be flow steering to the dmabuf bound RX queue: + +:: + + ethtool -N eth1 flow-type tcp4 ... queue 15, + + +Receiving data +-------------- + +The user application must signal to the kernel that it is capable of receiving +devmem data by passing the MSG_SOCK_DEVMEM flag to recvmsg: + +:: + + ret = recvmsg(fd, &msg, MSG_SOCK_DEVMEM); + +Applications that do not specify the MSG_SOCK_DEVMEM flag will receive an EFAULT +on devmem data. + +Devmem data is received directly into the dmabuf bound to the NIC in 'NIC +Setup', and the kernel signals such to the user via the SCM_DEVMEM_* cmsgs: + +:: + + for (cm = CMSG_FIRSTHDR(&msg); cm; cm = CMSG_NXTHDR(&msg, cm)) { + if (cm->cmsg_level != SOL_SOCKET || + (cm->cmsg_type != SCM_DEVMEM_DMABUF && + cm->cmsg_type != SCM_DEVMEM_LINEAR)) + continue; + + dmabuf_cmsg = (struct dmabuf_cmsg *)CMSG_DATA(cm); + + if (cm->cmsg_type == SCM_DEVMEM_DMABUF) { + /* Frag landed in dmabuf. + * + * dmabuf_cmsg->dmabuf_id is the dmabuf the + * frag landed on. + * + * dmabuf_cmsg->frag_offset is the offset into + * the dmabuf where the frag starts. + * + * dmabuf_cmsg->frag_size is the size of the + * frag. + * + * dmabuf_cmsg->frag_token is a token used to + * refer to this frag for later freeing. + */ + + struct dmabuf_token token; + token.token_start = dmabuf_cmsg->frag_token; + token.token_count = 1; + continue; + } + + if (cm->cmsg_type == SCM_DEVMEM_LINEAR) + /* Frag landed in linear buffer. + * + * dmabuf_cmsg->frag_size is the size of the + * frag. + */ + continue; + + } + +Applications may receive 2 cmsgs: + +- SCM_DEVMEM_DMABUF: this indicates the fragment landed in the dmabuf indicated + by dmabuf_id. + +- SCM_DEVMEM_LINEAR: this indicates the fragment landed in the linear buffer. + This typically happens when the NIC is unable to split the packet at the + header boundary, such that part (or all) of the payload landed in host + memory. + +Applications may receive no SO_DEVMEM_* cmsgs. That indicates non-devmem, +regular TCP data that landed on an RX queue not bound to a dmabuf. + + +Freeing frags +------------- + +Frags received via SCM_DEVMEM_DMABUF are pinned by the kernel while the user +processes the frag. The user must return the frag to the kernel via +SO_DEVMEM_DONTNEED: + +:: + + ret = setsockopt(client_fd, SOL_SOCKET, SO_DEVMEM_DONTNEED, &token, + sizeof(token)); + +The user must ensure the tokens are returned to the kernel in a timely manner. +Failure to do so will exhaust the limited dmabuf that is bound to the RX queue +and will lead to packet drops. + + +Implementation & Caveats +======================== + +Unreadable skbs +--------------- + +Devmem payloads are inaccessible to the kernel processing the packets. This +results in a few quirks for payloads of devmem skbs: + +- Loopback is not functional. Loopback relies on copying the payload, which is + not possible with devmem skbs. + +- Software checksum calculation fails. + +- TCP Dump and bpf can't access devmem packet payloads. + + +Testing +======= + +More realistic example code can be found in the kernel source under +tools/testing/selftests/net/ncdevmem.c + +ncdevmem is a devmem TCP netcat. It works very similarly to netcat, but +receives data directly into a udmabuf. + +To run ncdevmem, you need to run it a server on the machine under test, and you +need to run netcat on a peer to provide the TX data. + +ncdevmem has a validation mode as well that expects a repeating pattern of +incoming data and validates it as such: + +:: + + # On server: + ncdevmem -s -c -f eth1 -d 3 -n 0000:06:00.0 -l \ + -p 5201 -v 7 + + # On client: + yes $(echo -e \\x01\\x02\\x03\\x04\\x05\\x06) | \ + tr \\n \\0 | head -c 5G | nc 5201 -p 5201