From patchwork Tue Aug 15 13:11:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Thomas_Hellstr=C3=B6m?= X-Patchwork-Id: 135712 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:3f2:4152:657d with SMTP id x8csp827783vqo; Tue, 15 Aug 2023 19:37:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH+W9oXY4QTiOWi1A519HbGXcENxnrvnWfOuNyajdF6Iip4YNERDOM32JqeVff1FhSHLXb7 X-Received: by 2002:a05:6870:2197:b0:1bb:5bc3:7f23 with SMTP id l23-20020a056870219700b001bb5bc37f23mr642241oae.46.1692153421554; Tue, 15 Aug 2023 19:37:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692153421; cv=none; d=google.com; s=arc-20160816; b=D6zoT3KzJB2NDK4rzJK6nhaMOfGcPDTUDbYiVaBBZZr7o2ftG5+dO0jCd8LISfBXmk hYDAL4madN5RpKXw7ZUu3hQGuApfNWoOSjQ1CVqQ3iFJfGUC5HOqzBL3+zfiZ6h3Gr8B JH28Fvq9k8BVHCuALPyU0iNtIZCEccR+olIgCGjnRp55o3ERxLXFE99mER8OBdxpaoD3 EWiYXBE1sV8uLeTo72XXSOPc/t7p0U6DwsD1/9UiLc/XSvZxq4e/Px2uvDoUrZLuMGpF N6S6INmmptEMRtJvSQSOR8y9f3TbdtnH1sErNqx/E73bkvKwK53WbSO5KFacR/fXfxBI sEQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=9dUKUQYL6D+2VihxFkIwjDYcQziqUmGo8hINKxXWvBQ=; fh=SYf4ENVv4zKJGYkg0xmoDW8Wdc/9yc3nSWf++ghkaTA=; b=aak5jefHd7c6Npjjsyuq2mellMrSTQBJIrJehVCIqYLSlinwISZ6TVgmoM3hJn2wiQ dN+iIJcBwYQORgvdbxIJaLKKDoXEsDcSkMwPR2qN1S0X+AHnDDro5N55fXLF5nf6ns/0 Yr98B8O16M6c8k6Qz36qah9i9DL2r5CJLe0slzljpRkp4l+LCC5WtnKHYQhgFS/+D0Bc ZRcmhIHUs/Vraz53OxvDFAmOBpnLqLrNuYPmHzJnXtTNqtGOPPp9gZ+PQGhrseXjKbHb QD5lYnSWneAEK1shywxsJN/A8piTAlQ8P3OgVRG/gtLkNBvA4jhIZ82P6N2eA6K93k9x 0kCQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@intel.com header.s=Intel header.b=QraJdlFi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id em18-20020a17090b015200b00262f7dccaecsi10469712pjb.170.2023.08.15.19.36.47; Tue, 15 Aug 2023 19:37:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=fail header.i=@intel.com header.s=Intel header.b=QraJdlFi; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237392AbjHONMm (ORCPT + 99 others); Tue, 15 Aug 2023 09:12:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237411AbjHONMT (ORCPT ); Tue, 15 Aug 2023 09:12:19 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71FFC1987 for ; Tue, 15 Aug 2023 06:12:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692105136; x=1723641136; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=atd57xgbwjpMIEvd12NuIH1rfQR5AtnfgdNfQWZfXwc=; b=QraJdlFi7IF6cbJaQ7sMPFHmIMmr0LUK9d1+imdpDFfUYatEX7M90VcE Xu9jQN5wZkPU7FdgIWACjxGpN9xnxcP8aPA8NYi5Y23+WW+gTfTNbITbO 3JuNpae8Sxd0coUfA7/7GmeggdznD7fWeXgoOvbeQOFQn3o+pjR0wJNoV tInvgtT4f2+pYj0kd6TK9L3PNvwHCgEJHkvfAfVXgIukqCOfxja0XAQM6 gGQyEVUm2bPhjDJsIV4glB03ACtOLIAYyz6/nAhWVUZJYilLp1Ky5s4zr NBTeuij8sJiwmLzN3v313pNAikfaso6fCcc7CW7OrxKuwcyQOpakrXimO g==; X-IronPort-AV: E=McAfee;i="6600,9927,10803"; a="436165695" X-IronPort-AV: E=Sophos;i="6.01,174,1684825200"; d="scan'208";a="436165695" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Aug 2023 06:12:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10803"; a="768811686" X-IronPort-AV: E=Sophos;i="6.01,174,1684825200"; d="scan'208";a="768811686" Received: from fnygreen-mobl1.ger.corp.intel.com (HELO thellstr-mobl1.intel.com) ([10.249.254.187]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Aug 2023 06:12:07 -0700 From: =?utf-8?q?Thomas_Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Zanoni@vger.kernel.org, Paulo R , Nirmoy Das , Danilo Krummrich , Matthew Brost , Rodrigo Vivi , Joonas Lahtinen , Oak Zeng , Daniel Vetter , Maarten Lankhorst , Francois Dugast , dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: [PATCH v6] Documentation/gpu: Add a VM_BIND async document Date: Tue, 15 Aug 2023 15:11:40 +0200 Message-ID: <20230815131140.37844-1-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1774351466183050716 X-GMAIL-MSGID: 1774351466183050716 Add a motivation for and description of asynchronous VM_BIND operation v2: - Fix typos (Nirmoy Das) - Improve the description of a memory fence (Oak Zeng) - Add a reference to the document in the Xe RFC. - Add pointers to sample uAPI suggestions v3: - Address review comments (Danilo Krummrich) - Formatting fixes v4: - Address typos (Francois Dugast) - Explain why in-fences are not allowed for VM_BIND operations for long- running workloads (Matthew Brost) v5: - More typo- and style fixing - Further clarify the implications of disallowing in-fences for VM_BIND operations for long-running workloads (Matthew Brost) v6: - Point out that a gpu_vm is a virtual GPU Address space. (Danilo Krummrich) - For an explanation of dma-fences point to the dma-fence documentation. (Paolo Zanoni) - Clarify that VM_BIND errors are reported synchronously. (Paolo Zanoni) - Use an rst doc reference when pointing to the async vm_bind document from the xe merge plan. - Add the VM_BIND documentation to the drm documentation table-of-content, using an intermediate "Misc DRM driver uAPI- and feature implementation guidelines" Cc: Zanoni, Paulo R Signed-off-by: Thomas Hellström Acked-by: Nirmoy Das Reviewed-by: Danilo Krummrich Reviewed-by: Matthew Brost Reviewed-by: Rodrigo Vivi --- Documentation/gpu/drm-vm-bind-async.rst | 178 ++++++++++++++++++ .../gpu/implementation_guidelines.rst | 11 ++ Documentation/gpu/index.rst | 1 + Documentation/gpu/rfc/xe.rst | 4 +- 4 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 Documentation/gpu/drm-vm-bind-async.rst create mode 100644 Documentation/gpu/implementation_guidelines.rst diff --git a/Documentation/gpu/drm-vm-bind-async.rst b/Documentation/gpu/drm-vm-bind-async.rst new file mode 100644 index 000000000000..80cd93a5b59c --- /dev/null +++ b/Documentation/gpu/drm-vm-bind-async.rst @@ -0,0 +1,178 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +==================== +Asynchronous VM_BIND +==================== + +Nomenclature: +============= + +* ``VRAM``: On-device memory. Sometimes referred to as device local memory. + +* ``gpu_vm``: A virtual GPU address space. Typically per process, but + can be shared by multiple processes. + +* ``VM_BIND``: An operation or a list of operations to modify a gpu_vm using + an IOCTL. The operations include mapping and unmapping system- or + VRAM memory. + +* ``syncobj``: A container that abstracts synchronization objects. The + synchronization objects can be either generic, like dma-fences or + driver specific. A syncobj typically indicates the type of the + underlying synchronization object. + +* ``in-syncobj``: Argument to a VM_BIND IOCTL, the VM_BIND operation waits + for these before starting. + +* ``out-syncobj``: Argument to a VM_BIND_IOCTL, the VM_BIND operation + signals these when the bind operation is complete. + +* ``dma-fence``: A cross-driver synchronization object. A basic + understanding of dma-fences is required to digest this + document. Please refer to the ``DMA Fences`` section of the + :doc:`dma-buf doc `. + +* ``memory fence``: A synchronization object, different from a dma-fence. + A memory fence uses the value of a specified memory location to determine + signaled status. A memory fence can be awaited and signaled by both + the GPU and CPU. Memory fences are sometimes referred to as + user-fences, userspace-fences or gpu futexes and do not necessarily obey + the dma-fence rule of signaling within a "reasonable amount of time". + The kernel should thus avoid waiting for memory fences with locks held. + +* ``long-running workload``: A workload that may take more than the + current stipulated dma-fence maximum signal delay to complete and + which therefore needs to set the gpu_vm or the GPU execution context in + a certain mode that disallows completion dma-fences. + +* ``exec function``: An exec function is a function that revalidates all + affected gpu_vmas, submits a GPU command batch and registers the + dma_fence representing the GPU command's activity with all affected + dma_resvs. For completeness, although not covered by this document, + it's worth mentioning that an exec function may also be the + revalidation worker that is used by some drivers in compute / + long-running mode. + +* ``bind context``: A context identifier used for the VM_BIND + operation. VM_BIND operations that use the same bind context can be + assumed, where it matters, to complete in order of submission. No such + assumptions can be made for VM_BIND operations using separate bind contexts. + +* ``UMD``: User-mode driver. + +* ``KMD``: Kernel-mode driver. + + +Synchronous / Asynchronous VM_BIND operation +============================================ + +Synchronous VM_BIND +___________________ +With Synchronous VM_BIND, the VM_BIND operations all complete before the +IOCTL returns. A synchronous VM_BIND takes neither in-fences nor +out-fences. Synchronous VM_BIND may block and wait for GPU operations; +for example swap-in or clearing, or even previous binds. + +Asynchronous VM_BIND +____________________ +Asynchronous VM_BIND accepts both in-syncobjs and out-syncobjs. While the +IOCTL may return immediately, the VM_BIND operations wait for the in-syncobjs +before modifying the GPU page-tables, and signal the out-syncobjs when +the modification is done in the sense that the next exec function that +awaits for the out-syncobjs will see the change. Errors are reported +synchronously. +In low-memory situations the implementation may block, performing the +VM_BIND synchronously, because there might not be enough memory +immediately available for preparing the asynchronous operation. + +If the VM_BIND IOCTL takes a list or an array of operations as an argument, +the in-syncobjs needs to signal before the first operation starts to +execute, and the out-syncobjs signal after the last operation +completes. Operations in the operation list can be assumed, where it +matters, to complete in order. + +Since asynchronous VM_BIND operations may use dma-fences embedded in +out-syncobjs and internally in KMD to signal bind completion, any +memory fences given as VM_BIND in-fences need to be awaited +synchronously before the VM_BIND ioctl returns, since dma-fences, +required to signal in a reasonable amount of time, can never be made +to depend on memory fences that don't have such a restriction. + +To aid in supporting user-space queues, the VM_BIND may take a bind context. + +The purpose of an Asynchronous VM_BIND operation is for user-mode +drivers to be able to pipeline interleaved gpu_vm modifications and +exec functions. For long-running workloads, such pipelining of a bind +operation is not allowed and any in-fences need to be awaited +synchronously. The reason for this is twofold. First, any memory +fences gated by a long-running workload and used as in-syncobjs for the +VM_BIND operation will need to be awaited synchronously anyway (see +above). Second, any dma-fences used as in-syncobjs for VM_BIND +operations for long-running workloads will not allow for pipelining +anyway since long-running workloads don't allow for dma-fences as +out-syncobjs, so while theoretically possible the use of them is +questionable and should be rejected until there is a valuable use-case. +Note that this is not a limitation imposed by dma-fence rules, but +rather a limitation imposed to keep KMD implementation simple. It does +not affect using dma-fences as dependencies for the long-running +workload itself, which is allowed by dma-fence rules, but rather for +the VM_BIND operation only. + +Also for VM_BINDS for long-running gpu_vms the user-mode driver should typically +select memory fences as out-fences since that gives greater flexibility for +the kernel mode driver to inject other operations into the bind / +unbind operations. Like for example inserting breakpoints into batch +buffers. The workload execution can then easily be pipelined behind +the bind completion using the memory out-fence as the signal condition +for a GPU semaphore embedded by UMD in the workload. + +Multi-operation VM_BIND IOCTL error handling and interrupts +=========================================================== + +The VM_BIND operations of the IOCTL may error due to lack of resources +to complete and also due to interrupted waits. In both situations UMD +should preferably restart the IOCTL after taking suitable action. If +UMD has over-committed a memory resource, an -ENOSPC error will be +returned, and UMD may then unbind resources that are not used at the +moment and restart the IOCTL. On -EINTR, UMD should simply restart the +IOCTL and on -ENOMEM user-space may either attempt to free known +system memory resources or abort the operation. If aborting as a +result of a failed operation in a list of operations, some operations +may still have completed, and to get back to a known state, user-space +should therefore attempt to unbind all virtual memory regions touched +by the failing IOCTL. +Unbind operations are guaranteed not to cause any errors due to +resource constraints. +In between a failed VM_BIND IOCTL and a successful restart there may +be implementation defined restrictions on the use of the gpu_vm. For a +description why, please see KMD implementation details under `error +state saving`_. + +Sample uAPI implementations +=========================== +Suggested uAPI implementations at the moment of writing can be found for +the `Nouveau driver +`_. +and for the `Xe driver +`_. + +KMD implementation details +========================== + +Error state saving +__________________ +Open: When the VM_BIND IOCTL returns an error, some or even parts of +an operation may have been completed. If the IOCTL is restarted, in +order to know where to restart, the KMD can either put the gpu_vm in +an error state and save one instance of the needed restart state +internally. In this case, KMD needs to block further modifications of +the gpu_vm state that may cause additional failures requiring a +restart state save, until the error has been fully resolved. If the +uAPI instead defines a pointer to a UMD allocated cookie in the IOCTL +struct, it could also choose to store the restart state in that cookie. + +The restart state may, for example, be the number of successfully +completed operations. + +Easiest for UMD would of course be if KMD did a full unwind on error +so that no error state needs to be saved. diff --git a/Documentation/gpu/implementation_guidelines.rst b/Documentation/gpu/implementation_guidelines.rst new file mode 100644 index 000000000000..32907bbf4c99 --- /dev/null +++ b/Documentation/gpu/implementation_guidelines.rst @@ -0,0 +1,11 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +=========================================================== +Misc DRM driver uAPI- and feature implementation guidelines +=========================================================== + +.. toctree:: + + drm-vm-bind-async + + diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst index eee5996acf2c..015b3fe726de 100644 --- a/Documentation/gpu/index.rst +++ b/Documentation/gpu/index.rst @@ -17,6 +17,7 @@ GPU Driver Developer's Guide backlight vga-switcheroo vgaarbiter + implementation_guidelines todo rfc/index diff --git a/Documentation/gpu/rfc/xe.rst b/Documentation/gpu/rfc/xe.rst index 2516fe141db6..e095b23e3dfd 100644 --- a/Documentation/gpu/rfc/xe.rst +++ b/Documentation/gpu/rfc/xe.rst @@ -138,8 +138,8 @@ memory fences. Ideally with helper support so people don't get it wrong in all possible ways. As a key measurable result, the benefits of ASYNC VM_BIND and a discussion of -various flavors, error handling and a sample API should be documented here or in -a separate document pointed to by this document. +various flavors, error handling and sample API suggestions are documented in +:doc:`The ASYNC VM_BIND document `. Userptr integration and vm_bind -------------------------------