[v4,00/16] Enable DSA 2.0 Event Log and completion record faulting features

  Applications can send 64B descriptors to the DSA device via CPU
instructions MOVDIR64B or ENQCMD. The application can choose to have
the device write back a completion record (CR) in system memory to
indicate the status of the descriptor submitted on completion.

With the DSA hardware, the device is able to do on demand paging through
the hardware by faulting in the user pages that do not have physical memory
page backing with assistance from IOMMU. In the spec this was designated as
the block on fault feature. While this hardware feature made operation
simpler, it also stalls the device engines while the memory pages are being
faulted in through Page Request Service (PRS). For applications sharing the
same workqueue (wq) or wqs in the same group, operations are stalled if
there are no free engines. To avoid slowing the performance of all other
running applications sharing the same device engine(s), PRS can to be
disabled and software can deal with partial completion.

The block on fault feature on DSA 1.0 can be disabled for the wq. However,
PRS is not completely disabled for the whole path. It is not disabled for
CRs or batch list for a batch operation.

The other issue is the DSA 1.0 error reporting mechanism, SWERROR register.
The SWERROR register can only report a single error at a time until the
driver reads and acknowledges the error. The follow on errors cannot be
reported until the current error is "cleared" by the software by writing
a bit to the SWERR register. If a large number of faults arrive and the
software cannot clear them fast enough, overflowed errors will be dropped
by the device.

A CR is the optional 32 bytes (DSA) or 64 bytes (IAA) status that is
written back for a submitted descriptor. If the address for the CR faults,
the error is reported to the SWERROR register instead.

With DSA 2.0 hardware [1], the event log feature is added. All errors are
reported as an entry in a circular buffer reside in the system memory.
The system admin is responsible to configure the size of the circular
buffer large enough per device to handle the potential errors that may be
reported. If the buffer is full and another error needs to be reported,
the device engine will block until there's a free slot in the buffer.
An event log entry for a faulted CR will contain the error information,
the CR address that faulted, and the expected CR content the device had
originally intended to write.

DSA 2.0 also introduces per wq PRS disable knob. This will disable all PRS
operations for the specific wq. The device will still have Address
Translation Service (ATS) on. When ATS fails on a memory address for a CR,
an eventlog entry will be written by the hardware into the event log
ring buffer. The driver software is expected to parse the event log entry,
fault in the address of the CR, and the write the content of the CR to
the memory address.

This patch series will implement the DSA 2 event log support. The support
for the handling of the faulted user CR is added. The driver is also
adding the same support for batch operation descriptors. With a batch
operation the handling of the event log entry is a bit more complex.
The faulting CR could be for the batch descriptor or any of the operation
descriptors within the batch. The hardware generates a batch identifier
that is used by the driver software to correlate the event log entries for
the relevant descriptors of that batch.

The faulting of source and destination addresses for the operation is not
handled by the driver. That is left to be handled by the user application
by faulting in the memory and re-submit the remaining operation.

This series consists of three parts:
1. Patch 1: Make misc interrupt one shot. Event Log interrupt depends on
   this patch. This patch was released before but is not in upstream yet:
   https://lore.kernel.org/dmaengine/165125374675.311834.10460196228320964350.stgit@djiang5-desk3.ch.intel.com/
2. Patches 2-15: Enable Event Log and Completion Record faulting.
3. Patch 16: Configure PRS disable per WQ.

This series is applied cleanly on top of "Expose IAA 2.0 device
capabilities" series (which is in dmaengine next branch now):
https://lore.kernel.org/lkml/20230303213732.3357494-1-fenghua.yu@intel.com/

Change log:
v4:
- Use kthread_use_mm(), copy_to_user(), and kthread_unuse_mm() to switch
  to the mm, copy completion record to the mm, and switch back to the
  current mm. It's simpler than previous access_remote_vm() or emulation
  ways (Jason Gunthorpe, Christoph Hellwig, Tony Luck).
- Change dev_err() to dev_dbg_ratelimited() in CR page fault handler
  (Tony Luck).

v3:
- Since iommu_sva_find() will be removed in IOMMU and access_remote_vm()
  cannot be exported, the completion record copy function idxd_copy_cr()
  is rewritten by maintaining and finding mm in xarray and copy completion
  record to the mm.
  Please check discussion on iommu_sva_find() will be removed and
  access_remote_vm() cannot be exported:
  1. https://lore.kernel.org/lkml/ZAjSsm4%2FPDRqViwa@nvidia.com/
  2. https://lore.kernel.org/lkml/20230306163138.587484-1-fenghua.yu@intel.com/T/#m1fc97725a0e56ea269c8bdabacee447070d51846

v2:
- Define and export iommu_access_remote_vm() for IDXD driver to write
  completion record to user address space. This change removes
  patch 8 and 9 in v1 (Alistair Popple)
https://lore.kernel.org/lkml/20230306163138.587484-1-fenghua.yu@intel.com/

Dave Jiang (15):
  dmaengine: idxd: make misc interrupt one shot
  dmaengine: idxd: add event log size sysfs attribute
  dmaengine: idxd: setup event log configuration
  dmaengine: idxd: add interrupt handling for event log
  dmanegine: idxd: add debugfs for event log dump
  dmaengine: idxd: add per DSA wq workqueue for processing cr faults
  dmaengine: idxd: create kmem cache for event log fault items
  dmaengine: idxd: process user page faults for completion record
  dmaengine: idxd: add descs_completed field for completion record
  dmaengine: idxd: process batch descriptor completion record faults
  dmaengine: idxd: add per file user counters for completion record
    faults
  dmaengine: idxd: add a device to represent the file opened
  dmaengine: idxd: expose fault counters to sysfs
  dmaengine: idxd: add pid to exported sysfs attribute for opened file
  dmaengine: idxd: add per wq PRS disable

Fenghua Yu (1):
  dmaengine: idxd: add idxd_copy_cr() to copy user completion record
    during page fault handling

 .../ABI/stable/sysfs-driver-dma-idxd          |  43 +++
 drivers/dma/idxd/Makefile                     |   2 +-
 drivers/dma/idxd/cdev.c                       | 334 ++++++++++++++++--
 drivers/dma/idxd/debugfs.c                    | 138 ++++++++
 drivers/dma/idxd/device.c                     | 113 +++++-
 drivers/dma/idxd/idxd.h                       |  65 ++++
 drivers/dma/idxd/init.c                       |  53 +++
 drivers/dma/idxd/irq.c                        | 210 +++++++++--
 drivers/dma/idxd/registers.h                  | 105 +++++-
 drivers/dma/idxd/sysfs.c                      | 112 +++++-
 include/uapi/linux/idxd.h                     |  15 +-
 11 files changed, 1126 insertions(+), 64 deletions(-)
 create mode 100644 drivers/dma/idxd/debugfs.c
  

On 07-04-23, 13:31, Fenghua Yu wrote:
> Applications can send 64B descriptors to the DSA device via CPU
> instructions MOVDIR64B or ENQCMD. The application can choose to have
> the device write back a completion record (CR) in system memory to
> indicate the status of the descriptor submitted on completion.
> 
> With the DSA hardware, the device is able to do on demand paging through
> the hardware by faulting in the user pages that do not have physical memory
> page backing with assistance from IOMMU. In the spec this was designated as
> the block on fault feature. While this hardware feature made operation
> simpler, it also stalls the device engines while the memory pages are being
> faulted in through Page Request Service (PRS). For applications sharing the
> same workqueue (wq) or wqs in the same group, operations are stalled if
> there are no free engines. To avoid slowing the performance of all other
> running applications sharing the same device engine(s), PRS can to be
> disabled and software can deal with partial completion.
> 
> The block on fault feature on DSA 1.0 can be disabled for the wq. However,
> PRS is not completely disabled for the whole path. It is not disabled for
> CRs or batch list for a batch operation.
> 
> The other issue is the DSA 1.0 error reporting mechanism, SWERROR register.
> The SWERROR register can only report a single error at a time until the
> driver reads and acknowledges the error. The follow on errors cannot be
> reported until the current error is "cleared" by the software by writing
> a bit to the SWERR register. If a large number of faults arrive and the
> software cannot clear them fast enough, overflowed errors will be dropped
> by the device.
> 
> A CR is the optional 32 bytes (DSA) or 64 bytes (IAA) status that is
> written back for a submitted descriptor. If the address for the CR faults,
> the error is reported to the SWERROR register instead.
> 
> With DSA 2.0 hardware [1], the event log feature is added. All errors are
> reported as an entry in a circular buffer reside in the system memory.
> The system admin is responsible to configure the size of the circular
> buffer large enough per device to handle the potential errors that may be
> reported. If the buffer is full and another error needs to be reported,
> the device engine will block until there's a free slot in the buffer.
> An event log entry for a faulted CR will contain the error information,
> the CR address that faulted, and the expected CR content the device had
> originally intended to write.
> 
> DSA 2.0 also introduces per wq PRS disable knob. This will disable all PRS
> operations for the specific wq. The device will still have Address
> Translation Service (ATS) on. When ATS fails on a memory address for a CR,
> an eventlog entry will be written by the hardware into the event log
> ring buffer. The driver software is expected to parse the event log entry,
> fault in the address of the CR, and the write the content of the CR to
> the memory address.
> 
> This patch series will implement the DSA 2 event log support. The support
> for the handling of the faulted user CR is added. The driver is also
> adding the same support for batch operation descriptors. With a batch
> operation the handling of the event log entry is a bit more complex.
> The faulting CR could be for the batch descriptor or any of the operation
> descriptors within the batch. The hardware generates a batch identifier
> that is used by the driver software to correlate the event log entries for
> the relevant descriptors of that batch.
> 
> The faulting of source and destination addresses for the operation is not
> handled by the driver. That is left to be handled by the user application
> by faulting in the memory and re-submit the remaining operation.

Applied, thanks
  

On 4/12/23 10:18, Vinod Koul wrote:
> On 07-04-23, 13:31, Fenghua Yu wrote:
>> Applications can send 64B descriptors to the DSA device via CPU
>> instructions MOVDIR64B or ENQCMD. The application can choose to have
>> the device write back a completion record (CR) in system memory to
>> indicate the status of the descriptor submitted on completion.
> Applied, thanks
> 

Thank you very much, Vinod! This is really good news!

-Fenghua