[RFC,v1,3/4] swiotlb: Allow dynamic allocation of bounce buffers

  From: Petr Tesarik <petr.tesarik.ext@huawei.com>

The software IO TLB was designed with the assumption that it is not
used much, especially on 64-bit systems, so a small fixed memory
area (currently 64 MiB) is sufficient to handle the few cases which
still require a bounce buffer. However, these cases are not so rare
in some circumstances.

First, if SEV is active, all DMA must be done through shared
unencrypted pages, and SWIOTLB is used to make this happen without
changing device drivers. The software IO TLB size is increased to
6% of total memory in sev_setup_arch(), but that is more of an
approximation. The actual requirements may vary depending on which
drivers are used and the amount of I/O.

Second, on the Raspberry Pi 4, swiotlb is used by dma-buf for pages
moved from the rendering GPU (v3d driver), which can access all
memory, to the display output (vc4 driver), which is connected to a
bus with an address limit of 1 GiB and no IOMMU. These buffers can
be large (several megabytes) and cannot be handled by SWIOTLB,
because they exceed maximum segment size of 256 KiB. Such mapping
failures can be easily reproduced on a Raspberry Pi4: Starting
GNOME remote desktop results in a flood of kernel messages like
these:

[  387.937625] vc4-drm gpu: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 3136 (slots)
[  387.960381] vc4-drm gpu: swiotlb buffer is full (sz: 815104 bytes), total 32768 (slots), used 2 (slots)

This second example cannot be even solved without increasing the
segment size (and the complexity of {map,unmap}_single size). At
that point, it's better to allocate bounce buffers dynamically with
dma_direct_alloc_pages().

One caveat is that the DMA API often takes only the address of a
buffer, and the implementation (direct or IOMMU) checks whether it
belongs to the software IO TLB. This is easy if the IO TLB is a
single chunk of physically contiguous memory, but not if some
buffers are allocated dynamically. Testing on a Raspberry Pi 4
shows that there can be 1k+ such buffers. This requires something
better than a linked list. I'm using a maple tree to track
dynamically allocated buffers. This data structure was invented for
a similar use case, but there are some challenges:

1. The value is limited to ULONG_MAX, which is too little both for
   physical addresses (e.g. x86 PAE or 32-bit ARM LPAE) and DMA
   addresses (e.g. Xen guests on 32-bit ARM).

2. Since buffers are currently allocated with page granularity, a
   PFN can be used instead. However, some values are reserved by
   the maple tree implementation. Liam suggests to use
   xa_mk_value() in that case, but that reduces the usable range by
   half. Luckily, 31 bits are still enough to hold a PFN on all
   32-bit platforms.

3. Software IO TLB is used from interrupt context. The maple tree
   implementation is not IRQ-safe (MT_FLAGS_LOCK_IRQ does nothing
   AFAICS). Instead, I use an external lock, spin_lock_irqsave() and
   spin_unlock_irqrestore().

Note that bounce buffers are never allocated dynamically if the
software IO TLB is in fact a DMA restricted pool, which is intended
to be stay in its designated location in physical memory.

Signed-off-by: Petr Tesarik <petr.tesarik.ext@huawei.com>
---
 include/linux/swiotlb.h |  11 ++-
 kernel/dma/swiotlb.c    | 156 +++++++++++++++++++++++++++++++++++++---
 2 files changed, 157 insertions(+), 10 deletions(-)
  

[adding Alex as he has been interested in this in the past]

On Mon, Mar 20, 2023 at 01:28:15PM +0100, Petr Tesarik wrote:
> Second, on the Raspberry Pi 4, swiotlb is used by dma-buf for pages
> moved from the rendering GPU (v3d driver), which can access all
> memory, to the display output (vc4 driver), which is connected to a
> bus with an address limit of 1 GiB and no IOMMU. These buffers can
> be large (several megabytes) and cannot be handled by SWIOTLB,
> because they exceed maximum segment size of 256 KiB. Such mapping
> failures can be easily reproduced on a Raspberry Pi4: Starting
> GNOME remote desktop results in a flood of kernel messages like
> these:

Shouldn't we make sure dma-buf allocates the buffers for the most
restricted devices, and more importantly does something like a dma
coherent allocation instead of a dynamic mapping of random memory?

While a larger swiotlb works around this I don't think this fixes the root
cause.

> 1. The value is limited to ULONG_MAX, which is too little both for
>    physical addresses (e.g. x86 PAE or 32-bit ARM LPAE) and DMA
>    addresses (e.g. Xen guests on 32-bit ARM).
> 
> 2. Since buffers are currently allocated with page granularity, a
>    PFN can be used instead. However, some values are reserved by
>    the maple tree implementation. Liam suggests to use
>    xa_mk_value() in that case, but that reduces the usable range by
>    half. Luckily, 31 bits are still enough to hold a PFN on all
>    32-bit platforms.
> 
> 3. Software IO TLB is used from interrupt context. The maple tree
>    implementation is not IRQ-safe (MT_FLAGS_LOCK_IRQ does nothing
>    AFAICS). Instead, I use an external lock, spin_lock_irqsave() and
>    spin_unlock_irqrestore().
> 
> Note that bounce buffers are never allocated dynamically if the
> software IO TLB is in fact a DMA restricted pool, which is intended
> to be stay in its designated location in physical memory.

I'm a little worried about all that because it causes quite a bit
of overhead even for callers that don't end up going into the
dynamic range or do not use swiotlb at all.  I don't really have a
good answer here except for the usual avoid bounce buffering whenever
you can that might not always be easy to do.

> +	gfp = (attrs & DMA_ATTR_MAY_SLEEP) ? GFP_KERNEL : GFP_NOWAIT;
> +	slot = kmalloc(sizeof(*slot), gfp | __GFP_NOWARN);
> +	if (!slot)
> +		goto err;
> +
> +	slot->orig_addr = orig_addr;
> +	slot->alloc_size = alloc_size;
> +	slot->page = dma_direct_alloc_pages(dev, PAGE_ALIGN(alloc_size),
> +					    &slot->dma_addr, dir,
> +					    gfp | __GFP_NOWARN);
> +	if (!slot->page)
> +		goto err_free_slot;

Without GFP_NOIO allocations this will deadlock eventually.
  

On 3/28/2023 6:07 AM, Christoph Hellwig wrote:
> [adding Alex as he has been interested in this in the past]
> 
> On Mon, Mar 20, 2023 at 01:28:15PM +0100, Petr Tesarik wrote:
>> Second, on the Raspberry Pi 4, swiotlb is used by dma-buf for pages
>> moved from the rendering GPU (v3d driver), which can access all
>> memory, to the display output (vc4 driver), which is connected to a
>> bus with an address limit of 1 GiB and no IOMMU. These buffers can
>> be large (several megabytes) and cannot be handled by SWIOTLB,
>> because they exceed maximum segment size of 256 KiB. Such mapping
>> failures can be easily reproduced on a Raspberry Pi4: Starting
>> GNOME remote desktop results in a flood of kernel messages like
>> these:
> 
> Shouldn't we make sure dma-buf allocates the buffers for the most
> restricted devices, and more importantly does something like a dma
> coherent allocation instead of a dynamic mapping of random memory?
> 
> While a larger swiotlb works around this I don't think this fixes the root
> cause.

I tend to agree here. However, it's the DMABUF design itself that causes
some trouble. The buffer is allocated by the v3d driver, which does not
have the restriction, so the DMA API typically allocates an address
somewhere near the 4G boundary. Userspace then exports the buffer, sends
it to another process as a file descriptor and imports it into the vc4
driver, which requires DMA below 1G. In the beginning, v3d had no idea
that the buffer would be exported to userspace, much less that it would
be later imported into vc4.

Anyway, I suspected that the buffers need not be imported into the vc4
driver (also hinted by Eric Anholt in a 2018 blog post [1]), and it
seems I was right. I encountered the issue with Ubuntu 22.10; I
installed latest openSUSE Tumbleweed yesterday, and I was not able to
reproduce the issue there, most likely because the Mesa drivers have
been fixed meanwhile. This makes the specific case of the Raspberry Pi 4
drivers moot. The issue may still affect other combinations of drivers,
but I don't have any other real-world example ATM.

[1] https://anholt.github.io/twivc4/2018/02/12/twiv/

>> 1. The value is limited to ULONG_MAX, which is too little both for
>>    physical addresses (e.g. x86 PAE or 32-bit ARM LPAE) and DMA
>>    addresses (e.g. Xen guests on 32-bit ARM).
>>
>> 2. Since buffers are currently allocated with page granularity, a
>>    PFN can be used instead. However, some values are reserved by
>>    the maple tree implementation. Liam suggests to use
>>    xa_mk_value() in that case, but that reduces the usable range by
>>    half. Luckily, 31 bits are still enough to hold a PFN on all
>>    32-bit platforms.
>>
>> 3. Software IO TLB is used from interrupt context. The maple tree
>>    implementation is not IRQ-safe (MT_FLAGS_LOCK_IRQ does nothing
>>    AFAICS). Instead, I use an external lock, spin_lock_irqsave() and
>>    spin_unlock_irqrestore().
>>
>> Note that bounce buffers are never allocated dynamically if the
>> software IO TLB is in fact a DMA restricted pool, which is intended
>> to be stay in its designated location in physical memory.
> 
> I'm a little worried about all that because it causes quite a bit
> of overhead even for callers that don't end up going into the
> dynamic range or do not use swiotlb at all.  I don't really have a
> good answer here except for the usual avoid bounce buffering whenever
> you can that might not always be easy to do.

I'm also worried about all this overhead. OTOH I was not able to confirm
it, because the difference between two successive fio test runs on an
unmodified kernel was bigger than the difference between a vanilla and a
patched kernel, except the maximum completion latency, which OTOH
affected less than 0.01% of all requests.

BTW my testing also suggests that the streaming DMA API is quite
inefficient, because UAS performance _improved_ with swiotlb=force.
Sure, this should probably be addressed in the UAS and/or xHCI driver,
but what I mean is that moving away from swiotlb may even cause
performance regressions, which is counter-intuitive. At least I would
_not_ have expected it.

>> +	gfp = (attrs & DMA_ATTR_MAY_SLEEP) ? GFP_KERNEL : GFP_NOWAIT;
>> +	slot = kmalloc(sizeof(*slot), gfp | __GFP_NOWARN);
>> +	if (!slot)
>> +		goto err;
>> +
>> +	slot->orig_addr = orig_addr;
>> +	slot->alloc_size = alloc_size;
>> +	slot->page = dma_direct_alloc_pages(dev, PAGE_ALIGN(alloc_size),
>> +					    &slot->dma_addr, dir,
>> +					    gfp | __GFP_NOWARN);
>> +	if (!slot->page)
>> +		goto err_free_slot;
> 
> Without GFP_NOIO allocations this will deadlock eventually.

Ah, that would affect the non-sleeping case (GFP_KERNEL), right?

Petr T
  

On 3/28/2023 9:54 AM, Petr Tesarik wrote:
> On 3/28/2023 6:07 AM, Christoph Hellwig wrote:
>> [adding Alex as he has been interested in this in the past]
>>
>[...]>> I'm a little worried about all that because it causes quite a bit
>> of overhead even for callers that don't end up going into the
>> dynamic range or do not use swiotlb at all.  I don't really have a
>> good answer here except for the usual avoid bounce buffering whenever
>> you can that might not always be easy to do.
> 
> I'm also worried about all this overhead.

Oh, wait! I can do at least something for devices which do not use
swiotlb at all.

If a device does not use bounce buffers, it cannot pass an address
that belongs to the swiotlb. Consequently, the potentially
expensive check can be skipped. This avoids the dynamic lookup
penalty for devices which do not need the swiotlb.

Note that the counter always remains zero if dma_io_tlb_mem is
NULL, so the NULL check is not required.

diff --git a/drivers/base/core.c b/drivers/base/core.c
index a3e14143ec0c..f36638f207b8 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -2957,6 +2957,7 @@ void device_initialize(struct device *dev)
 #endif
 #ifdef CONFIG_SWIOTLB
 	dev->dma_io_tlb_mem = &io_tlb_default_mem;
+	atomic_set(&dev->dma_io_tlb_cnt, 0);
 #endif
 }
 EXPORT_SYMBOL_GPL(device_initialize);
diff --git a/include/linux/device.h b/include/linux/device.h
index 44e3acae7b36..cfdddce4cc30 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -504,6 +504,7 @@ struct device_physical_location {
  * @dma_mem:	Internal for coherent mem override.
  * @cma_area:	Contiguous memory area for dma allocations
  * @dma_io_tlb_mem: Pointer to the swiotlb pool used.  Not for driver use.
+ * @dma_io_tlb_cnt: Number of buffers mapped from the swiotlb pool.
  * @archdata:	For arch-specific additions.
  * @of_node:	Associated device tree node.
  * @fwnode:	Associated device node supplied by platform firmware.
@@ -609,6 +610,7 @@ struct device {
 #endif
 #ifdef CONFIG_SWIOTLB
 	struct io_tlb_mem *dma_io_tlb_mem;
+	atomic_t dma_io_tlb_cnt;
 #endif
 	/* arch specific additions */
 	struct dev_archdata	archdata;
diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index 628e25ad7db7..7a115f4db49d 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -122,7 +122,7 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr)
 {
 	struct io_tlb_mem *mem = dev->dma_io_tlb_mem;
 
-	return mem &&
+	return atomic_read(&dev->dma_io_tlb_cnt) &&
 		(is_swiotlb_fixed(mem, paddr) ||
 		 (mem->allow_dyn && is_swiotlb_dyn(mem, paddr)));
 }
diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index 3efaefebb6af..3dda1d3a39e8 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -954,6 +954,8 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr,
 		return tlb_addr;
 	}
 
+	atomic_inc(&dev->dma_io_tlb_cnt);
+
 	/*
 	 * When dir == DMA_FROM_DEVICE we could omit the copy from the orig
 	 * to the tlb buffer, if we knew for sure the device will
@@ -1030,6 +1032,7 @@ void swiotlb_tbl_unmap_single(struct device *dev, phys_addr_t tlb_addr,
 		swiotlb_release_slots(dev, tlb_addr);
 	else
 		swiotlb_dyn_unmap(dev, tlb_addr, dir);
+	atomic_dec(&dev->dma_io_tlb_cnt);
 }
 
 void swiotlb_sync_single_for_device(struct device *dev, phys_addr_t tlb_addr,
  

On Tue, 28 Mar 2023 09:54:35 +0200
Petr Tesarik <petrtesarik@huaweicloud.com> wrote:

> On 3/28/2023 6:07 AM, Christoph Hellwig wrote:
> > [adding Alex as he has been interested in this in the past]
> >
> > On Mon, Mar 20, 2023 at 01:28:15PM +0100, Petr Tesarik wrote:
> >> Second, on the Raspberry Pi 4, swiotlb is used by dma-buf for pages
> >> moved from the rendering GPU (v3d driver), which can access all
> >> memory, to the display output (vc4 driver), which is connected to a
> >> bus with an address limit of 1 GiB and no IOMMU. These buffers can
> >> be large (several megabytes) and cannot be handled by SWIOTLB,
> >> because they exceed maximum segment size of 256 KiB. Such mapping
> >> failures can be easily reproduced on a Raspberry Pi4: Starting
> >> GNOME remote desktop results in a flood of kernel messages like
> >> these:
> >
> > Shouldn't we make sure dma-buf allocates the buffers for the most
> > restricted devices, and more importantly does something like a dma
> > coherent allocation instead of a dynamic mapping of random memory?
> >
> > While a larger swiotlb works around this I don't think this fixes the root
> > cause.
>
> I tend to agree here. However, it's the DMABUF design itself that causes
> some trouble. The buffer is allocated by the v3d driver, which does not
> have the restriction, so the DMA API typically allocates an address
> somewhere near the 4G boundary. Userspace then exports the buffer, sends
> it to another process as a file descriptor and imports it into the vc4
> driver, which requires DMA below 1G. In the beginning, v3d had no idea
> that the buffer would be exported to userspace, much less that it would
> be later imported into vc4.
>
> Anyway, I suspected that the buffers need not be imported into the vc4
> driver (also hinted by Eric Anholt in a 2018 blog post [1]), and it
> seems I was right. I encountered the issue with Ubuntu 22.10; I
> installed latest openSUSE Tumbleweed yesterday, and I was not able to
> reproduce the issue there, most likely because the Mesa drivers have
> been fixed meanwhile. This makes the specific case of the Raspberry Pi 4
> drivers moot. The issue may still affect other combinations of drivers,
> but I don't have any other real-world example ATM.

I'm only seeing this problem with Wayland, no issue when switching Ubuntu to
X. It seems Tumbleweed is using X by default.

...Juerg


> [1] https://anholt.github.io/twivc4/2018/02/12/twiv/
>
> >> 1. The value is limited to ULONG_MAX, which is too little both for
> >>    physical addresses (e.g. x86 PAE or 32-bit ARM LPAE) and DMA
> >>    addresses (e.g. Xen guests on 32-bit ARM).
> >>
> >> 2. Since buffers are currently allocated with page granularity, a
> >>    PFN can be used instead. However, some values are reserved by
> >>    the maple tree implementation. Liam suggests to use
> >>    xa_mk_value() in that case, but that reduces the usable range by
> >>    half. Luckily, 31 bits are still enough to hold a PFN on all
> >>    32-bit platforms.
> >>
> >> 3. Software IO TLB is used from interrupt context. The maple tree
> >>    implementation is not IRQ-safe (MT_FLAGS_LOCK_IRQ does nothing
> >>    AFAICS). Instead, I use an external lock, spin_lock_irqsave() and
> >>    spin_unlock_irqrestore().
> >>
> >> Note that bounce buffers are never allocated dynamically if the
> >> software IO TLB is in fact a DMA restricted pool, which is intended
> >> to be stay in its designated location in physical memory.
> >
> > I'm a little worried about all that because it causes quite a bit
> > of overhead even for callers that don't end up going into the
> > dynamic range or do not use swiotlb at all.  I don't really have a
> > good answer here except for the usual avoid bounce buffering whenever
> > you can that might not always be easy to do.
>
> I'm also worried about all this overhead. OTOH I was not able to confirm
> it, because the difference between two successive fio test runs on an
> unmodified kernel was bigger than the difference between a vanilla and a
> patched kernel, except the maximum completion latency, which OTOH
> affected less than 0.01% of all requests.
>
> BTW my testing also suggests that the streaming DMA API is quite
> inefficient, because UAS performance _improved_ with swiotlb=force.
> Sure, this should probably be addressed in the UAS and/or xHCI driver,
> but what I mean is that moving away from swiotlb may even cause
> performance regressions, which is counter-intuitive. At least I would
> _not_ have expected it.
>
> >> +	gfp = (attrs & DMA_ATTR_MAY_SLEEP) ? GFP_KERNEL : GFP_NOWAIT;
> >> +	slot = kmalloc(sizeof(*slot), gfp | __GFP_NOWARN);
> >> +	if (!slot)
> >> +		goto err;
> >> +
> >> +	slot->orig_addr = orig_addr;
> >> +	slot->alloc_size = alloc_size;
> >> +	slot->page = dma_direct_alloc_pages(dev, PAGE_ALIGN(alloc_size),
> >> +					    &slot->dma_addr, dir,
> >> +					    gfp | __GFP_NOWARN);
> >> +	if (!slot->page)
> >> +		goto err_free_slot;
> >
> > Without GFP_NOIO allocations this will deadlock eventually.
>
> Ah, that would affect the non-sleeping case (GFP_KERNEL), right?
>
> Petr T
>
  

Hi Juerg,

On Fri, 31 Mar 2023 07:26:09 +0000
Juerg Haefliger <juergh@proton.me> wrote:

> On Tue, 28 Mar 2023 09:54:35 +0200
> Petr Tesarik <petrtesarik@huaweicloud.com> wrote:
> 
>[...]
> > Anyway, I suspected that the buffers need not be imported into the vc4
> > driver (also hinted by Eric Anholt in a 2018 blog post [1]), and it
> > seems I was right. I encountered the issue with Ubuntu 22.10; I
> > installed latest openSUSE Tumbleweed yesterday, and I was not able to
> > reproduce the issue there, most likely because the Mesa drivers have
> > been fixed meanwhile. This makes the specific case of the Raspberry Pi 4
> > drivers moot. The issue may still affect other combinations of drivers,
> > but I don't have any other real-world example ATM.  
> 
> I'm only seeing this problem with Wayland, no issue when switching Ubuntu to
> X. It seems Tumbleweed is using X by default.

I know; I was the team lead of SUSE low-level graphics engineers until
end of last year... I have just double-checked, but this output of
wayland-info in the GNOME session accessed over RDP is quite convincing:

interface: 'wl_compositor',                              version:  5, name:  1
interface: 'wl_shm',                                     version:  1, name:  2
        formats (fourcc):
        0x48344258 = 'XB4H'
        0x48344241 = 'AB4H'
        0x48345258 = 'XR4H'
        0x48345241 = 'AR4H'
        0x30334258 = 'XB30'
        0x30334241 = 'AB30'
        0x30335258 = 'XR30'
        0x30335241 = 'AR30'
        0x36314752 = 'RG16'
        0x34324258 = 'XB24'
        0x34324241 = 'AB24'
                 1 = 'XR24'
                 0 = 'AR24'
interface: 'wl_output',                                  version:  3, name:  3
        x: 0, y: 0, scale: 1,
        physical_width: 430 mm, physical_height: 270 mm,
        make: 'FUS', model: 'P20W-5 ECO',
        subpixel_orientation: unknown, output_transform: normal,
        mode:
                width: 1680 px, height: 1050 px, refresh: 59.954 Hz,
                flags: current preferred
interface: 'zxdg_output_manager_v1',                     version:  3, name:  4
        xdg_output_v1
                output: 3
                name: 'HDMI-1'
                description: 'Fujitsu Siemens Computers GmbH 20"'
                logical_x: 0, logical_y: 0
                logical_width: 1680, logical_height: 1050
interface: 'wl_data_device_manager',                     version:  3, name:  5
interface: 'zwp_primary_selection_device_manager_v1',    version:  1, name:  6
interface: 'wl_subcompositor',                           version:  1, name:  7
interface: 'xdg_wm_base',                                version:  4, name:  8
interface: 'gtk_shell1',                                 version:  5, name:  9
interface: 'wp_viewporter',                              version:  1, name: 10
interface: 'zwp_pointer_gestures_v1',                    version:  3, name: 11
interface: 'zwp_tablet_manager_v2',                      version:  1, name: 12
interface: 'wl_seat',                                    version:  8, name: 13
        name: seat0
        capabilities: pointer keyboard
        keyboard repeat rate: 33
        keyboard repeat delay: 500
interface: 'zwp_relative_pointer_manager_v1',            version:  1, name: 14
interface: 'zwp_pointer_constraints_v1',                 version:  1, name: 15
interface: 'zxdg_exporter_v1',                           version:  1, name: 16
interface: 'zxdg_importer_v1',                           version:  1, name: 17
interface: 'zwp_linux_dmabuf_v1',                        version:  3, name: 18
        formats (fourcc) and modifiers (names):
        0x48344258 = 'XB4H'; 0x00ffffffffffffff = INVALID
        0x48344241 = 'AB4H'; 0x00ffffffffffffff = INVALID
        0x36314752 = 'RG16'; 0x00ffffffffffffff = INVALID
        0x30334258 = 'XB30'; 0x00ffffffffffffff = INVALID
        0x30335258 = 'XR30'; 0x00ffffffffffffff = INVALID
        0x30334241 = 'AB30'; 0x00ffffffffffffff = INVALID
        0x30335241 = 'AR30'; 0x00ffffffffffffff = INVALID
        0x34324258 = 'XB24'; 0x00ffffffffffffff = INVALID
        0x34325258 = 'XR24'; 0x00ffffffffffffff = INVALID
        0x34324241 = 'AB24'; 0x00ffffffffffffff = INVALID
        0x34325241 = 'AR24'; 0x00ffffffffffffff = INVALID
interface: 'wp_single_pixel_buffer_manager_v1',          version:  1, name: 19
interface: 'zwp_keyboard_shortcuts_inhibit_manager_v1',  version:  1, name: 20
interface: 'zwp_text_input_manager_v3',                  version:  1, name: 21
interface: 'wp_presentation',                            version:  1, name: 22
        presentation clock id: 1 (CLOCK_MONOTONIC)
interface: 'xdg_activation_v1',                          version:  1, name: 23

Petr T
  

On Fri, 31 Mar 2023 11:00:43 +0200
Petr Tesařík <petr@tesarici.cz> wrote:

> Hi Juerg,
>
> On Fri, 31 Mar 2023 07:26:09 +0000
> Juerg Haefliger <juergh@proton.me> wrote:
>
> > On Tue, 28 Mar 2023 09:54:35 +0200
> > Petr Tesarik <petrtesarik@huaweicloud.com> wrote:
> >
> >[...]
> > > Anyway, I suspected that the buffers need not be imported into the vc4
> > > driver (also hinted by Eric Anholt in a 2018 blog post [1]), and it
> > > seems I was right. I encountered the issue with Ubuntu 22.10; I
> > > installed latest openSUSE Tumbleweed yesterday, and I was not able to
> > > reproduce the issue there, most likely because the Mesa drivers have
> > > been fixed meanwhile. This makes the specific case of the Raspberry Pi 4
> > > drivers moot. The issue may still affect other combinations of drivers,
> > > but I don't have any other real-world example ATM.
> >
> > I'm only seeing this problem with Wayland, no issue when switching Ubuntu to
> > X. It seems Tumbleweed is using X by default.
>
> I know; I was the team lead of SUSE low-level graphics engineers until
> end of last year... I have just double-checked, but this output of
> wayland-info in the GNOME session accessed over RDP is quite convincing:

It sure is but how did you get that?? For me it's:
$ wayland-info
failed to create display: No such file or directory

And from strace:
connect(3, {sa_family=AF_UNIX, sun_path="/run/user/1000/wayland-0"}, 27) = -1 ENOENT (No such file or directory)

Which is kind of expected when running X, no?
$ ps -ef | grep -iP 'xorg|wayland'
opensuse  1377  1375  0 09:13 tty2     00:00:16 /usr/bin/Xorg.bin vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -nolisten tcp -background none -noreset -keeptty -novtswitch -verbose 3

What am I missing?

...Juerg



> interface: 'wl_compositor',                              version:  5, name:  1
> interface: 'wl_shm',                                     version:  1, name:  2
>         formats (fourcc):
>         0x48344258 = 'XB4H'
>         0x48344241 = 'AB4H'
>         0x48345258 = 'XR4H'
>         0x48345241 = 'AR4H'
>         0x30334258 = 'XB30'
>         0x30334241 = 'AB30'
>         0x30335258 = 'XR30'
>         0x30335241 = 'AR30'
>         0x36314752 = 'RG16'
>         0x34324258 = 'XB24'
>         0x34324241 = 'AB24'
>                  1 = 'XR24'
>                  0 = 'AR24'
> interface: 'wl_output',                                  version:  3, name:  3
>         x: 0, y: 0, scale: 1,
>         physical_width: 430 mm, physical_height: 270 mm,
>         make: 'FUS', model: 'P20W-5 ECO',
>         subpixel_orientation: unknown, output_transform: normal,
>         mode:
>                 width: 1680 px, height: 1050 px, refresh: 59.954 Hz,
>                 flags: current preferred
> interface: 'zxdg_output_manager_v1',                     version:  3, name:  4
>         xdg_output_v1
>                 output: 3
>                 name: 'HDMI-1'
>                 description: 'Fujitsu Siemens Computers GmbH 20"'
>                 logical_x: 0, logical_y: 0
>                 logical_width: 1680, logical_height: 1050
> interface: 'wl_data_device_manager',                     version:  3, name:  5
> interface: 'zwp_primary_selection_device_manager_v1',    version:  1, name:  6
> interface: 'wl_subcompositor',                           version:  1, name:  7
> interface: 'xdg_wm_base',                                version:  4, name:  8
> interface: 'gtk_shell1',                                 version:  5, name:  9
> interface: 'wp_viewporter',                              version:  1, name: 10
> interface: 'zwp_pointer_gestures_v1',                    version:  3, name: 11
> interface: 'zwp_tablet_manager_v2',                      version:  1, name: 12
> interface: 'wl_seat',                                    version:  8, name: 13
>         name: seat0
>         capabilities: pointer keyboard
>         keyboard repeat rate: 33
>         keyboard repeat delay: 500
> interface: 'zwp_relative_pointer_manager_v1',            version:  1, name: 14
> interface: 'zwp_pointer_constraints_v1',                 version:  1, name: 15
> interface: 'zxdg_exporter_v1',                           version:  1, name: 16
> interface: 'zxdg_importer_v1',                           version:  1, name: 17
> interface: 'zwp_linux_dmabuf_v1',                        version:  3, name: 18
>         formats (fourcc) and modifiers (names):
>         0x48344258 = 'XB4H'; 0x00ffffffffffffff = INVALID
>         0x48344241 = 'AB4H'; 0x00ffffffffffffff = INVALID
>         0x36314752 = 'RG16'; 0x00ffffffffffffff = INVALID
>         0x30334258 = 'XB30'; 0x00ffffffffffffff = INVALID
>         0x30335258 = 'XR30'; 0x00ffffffffffffff = INVALID
>         0x30334241 = 'AB30'; 0x00ffffffffffffff = INVALID
>         0x30335241 = 'AR30'; 0x00ffffffffffffff = INVALID
>         0x34324258 = 'XB24'; 0x00ffffffffffffff = INVALID
>         0x34325258 = 'XR24'; 0x00ffffffffffffff = INVALID
>         0x34324241 = 'AB24'; 0x00ffffffffffffff = INVALID
>         0x34325241 = 'AR24'; 0x00ffffffffffffff = INVALID
> interface: 'wp_single_pixel_buffer_manager_v1',          version:  1, name: 19
> interface: 'zwp_keyboard_shortcuts_inhibit_manager_v1',  version:  1, name: 20
> interface: 'zwp_text_input_manager_v3',                  version:  1, name: 21
> interface: 'wp_presentation',                            version:  1, name: 22
>         presentation clock id: 1 (CLOCK_MONOTONIC)
> interface: 'xdg_activation_v1',                          version:  1, name: 23
>
> Petr T
  

On Tue, Mar 28, 2023 at 09:54:35AM +0200, Petr Tesarik wrote:
> I tend to agree here. However, it's the DMABUF design itself that causes
> some trouble. The buffer is allocated by the v3d driver, which does not
> have the restriction, so the DMA API typically allocates an address
> somewhere near the 4G boundary. Userspace then exports the buffer, sends
> it to another process as a file descriptor and imports it into the vc4
> driver, which requires DMA below 1G. In the beginning, v3d had no idea
> that the buffer would be exported to userspace, much less that it would
> be later imported into vc4.

Then we need to either:

 a) figure out a way to communicate these addressing limitations
 b) find a way to migrate a buffer into other memory, similar to
    how page migration works for page cache

> BTW my testing also suggests that the streaming DMA API is quite
> inefficient, because UAS performance _improved_ with swiotlb=force.
> Sure, this should probably be addressed in the UAS and/or xHCI driver,
> but what I mean is that moving away from swiotlb may even cause
> performance regressions, which is counter-intuitive. At least I would
> _not_ have expected it.

That is indeed very odd.  Are you running with a very slow iommu
driver there?   Or what is the actual use case there in general?

> >> +	gfp = (attrs & DMA_ATTR_MAY_SLEEP) ? GFP_KERNEL : GFP_NOWAIT;
> >> +	slot = kmalloc(sizeof(*slot), gfp | __GFP_NOWARN);
> >> +	if (!slot)
> >> +		goto err;
> >> +
> >> +	slot->orig_addr = orig_addr;
> >> +	slot->alloc_size = alloc_size;
> >> +	slot->page = dma_direct_alloc_pages(dev, PAGE_ALIGN(alloc_size),
> >> +					    &slot->dma_addr, dir,
> >> +					    gfp | __GFP_NOWARN);
> >> +	if (!slot->page)
> >> +		goto err_free_slot;
> > 
> > Without GFP_NOIO allocations this will deadlock eventually.
> 
> Ah, that would affect the non-sleeping case (GFP_KERNEL), right?

Yes.
  

On Tue, Mar 28, 2023 at 02:43:03PM +0200, Petr Tesarik wrote:
> Oh, wait! I can do at least something for devices which do not use
> swiotlb at all.
> 
> If a device does not use bounce buffers, it cannot pass an address
> that belongs to the swiotlb. Consequently, the potentially
> expensive check can be skipped. This avoids the dynamic lookup
> penalty for devices which do not need the swiotlb.
> 
> Note that the counter always remains zero if dma_io_tlb_mem is
> NULL, so the NULL check is not required.

Hmm, that's yet another atomic for each map/unmap, and bloats
struct device.

(Btw, in case anyone is interested, we really need to get started
on moving the dma fields out of struct device into a sub-struct
only allocated for DMA capable busses)
  

On Fri, 7 Apr 2023 07:57:04 +0200
Christoph Hellwig <hch@lst.de> wrote:

> On Tue, Mar 28, 2023 at 02:43:03PM +0200, Petr Tesarik wrote:
> > Oh, wait! I can do at least something for devices which do not use
> > swiotlb at all.
> > 
> > If a device does not use bounce buffers, it cannot pass an address
> > that belongs to the swiotlb. Consequently, the potentially
> > expensive check can be skipped. This avoids the dynamic lookup
> > penalty for devices which do not need the swiotlb.
> > 
> > Note that the counter always remains zero if dma_io_tlb_mem is
> > NULL, so the NULL check is not required.  
> 
> Hmm, that's yet another atomic for each map/unmap, and bloats
> struct device.

I'm not sure how bad it is to bloat struct device. It is already quite
large, e.g. in my x86 build it is 768 bytes (exact size depends on
config options), and nobody seems to be concerned...

Regarding the atomic operations, I am currently testing a slightly
different approach, which merely sets a flag if there are any
dynamically allocated bounce buffers. The atomic check changes to
smp_load_acquire(), and the atomic inc/dec to smp_store_release()
only if the flag changes. That said, if I hammer this path with heavy
parallel I/O, I can still see some performance cost for devices that
use swiotlb, but at least devices that do not need such bounce buffers
seem to be unaffected then.

> (Btw, in case anyone is interested, we really need to get started
> on moving the dma fields out of struct device into a sub-struct
> only allocated for DMA capable busses)

I like this idea. In fact, my WIP topic branch now moves the swiotlb
fields into a separate struct, but I can surely go further and move all
DMA-related fields. I doubt it is worth to allocate it separately,
though. We are talking about replacing some 100 bytes (in the worst
case) with a pointer to a dynamically allocated struct, but the
dynamic allocator adds some overhead. I believe it pays off only if the
vast majority of struct device instances do not need these DMA fields,
but is that really the case?

Petr T
  

On Fri, 7 Apr 2023 07:55:48 +0200
Christoph Hellwig <hch@lst.de> wrote:

> On Tue, Mar 28, 2023 at 09:54:35AM +0200, Petr Tesarik wrote:
> > I tend to agree here. However, it's the DMABUF design itself that causes
> > some trouble. The buffer is allocated by the v3d driver, which does not
> > have the restriction, so the DMA API typically allocates an address
> > somewhere near the 4G boundary. Userspace then exports the buffer, sends
> > it to another process as a file descriptor and imports it into the vc4
> > driver, which requires DMA below 1G. In the beginning, v3d had no idea
> > that the buffer would be exported to userspace, much less that it would
> > be later imported into vc4.  
> 
> Then we need to either:
> 
>  a) figure out a way to communicate these addressing limitations

AFAICS this would require a complete overhaul of the dma-buf userspace
API so that intended imports are communicated at export time. In other
words, it would be quite intrusive. Not my preferrence.

>  b) find a way to migrate a buffer into other memory, similar to
>     how page migration works for page cache

Let me express the idea in my own words to make sure I get it right.
When a DMA buffer is imported, but before it is ultimately pinned in
memory, the importing device driver checks whether the buffer meets its
DMA constraints. If not, it calls a function provided by the exporting
device driver to migrate the buffer. This makes sense, but:

  1) The operation must be implemented in the exporting driver; this
     will take some time.

  2) In theory, there may be no overlap between the exporting device
     and the importing device. OTOH I'm not aware of any real-world
     example, so we can probably return a suitable error code, and
     that's it.

Anyway, I have already written in another reply that my original use
case is moot, because a more recent distribution can do the job without
using dma-buf, so it has been fixed in user space, be it in GNOME,
pipewire, or Mesa (I don't really have to know).

At this point I would go with the assumption that large buffers
allocated by media subsystems will not hit swiotlb. Consequently, I
don't plan to spend more time on this branch of the story.

> > BTW my testing also suggests that the streaming DMA API is quite
> > inefficient, because UAS performance _improved_ with swiotlb=force.
> > Sure, this should probably be addressed in the UAS and/or xHCI driver,
> > but what I mean is that moving away from swiotlb may even cause
> > performance regressions, which is counter-intuitive. At least I would
> > _not_ have expected it.  
> 
> That is indeed very odd.  Are you running with a very slow iommu
> driver there?   Or what is the actual use case there in general?

This was on a Raspberry Pi 4, which does not have any IOMMU. IOW it
looks like copying data around can be faster than sending it straight
to the device. When I have some more time, I must investigate what is
really happening there, because it does not make any sense to me.

Petr T
  

On Fri, Apr 07, 2023 at 12:46:27PM +0200, Petr Tesařík wrote:
> >  b) find a way to migrate a buffer into other memory, similar to
> >     how page migration works for page cache
> 
> Let me express the idea in my own words to make sure I get it right.
> When a DMA buffer is imported, but before it is ultimately pinned in
> memory, the importing device driver checks whether the buffer meets its
> DMA constraints. If not, it calls a function provided by the exporting
> device driver to migrate the buffer.

Yes.

> This makes sense, but:
> 
>   1) The operation must be implemented in the exporting driver; this
>      will take some time.
> 
>   2) In theory, there may be no overlap between the exporting device
>      and the importing device. OTOH I'm not aware of any real-world
>      example, so we can probably return a suitable error code, and
>      that's it.

Indeed.  And if there is no overlap, which as you said is indeed
very unlikely but in theory possible, we could still keep migrating
forther and back.

One important thing that we should do is to consolidate more of the
dma-buf implementation code.  Right now they just seem to be a wild
mess of copy and pasted boilerplate code unfortunately.

> Anyway, I have already written in another reply that my original use
> case is moot, because a more recent distribution can do the job without
> using dma-buf, so it has been fixed in user space, be it in GNOME,
> pipewire, or Mesa (I don't really have to know).
> 
> At this point I would go with the assumption that large buffers
> allocated by media subsystems will not hit swiotlb. Consequently, I
> don't plan to spend more time on this branch of the story.

Sounds fine to me, and thanks for taking the effort so far.

> > > BTW my testing also suggests that the streaming DMA API is quite
> > > inefficient, because UAS performance _improved_ with swiotlb=force.
> > > Sure, this should probably be addressed in the UAS and/or xHCI driver,
> > > but what I mean is that moving away from swiotlb may even cause
> > > performance regressions, which is counter-intuitive. At least I would
> > > _not_ have expected it.  
> > 
> > That is indeed very odd.  Are you running with a very slow iommu
> > driver there?   Or what is the actual use case there in general?
> 
> This was on a Raspberry Pi 4, which does not have any IOMMU. IOW it
> looks like copying data around can be faster than sending it straight
> to the device. When I have some more time, I must investigate what is
> really happening there, because it does not make any sense to me.

If you're not using an IOMMU that doesn't actually make any sense to
me.  swiotlb calls into exactly the same routines as dma-direct does
for the dma setup on each I/O, just after copying the data.  So if you
do have some spare cycles to investigate what is going on here, I'd
be really curious about the results.
  

On 4/7/2023 12:15 PM, Petr Tesařík wrote:
> On Fri, 7 Apr 2023 07:57:04 +0200
> Christoph Hellwig <hch@lst.de> wrote:
>[...]>> (Btw, in case anyone is interested, we really need to get started
>> on moving the dma fields out of struct device into a sub-struct
>> only allocated for DMA capable busses)
> 
> I like this idea. In fact, my WIP topic branch now moves the swiotlb
> fields into a separate struct, but I can surely go further and move all
> DMA-related fields.

I have looked into this now, and it looks like a nice cleanup. The
challenge is to get these patches reviewed by all affected maintainers,
and it would be blocking my work on dynamically allocated bounce buffers.

How about moving only some fields initially (coherent override, cma and
swiotlb)? These few are not used outside kernel/dma, but at least the
swiotlb part makes my patches easier to follow. I can move the rest as
soon as the dynamic patch series is merged.

Petr Tesarik
  

Hi Christoph!

I'd like to follow up on this sub-thread:

On Fri, 7 Apr 2023 12:15:55 +0200
Petr Tesařík <petr@tesarici.cz> wroe:

> On Fri, 7 Apr 2023 07:57:04 +0200
> Christoph Hellwig <hch@lst.de> wrote:
>[...]
> > (Btw, in case anyone is interested, we really need to get started
> > on moving the dma fields out of struct device into a sub-struct
> > only allocated for DMA capable busses)  
> 
> I like this idea. In fact, my WIP topic branch now moves the swiotlb
> fields into a separate struct,

As you have noticed, I have removed that commit again in v2.

The reason is that I'm not sure about the intended goal. I have looked
around for examples of moving fields out of struct device and found
different approaches:

A. struct dev_msi_info
   The MSI fields are merely grouped in a separate struct, which is
   defined in device.h and embedded in struct device. I don't see much
   benefit.

B. struct dev_pm_info
   This struct is also embedded in struct device, but it is defined in
   <linux/pm.h>, which is mentioned in MAINTAINERS. The benefit is that
   further changes are reviewed by this maintainer. The downside is
   that device.h includes pm.h.

C. struct dev_pin_info
   This struct is merely declared in device.h and defined
   pinctrl/devinfo.h (which is not included). Only a pointer to this
   struct is stored in struct device. Of course, the pointer must be
   initialized (and released) somehow.

Here my question: What did you want for DMA fields?

A. Only grouping those fields in their own struct?
B. Or move the definition to another include file (cf. MAINTAINERS)?
C. Or store a pointer in struct device?

Since you mentioned "allocated", it sounds like you want to achieve C,
but:

1. Is it worth the extra dereference for every use?
2. How should the struct be allocated? Presumably not with kmalloc() in
   device_initialize(), because I don't know how to determine if a
   device is DMA capable this low in the call stack. So, should it be
   allocated together with the containing structure? AFAICS this would
   mean changing nearly all device drivers...

As you can see, I need some more guidance from you before I can start
working on this. ;-)

Petr T
  

On 2023-04-21 14:03, Petr Tesařík wrote:
> Hi Christoph!
> 
> I'd like to follow up on this sub-thread:
> 
> On Fri, 7 Apr 2023 12:15:55 +0200
> Petr Tesařík <petr@tesarici.cz> wroe:
> 
>> On Fri, 7 Apr 2023 07:57:04 +0200
>> Christoph Hellwig <hch@lst.de> wrote:
>> [...]
>>> (Btw, in case anyone is interested, we really need to get started
>>> on moving the dma fields out of struct device into a sub-struct
>>> only allocated for DMA capable busses)
>>
>> I like this idea. In fact, my WIP topic branch now moves the swiotlb
>> fields into a separate struct,
> 
> As you have noticed, I have removed that commit again in v2.
> 
> The reason is that I'm not sure about the intended goal. I have looked
> around for examples of moving fields out of struct device and found
> different approaches:
> 
> A. struct dev_msi_info
>     The MSI fields are merely grouped in a separate struct, which is
>     defined in device.h and embedded in struct device. I don't see much
>     benefit.
> 
> B. struct dev_pm_info
>     This struct is also embedded in struct device, but it is defined in
>     <linux/pm.h>, which is mentioned in MAINTAINERS. The benefit is that
>     further changes are reviewed by this maintainer. The downside is
>     that device.h includes pm.h.
> 
> C. struct dev_pin_info
>     This struct is merely declared in device.h and defined
>     pinctrl/devinfo.h (which is not included). Only a pointer to this
>     struct is stored in struct device. Of course, the pointer must be
>     initialized (and released) somehow.
> 
> Here my question: What did you want for DMA fields?
> 
> A. Only grouping those fields in their own struct?
> B. Or move the definition to another include file (cf. MAINTAINERS)?
> C. Or store a pointer in struct device?

dev->dma_parms is already this, and IIRC still has some very old 
comments somewhere about consolidating the other DMA-related fields in 
there.

> Since you mentioned "allocated", it sounds like you want to achieve C,
> but:
> 
> 1. Is it worth the extra dereference for every use?
> 2. How should the struct be allocated? Presumably not with kmalloc() in
>     device_initialize(), because I don't know how to determine if a
>     device is DMA capable this low in the call stack. So, should it be
>     allocated together with the containing structure? AFAICS this would
>     mean changing nearly all device drivers...

The bus code knows whether it's a DMA-capable bus or not, and as such 
should already be providing a .dma_configure method and/or performing 
some initialisation of DMA fields. Many of the ones that would need to 
are already providing dma_parms, in fact.

Thanks,
Robin.

> 
> As you can see, I need some more guidance from you before I can start
> working on this. ;-)
> 
> Petr T
  

Hi Robin,

On Fri, 21 Apr 2023 15:58:18 +0100
Robin Murphy <robin.murphy@arm.com> wrote:

> On 2023-04-21 14:03, Petr Tesařík wrote:
>[...]
> > Here my question: What did you want for DMA fields?
> > 
> > A. Only grouping those fields in their own struct?
> > B. Or move the definition to another include file (cf. MAINTAINERS)?
> > C. Or store a pointer in struct device?  
> 
> dev->dma_parms is already this, and IIRC still has some very old 
> comments somewhere about consolidating the other DMA-related fields in 
> there.

Thank you for the hint! I have actually seen dma_parms, but since it
can be NULL and was initialized from various drivers, it did not occur
to me that NULL simply means not DMA-capable.

This is really helpful.

Petr T

> > Since you mentioned "allocated", it sounds like you want to achieve C,
> > but:
> > 
> > 1. Is it worth the extra dereference for every use?
> > 2. How should the struct be allocated? Presumably not with kmalloc() in
> >     device_initialize(), because I don't know how to determine if a
> >     device is DMA capable this low in the call stack. So, should it be
> >     allocated together with the containing structure? AFAICS this would
> >     mean changing nearly all device drivers...  
> 
> The bus code knows whether it's a DMA-capable bus or not, and as such 
> should already be providing a .dma_configure method and/or performing 
> some initialisation of DMA fields. Many of the ones that would need to 
> are already providing dma_parms, in fact.
> 
> Thanks,
> Robin.
> 
> > 
> > As you can see, I need some more guidance from you before I can start
> > working on this. ;-)
> > 
> > Petr T
  

On Fri, Apr 21, 2023 at 05:09:34PM +0200, Petr Tesařík wrote:
> > > A. Only grouping those fields in their own struct?
> > > B. Or move the definition to another include file (cf. MAINTAINERS)?
> > > C. Or store a pointer in struct device?  
> > 
> > dev->dma_parms is already this, and IIRC still has some very old 
> > comments somewhere about consolidating the other DMA-related fields in 
> > there.
> 
> Thank you for the hint! I have actually seen dma_parms, but since it
> can be NULL and was initialized from various drivers, it did not occur
> to me that NULL simply means not DMA-capable.

Yes, dma_parms are still optional.  A much better hint is the dma_mask
itself, which for historic reasons is implemented in a completely
awfull way as a pointer to something stored in the containing struture,
which all kinds of platform devices or minor buses doing nasty hacks
there.
  

On Thu, 06 Apr 2023 11:44:55 +0000
Juerg Haefliger <juergh@proton.me> wrote:

> On Fri, 31 Mar 2023 11:00:43 +0200
> Petr Tesařík <petr@tesarici.cz> wrote:
> 
> > Hi Juerg,
> >
> > On Fri, 31 Mar 2023 07:26:09 +0000
> > Juerg Haefliger <juergh@proton.me> wrote:
> >  
> > > On Tue, 28 Mar 2023 09:54:35 +0200
> > > Petr Tesarik <petrtesarik@huaweicloud.com> wrote:
> > >
> > >[...]  
> > > > Anyway, I suspected that the buffers need not be imported into the vc4
> > > > driver (also hinted by Eric Anholt in a 2018 blog post [1]), and it
> > > > seems I was right. I encountered the issue with Ubuntu 22.10; I
> > > > installed latest openSUSE Tumbleweed yesterday, and I was not able to
> > > > reproduce the issue there, most likely because the Mesa drivers have
> > > > been fixed meanwhile. This makes the specific case of the Raspberry Pi 4
> > > > drivers moot. The issue may still affect other combinations of drivers,
> > > > but I don't have any other real-world example ATM.  
> > >
> > > I'm only seeing this problem with Wayland, no issue when switching Ubuntu to
> > > X. It seems Tumbleweed is using X by default.  
> >
> > I know; I was the team lead of SUSE low-level graphics engineers until
> > end of last year... I have just double-checked, but this output of
> > wayland-info in the GNOME session accessed over RDP is quite convincing:  
> 
> It sure is but how did you get that??

I'm sorry for late reply.

Yes, the default is GNOME on X.org, but you can change it on the GDM
login screen. After choosing your account, a settings gear icon appears
in the lower right corner. Click on it and choose GNOME. That starts a
Wayland session.

HTH
Petr T