[V4,00/17] io_uring/ublk: add IORING_OP_FUSED_CMD

  Hello Jens,

Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
64byte SQE(slave) is another normal 64byte OP. For any OP which needs
to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
and its ->issue() can retrieve/import buffer from master request's
fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
submits slave OP just like normal OP issued from userspace, that said,
SQE order is kept, and batching handling is done too.

Please see detailed design in commit log of the 2th patch, and one big
point is how to handle buffer ownership.

With this way, it is easy to support zero copy for ublk/fuse device.

Basically userspace can specify any sub-buffer of the ublk block request
buffer from the fused command just by setting 'offset/len'
in the slave SQE for running slave OP. This way is flexible to implement
io mapping: mirror, stripped, ...

The 4th & 5th patches enable fused slave support for the following OPs:

	OP_READ/OP_WRITE
	OP_SEND/OP_RECV/OP_SEND_ZC

The other ublk patches cleans ublk driver and implement fused command
for supporting zero copy.

Follows userspace code, which supports 128byte SQE fused command only:

https://github.com/ming1/ubdsrv/tree/fused-cmd-zc-v2

All three(loop, nbd and qcow2) ublk targets have supported zero copy by passing:

	ublk add -t [loop|nbd|qcow2] -z .... 

Basic fs mount/kernel building and builtin test are done, and also not
observe regression on xfstest test over ublk-loop with zero copy.

Also add liburing test case for covering fused command based on miniublk
of blktest(supports 64byte normal SQE only)

https://github.com/ming1/liburing/commits/fused_cmd_miniublk

Performance improvement is obvious on memory bandwidth related workloads,
such as, 1~2X improvement on 64K/512K BS IO test on loop with ramfs backing file.
ublk-null shows 5X IOPS improvement on big BS test when the copy is avoided.

Please review and consider for v6.4.

V4:
	- improve APIs naming(patch 1 ~ 4)
	- improve documents and commit log(patch 2)
	- add buffer direction bit to opdef, suggested by Jens(patch 2)
	- add ublk zero copy document for cover: technical requirements(most related with
	buffer lifetime), and explains why splice isn't good and how fused command solves it(patch 17)
	- fix sparse warning(patch 7)
	- supports 64byte SQE fused command(patch 3)

V3:
	- fix build warning reported by kernel test robot
	- drop patch for checking fused flags on existed drivers with
	  ->uring_command(), which isn't necessary, since we do not do that
      when adding new ioctl or uring command
    - inline io_init_rq() for core code, so just export io_init_slave_req
	- return result of failed slave request unconditionally since REQ_F_CQE_SKIP
	will be cleared
	- pass xfstest over ublk-loop

V2:
	- don't resue io_mapped_ubuf (io_uring)
	- remove REQ_F_FUSED_MASTER_BIT (io_uring)
	- fix compile warning (io_uring)
	- rebase on v6.3-rc1 (io_uring)
	- grabbing io request reference when handling fused command 
	- simplify ublk_copy_user_pages() by iov iterator
	- add read()/write() for userspace to read/write ublk io buffer, so
	that some corner cases(read zero, passthrough request(report zones)) can
	be handled easily in case of zero copy; this way also helps to switch to
	zero copy completely
	- misc cleanup


Ming Lei (17):
  io_uring: increase io_kiocb->flags into 64bit
  io_uring: add IORING_OP_FUSED_CMD
  io_uring: support normal SQE for fused command
  io_uring: support OP_READ/OP_WRITE for fused slave request
  io_uring: support OP_SEND_ZC/OP_RECV for fused slave request
  block: ublk_drv: mark device as LIVE before adding disk
  block: ublk_drv: add common exit handling
  block: ublk_drv: don't consider flush request in map/unmap io
  block: ublk_drv: add two helpers to clean up map/unmap request
  block: ublk_drv: clean up several helpers
  block: ublk_drv: cleanup 'struct ublk_map_data'
  block: ublk_drv: cleanup ublk_copy_user_pages
  block: ublk_drv: grab request reference when the request is handled by
    userspace
  block: ublk_drv: support to copy any part of request pages
  block: ublk_drv: add read()/write() support for ublk char device
  block: ublk_drv: don't check buffer in case of zero copy
  block: ublk_drv: apply io_uring FUSED_CMD for supporting zero copy

 Documentation/block/ublk.rst   | 126 ++++++-
 drivers/block/ublk_drv.c       | 605 ++++++++++++++++++++++++++-------
 include/linux/io_uring.h       |  49 ++-
 include/linux/io_uring_types.h |  80 +++--
 include/uapi/linux/io_uring.h  |   9 +-
 include/uapi/linux/ublk_cmd.h  |  37 +-
 io_uring/Makefile              |   2 +-
 io_uring/fused_cmd.c           | 259 ++++++++++++++
 io_uring/fused_cmd.h           |  11 +
 io_uring/io_uring.c            |  50 ++-
 io_uring/io_uring.h            |   4 +
 io_uring/net.c                 |  30 +-
 io_uring/opdef.c               |  22 ++
 io_uring/opdef.h               |   7 +
 io_uring/rw.c                  |  20 ++
 15 files changed, 1132 insertions(+), 179 deletions(-)
 create mode 100644 io_uring/fused_cmd.c
 create mode 100644 io_uring/fused_cmd.h
  

Ming Lei wrote:
> Hello Jens,
> 
> Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> and its ->issue() can retrieve/import buffer from master request's
> fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> submits slave OP just like normal OP issued from userspace, that said,
> SQE order is kept, and batching handling is done too.

Hi Ming,

io_uring and ublk are starting to be more on my radar these days. I
wanted to take a look at this series, but could not get past the
distracting "master"/"slave" terminology in this lead-in paragraph let
alone start looking at patches.

Frankly, the description sounds more like "head"/"tail", or even
"fuse0"/"fuse1" because, for example, who is to say you might not have
larger fused ops in the future and need terminology to address
"fuse{0,1,2,3}"?

Once that's fixed up I can take a look at forwarding on to others that
might be interested in this use case.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/coding-style.rst#n338

Thanks in advance for fixing that up!
  

Hi Dan,

On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
> Ming Lei wrote:
> > Hello Jens,
> > 
> > Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> > be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> > 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> > to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> > and its ->issue() can retrieve/import buffer from master request's
> > fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> > this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> > submits slave OP just like normal OP issued from userspace, that said,
> > SQE order is kept, and batching handling is done too.
> 
> Hi Ming,
> 
> io_uring and ublk are starting to be more on my radar these days. I
> wanted to take a look at this series, but could not get past the
> distracting "master"/"slave" terminology in this lead-in paragraph let
> alone start looking at patches.
> 
> Frankly, the description sounds more like "head"/"tail", or even
> "fuse0"/"fuse1" because, for example, who is to say you might not have

The term "master/slave" is from patches.

The master command not only provides buffer for slave request, but also requires
slave request for serving master command, and master command is always completed
after all slave request are done.

That is why it is named as master/slave. Actually Jens raised the similar concern
and I hate the name too, but it is always hard to figure out perfect name, or
any other name for reflecting the relation? (head/tail, fuse0/1 can't
do that, IMO)

> larger fused ops in the future and need terminology to address
> "fuse{0,1,2,3}"?

Yeah, definitely, the interface can be extended in future to support
multiple "slave" requests.

Thanks,
Ming
  

On 3/27/23 7:16 PM, Ming Lei wrote:
> Hi Dan,
> 
> On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
>> Ming Lei wrote:
>>> Hello Jens,
>>>
>>> Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
>>> be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
>>> 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
>>> to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
>>> and its ->issue() can retrieve/import buffer from master request's
>>> fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
>>> this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
>>> submits slave OP just like normal OP issued from userspace, that said,
>>> SQE order is kept, and batching handling is done too.
>>
>> Hi Ming,
>>
>> io_uring and ublk are starting to be more on my radar these days. I
>> wanted to take a look at this series, but could not get past the
>> distracting "master"/"slave" terminology in this lead-in paragraph let
>> alone start looking at patches.
>>
>> Frankly, the description sounds more like "head"/"tail", or even
>> "fuse0"/"fuse1" because, for example, who is to say you might not have
> 
> The term "master/slave" is from patches.
> 
> The master command not only provides buffer for slave request, but also requires
> slave request for serving master command, and master command is always completed
> after all slave request are done.
> 
> That is why it is named as master/slave. Actually Jens raised the similar concern
> and I hate the name too, but it is always hard to figure out perfect name, or
> any other name for reflecting the relation? (head/tail, fuse0/1 can't
> do that, IMO)

Indeed. What about primary/secondary? And it'd be quite possible to have
multiple secondaries too.
  

Ming Lei wrote:
> Hi Dan,
> 
> On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
> > Ming Lei wrote:
> > > Hello Jens,
> > > 
> > > Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> > > be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> > > 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> > > to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> > > and its ->issue() can retrieve/import buffer from master request's
> > > fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> > > this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> > > submits slave OP just like normal OP issued from userspace, that said,
> > > SQE order is kept, and batching handling is done too.
> > 
> > Hi Ming,
> > 
> > io_uring and ublk are starting to be more on my radar these days. I
> > wanted to take a look at this series, but could not get past the
> > distracting "master"/"slave" terminology in this lead-in paragraph let
> > alone start looking at patches.
> > 
> > Frankly, the description sounds more like "head"/"tail", or even
> > "fuse0"/"fuse1" because, for example, who is to say you might not have
> 
> The term "master/slave" is from patches.

From what patches?

I did not understand this explanation either:

https://lore.kernel.org/all/ZBXjH5ipRUwtYIVF@ovpn-8-18.pek2.redhat.com/

> The master command not only provides buffer for slave request, but also requires
> slave request for serving master command, and master command is always completed
> after all slave request are done.

In terms of core kernel concepts that description aligns more with
idiomatic "parent"/"child" relationships where the child object holds a
reference on the parent.

> That is why it is named as master/slave.

That explanation did not clarify.

> Actually Jens raised the similar concern

Thanks Jens!

> ...and I hate the name too, but it is always hard to figure out
> perfect name, or any other name for reflecting the relation?
> (head/tail, fuse0/1 can't do that, IMO)

Naming is hard, and master/slave is not appropriate so this needs a new
name. The reason I mentioned "head"/"tail" is not for ring buffer
purposes but more for its similarity to pages and folios where the folio
is not unreferenced until all tail pages are unreferenced.

In short there are several options that add more clarity and avoid
running afoul of coding-style.

> > larger fused ops in the future and need terminology to address
> > "fuse{0,1,2,3}"?
> 
> Yeah, definitely, the interface can be extended in future to support
> multiple "slave" requests.

Right, so why not just name them fuse0,1...n and specify that fuse0 is
the head of a fused op?
  

On Mon, Mar 27, 2023 at 07:29:36PM -0600, Jens Axboe wrote:
> On 3/27/23 7:16 PM, Ming Lei wrote:
> > Hi Dan,
> > 
> > On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
> >> Ming Lei wrote:
> >>> Hello Jens,
> >>>
> >>> Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> >>> be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> >>> 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> >>> to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> >>> and its ->issue() can retrieve/import buffer from master request's
> >>> fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> >>> this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> >>> submits slave OP just like normal OP issued from userspace, that said,
> >>> SQE order is kept, and batching handling is done too.
> >>
> >> Hi Ming,
> >>
> >> io_uring and ublk are starting to be more on my radar these days. I
> >> wanted to take a look at this series, but could not get past the
> >> distracting "master"/"slave" terminology in this lead-in paragraph let
> >> alone start looking at patches.
> >>
> >> Frankly, the description sounds more like "head"/"tail", or even
> >> "fuse0"/"fuse1" because, for example, who is to say you might not have
> > 
> > The term "master/slave" is from patches.
> > 
> > The master command not only provides buffer for slave request, but also requires
> > slave request for serving master command, and master command is always completed
> > after all slave request are done.
> > 
> > That is why it is named as master/slave. Actually Jens raised the similar concern
> > and I hate the name too, but it is always hard to figure out perfect name, or
> > any other name for reflecting the relation? (head/tail, fuse0/1 can't
> > do that, IMO)
> 
> Indeed. What about primary/secondary? And it'd be quite possible to have
> multiple secondaries too.

OK, I will take primary/secondary in V5 if no better name is suggested.


Thanks,
Ming
  

On Mon, Mar 27, 2023 at 06:31:37PM -0700, Dan Williams wrote:
> Ming Lei wrote:
> > Hi Dan,
> > 
> > On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
> > > Ming Lei wrote:
> > > > Hello Jens,
> > > > 
> > > > Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> > > > be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> > > > 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> > > > to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> > > > and its ->issue() can retrieve/import buffer from master request's
> > > > fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> > > > this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> > > > submits slave OP just like normal OP issued from userspace, that said,
> > > > SQE order is kept, and batching handling is done too.
> > > 
> > > Hi Ming,
> > > 
> > > io_uring and ublk are starting to be more on my radar these days. I
> > > wanted to take a look at this series, but could not get past the
> > > distracting "master"/"slave" terminology in this lead-in paragraph let
> > > alone start looking at patches.
> > > 
> > > Frankly, the description sounds more like "head"/"tail", or even
> > > "fuse0"/"fuse1" because, for example, who is to say you might not have
> > 
> > The term "master/slave" is from patches.
> 
> From what patches?

https://lore.kernel.org/linux-block/20230324135808.855245-3-ming.lei@redhat.com/T/#u

> 
> I did not understand this explanation either:
> 
> https://lore.kernel.org/all/ZBXjH5ipRUwtYIVF@ovpn-8-18.pek2.redhat.com/

Jens just suggested primary/secondary, which looks better, and I will
use them in this thread and next version.

> 
> > The master command not only provides buffer for slave request, but also requires
> > slave request for serving master command, and master command is always completed
> > after all slave request are done.
> 
> In terms of core kernel concepts that description aligns more with
> idiomatic "parent"/"child" relationships where the child object holds a
> reference on the parent.

Yeah, holding reference is true for both two relationships.

But "parent"/"child" relationship is often one long-time relation, but here
both requests are short-time objects, just the secondary requests need to
grab primary command buffer for running IO. After secondary requests IO
is done, the relation is over. So it is sort of temporary/short-term relation,
like contract.

Also the buffer meta(bvec) data are readable for all secondary requests,
and secondary requests have to use buffer in the primary command allowed
direction. So the relation is very limited.

> 
> > That is why it is named as master/slave.
> 
> That explanation did not clarify.

Hope the above words help.

> 
> > Actually Jens raised the similar concern
> 
> Thanks Jens!
> 
> > ...and I hate the name too, but it is always hard to figure out
> > perfect name, or any other name for reflecting the relation?
> > (head/tail, fuse0/1 can't do that, IMO)
> 
> Naming is hard, and master/slave is not appropriate so this needs a new
> name. The reason I mentioned "head"/"tail" is not for ring buffer
> purposes but more for its similarity to pages and folios where the folio
> is not unreferenced until all tail pages are unreferenced.
> 
> In short there are several options that add more clarity and avoid
> running afoul of coding-style.
> 
> > > larger fused ops in the future and need terminology to address
> > > "fuse{0,1,2,3}"?
> > 
> > Yeah, definitely, the interface can be extended in future to support
> > multiple "slave" requests.
> 
> Right, so why not just name them fuse0,1...n and specify that fuse0 is
> the head of a fused op?

fuse0, 1...n often means all these objects sharing common property, such
as, all are objects of same class. However, here we do know primary is
completely different with secondary.


Thanks,
Ming
  

On 2023/3/28 09:16, Ming Lei wrote:
> Hi Dan,
> 
> On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
>> Ming Lei wrote:
>>> Hello Jens,
>>>
>>> Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
>>> be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
>>> 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
>>> to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
>>> and its ->issue() can retrieve/import buffer from master request's
>>> fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
>>> this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
>>> submits slave OP just like normal OP issued from userspace, that said,
>>> SQE order is kept, and batching handling is done too.
>>
>> Hi Ming,
>>
>> io_uring and ublk are starting to be more on my radar these days. I
>> wanted to take a look at this series, but could not get past the
>> distracting "master"/"slave" terminology in this lead-in paragraph let
>> alone start looking at patches.
>>
>> Frankly, the description sounds more like "head"/"tail", or even
>> "fuse0"/"fuse1" because, for example, who is to say you might not have
> 
> The term "master/slave" is from patches.
> 
> The master command not only provides buffer for slave request, but also requires
> slave request for serving master command, and master command is always completed
> after all slave request are done.
> 
> That is why it is named as master/slave. Actually Jens raised the similar concern
> and I hate the name too, but it is always hard to figure out perfect name, or
> any other name for reflecting the relation? (head/tail, fuse0/1 can't
> do that, IMO)
> 
>> larger fused ops in the future and need terminology to address
>> "fuse{0,1,2,3}"?
> 
> Yeah, definitely, the interface can be extended in future to support
> multiple "slave" requests.

I guess master/slave (especially now) have bad meaning to
English-language guys so it's better to avoid it.

Thanks,
Gao Xiang

> 
> Thanks,
> Ming
  

On Tue, Mar 28, 2023 at 11:13:53AM +0800, Gao Xiang wrote:
> 
> 
> On 2023/3/28 09:16, Ming Lei wrote:
> > Hi Dan,
> > 
> > On Mon, Mar 27, 2023 at 05:36:33PM -0700, Dan Williams wrote:
> > > Ming Lei wrote:
> > > > Hello Jens,
> > > > 
> > > > Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
> > > > be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
> > > > 64byte SQE(slave) is another normal 64byte OP. For any OP which needs
> > > > to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
> > > > and its ->issue() can retrieve/import buffer from master request's
> > > > fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
> > > > this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
> > > > submits slave OP just like normal OP issued from userspace, that said,
> > > > SQE order is kept, and batching handling is done too.
> > > 
> > > Hi Ming,
> > > 
> > > io_uring and ublk are starting to be more on my radar these days. I
> > > wanted to take a look at this series, but could not get past the
> > > distracting "master"/"slave" terminology in this lead-in paragraph let
> > > alone start looking at patches.
> > > 
> > > Frankly, the description sounds more like "head"/"tail", or even
> > > "fuse0"/"fuse1" because, for example, who is to say you might not have
> > 
> > The term "master/slave" is from patches.
> > 
> > The master command not only provides buffer for slave request, but also requires
> > slave request for serving master command, and master command is always completed
> > after all slave request are done.
> > 
> > That is why it is named as master/slave. Actually Jens raised the similar concern
> > and I hate the name too, but it is always hard to figure out perfect name, or
> > any other name for reflecting the relation? (head/tail, fuse0/1 can't
> > do that, IMO)
> > 
> > > larger fused ops in the future and need terminology to address
> > > "fuse{0,1,2,3}"?
> > 
> > Yeah, definitely, the interface can be extended in future to support
> > multiple "slave" requests.
> 
> I guess master/slave (especially now) have bad meaning to
> English-language guys so it's better to avoid it.

Absolutely no offense given English isn't my native language, so
let's move on with V5.


Thanks,
Ming
  

On Fri, Mar 24, 2023 at 09:57:51PM +0800, Ming Lei wrote:
>Hello Jens,
>
>Add IORING_OP_FUSED_CMD, it is one special URING_CMD, which has to
>be SQE128. The 1st SQE(master) is one 64byte URING_CMD, and the 2nd
>64byte SQE(slave) is another normal 64byte OP. For any OP which needs
>to support slave OP, io_issue_defs[op].fused_slave needs to be set as 1,
>and its ->issue() can retrieve/import buffer from master request's
>fused_cmd_kbuf. The slave OP is actually submitted from kernel, part of
>this idea is from Xiaoguang's ublk ebpf patchset, but this patchset
>submits slave OP just like normal OP issued from userspace, that said,
>SQE order is kept, and batching handling is done too.
>
>Please see detailed design in commit log of the 2th patch, and one big
>point is how to handle buffer ownership.
>
>With this way, it is easy to support zero copy for ublk/fuse device.
>
>Basically userspace can specify any sub-buffer of the ublk block request
>buffer from the fused command just by setting 'offset/len'
>in the slave SQE for running slave OP. 

Wondering if this new OP can also be used to do larger IO (than
device limit) on nvme-passthrough?
For example, 1MB IO on NVMe than has 512k or 256K maximum transfer
size.
  

Ming Lei wrote:
> Jens just suggested primary/secondary, which looks better, and I will
> use them in this thread and next version.

Sounds good, thank you.