block: introduce content activity based ioprio

  From: Zhaoyang Huang <zhaoyang.huang@unisoc.com>

Currently, request's ioprio are set via task's schedule priority(when no
blkcg configured), which has high priority tasks possess the privilege on
both of CPU and IO scheduling.
This commit try to introduce a feature which adjust the request ioprio
based on the folio's activity. The original idea comes from LRU_GEN
which provides more precised folio activity than before. This commit try
to adjust the request's ioprio when certain part of its folios are hot,
which indicate that this request carry important contents and need be
scheduled ealier.

This commit is verified via bellowing script[1] on a v6.6 android system
and get significant improved fault time as expected[2] while dd's cost
also shrink from 55s to 40s.

1. fault_latency.bin is an ebpf based test tool which measure all task's
   iowait latency during page fault when scheduled out/in.
2. costmem generate page fault by mmaping a file and access the VA.
3. dd generate concurrent vfs io.

[1]
/fault_latency.bin 1 5 > /data/dd_costmem &
costmem -c0 -a2048000 -b128000 -o0 1>/dev/null &
costmem -c0 -a2048000 -b128000 -o0 1>/dev/null &
costmem -c0 -a2048000 -b128000 -o0 1>/dev/null &
costmem -c0 -a2048000 -b128000 -o0 1>/dev/null &
dd if=/dev/block/sda of=/data/ddtest bs=1024 count=2048000 &
dd if=/dev/block/sda of=/data/ddtest1 bs=1024 count=2048000 &
dd if=/dev/block/sda of=/data/ddtest2 bs=1024 count=2048000 &
dd if=/dev/block/sda of=/data/ddtest3 bs=1024 count=2048000

[2]
                       mainline		commit
io wait                836us            156us

This commit is also verified by the case of launching camera APP which is
usually considered as heavy working load on both of memory and IO, which
shows 12%-24% improvement.

		ttl = 0		ttl = 50	ttl = 100
mainline        2267ms		2420ms		2316ms
commit          1992ms          1806ms          1998ms

Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
---
 block/Kconfig               |  8 ++++++++
 block/bio.c                 | 10 ++++++++++
 block/blk-mq.c              | 21 +++++++++++++++++++++
 include/uapi/linux/ioprio.h | 20 +++++++++++++++-----
 4 files changed, 54 insertions(+), 5 deletions(-)
  

On Tue, Jan 23, 2024 at 05:33:52PM +0800, zhaoyang.huang wrote:
>  #define ALLOC_CACHE_MAX		256
> @@ -1069,12 +1070,21 @@ EXPORT_SYMBOL_GPL(bio_add_zone_append_page);
>  void __bio_add_page(struct bio *bio, struct page *page,
>  		unsigned int len, unsigned int off)
>  {
> +	int class, level, hint, activity;
> +
> +	class = IOPRIO_PRIO_CLASS(bio->bi_ioprio);
> +	level = IOPRIO_PRIO_LEVEL(bio->bi_ioprio);
> +	hint = IOPRIO_PRIO_HINT(bio->bi_ioprio);
> +	activity = IOPRIO_PRIO_ACTIVITY(bio->bi_ioprio);
> +
>  	WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED));
>  	WARN_ON_ONCE(bio_full(bio, len));
>  
>  	bvec_set_page(&bio->bi_io_vec[bio->bi_vcnt], page, len, off);
>  	bio->bi_iter.bi_size += len;
>  	bio->bi_vcnt++;
> +	activity += (bio->bi_vcnt <= IOPRIO_NR_ACTIVITY && PageWorkingset(page)) ? 1 : 0;

The block layer must not look at page bits.  I've fixed all this crap
with a lot of work and we're not going to re-add it for another qute
hack.  The place to figure out any kind of I/O priority is the file
system (preferably using generic helpers).
  

On 1/23/24 01:33, zhaoyang.huang wrote:
> +	activity += (bio->bi_vcnt <= IOPRIO_NR_ACTIVITY && PageWorkingset(page)) ? 1 : 0;
> +	bio->bi_ioprio = IOPRIO_PRIO_VALUE_ACTIVITY(class, level, hint, activity);

Shouldn't any code that is related to the page cache occur in filesystem code
(fs/) instead of in the block layer (block/)?

Thanks,

Bart.
  

On Tue, Jan 23, 2024 at 9:06 PM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Tue, Jan 23, 2024 at 05:33:52PM +0800, zhaoyang.huang wrote:
> >  #define ALLOC_CACHE_MAX              256
> > @@ -1069,12 +1070,21 @@ EXPORT_SYMBOL_GPL(bio_add_zone_append_page);
> >  void __bio_add_page(struct bio *bio, struct page *page,
> >               unsigned int len, unsigned int off)
> >  {
> > +     int class, level, hint, activity;
> > +
> > +     class = IOPRIO_PRIO_CLASS(bio->bi_ioprio);
> > +     level = IOPRIO_PRIO_LEVEL(bio->bi_ioprio);
> > +     hint = IOPRIO_PRIO_HINT(bio->bi_ioprio);
> > +     activity = IOPRIO_PRIO_ACTIVITY(bio->bi_ioprio);
> > +
> >       WARN_ON_ONCE(bio_flagged(bio, BIO_CLONED));
> >       WARN_ON_ONCE(bio_full(bio, len));
> >
> >       bvec_set_page(&bio->bi_io_vec[bio->bi_vcnt], page, len, off);
> >       bio->bi_iter.bi_size += len;
> >       bio->bi_vcnt++;
> > +     activity += (bio->bi_vcnt <= IOPRIO_NR_ACTIVITY && PageWorkingset(page)) ? 1 : 0;
>
> The block layer must not look at page bits.  I've fixed all this crap
> with a lot of work and we're not going to re-add it for another qute
> hack.  The place to figure out any kind of I/O priority is the file
> system (preferably using generic helpers).
ok, will use helper function in next version
>