diff mbox series

[RESEND] fs/aio: Replace kmap{,_atomic}() with kmap_local_page()

Message ID 20221016150656.5803-1-fmdefrancesco@gmail.com
State New
Headers
Series [RESEND] fs/aio: Replace kmap{,_atomic}() with kmap_local_page() |

Commit Message

Fabio M. De Francesco Oct. 16, 2022, 3:06 p.m. UTC 
  The use of kmap() and kmap_atomic() are being deprecated in favor of
kmap_local_page().

There are two main problems with kmap(): (1) It comes with an overhead as
the mapping space is restricted and protected by a global lock for
synchronization and (2) it also requires global TLB invalidation when the
kmap’s pool wraps and it might block when the mapping space is fully
utilized until a slot becomes available.

With kmap_local_page() the mappings are per thread, CPU local, can take
page faults, and can be called from any context (including interrupts).
It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
the tasks can be preempted and, when they are scheduled to run again, the
kernel virtual addresses are restored and still valid.

Since its use in fs/aio.c is safe everywhere, it should be preferred.

Therefore, replace kmap() and kmap_atomic() with kmap_local_page() in
fs/aio.c.

Tested with xfstests on a QEMU/KVM x86_32 VM, 6GB RAM, booting a kernel
with HIGHMEM64GB enabled.

Cc: "Venkataramanan, Anirudh" <anirudh.venkataramanan@intel.com>
Suggested-by: Ira Weiny <ira.weiny@intel.com>
Reviewed-by: Ira Weiny <ira.weiny@intel.com>
Signed-off-by: Fabio M. De Francesco <fmdefrancesco@gmail.com>
---

I've tested with "./check -g aio". The tests in this group fail 3/26
times, with and without my patch. Therefore, these changes don't introduce
further errors. I'm not aware of any further tests I may run, so that
any suggestions would be precious and much appreciated :-)

I'm resending this patch because some recipients were missing in the 
previous submissions. In the meantime I'm also adding some more information 
in the commit message. There are no changes in the code.

 fs/aio.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)

Comments

Jeff Moyer Oct. 19, 2022, 3:41 p.m. UTC | #1 
"Fabio M. De Francesco" <fmdefrancesco@gmail.com> writes:

> The use of kmap() and kmap_atomic() are being deprecated in favor of
> kmap_local_page().
>
> There are two main problems with kmap(): (1) It comes with an overhead as
> the mapping space is restricted and protected by a global lock for
> synchronization and (2) it also requires global TLB invalidation when the
> kmap’s pool wraps and it might block when the mapping space is fully
> utilized until a slot becomes available.
>
> With kmap_local_page() the mappings are per thread, CPU local, can take
> page faults, and can be called from any context (including interrupts).
> It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
> the tasks can be preempted and, when they are scheduled to run again, the
> kernel virtual addresses are restored and still valid.
>
> Since its use in fs/aio.c is safe everywhere, it should be preferred.

That sentence is very ambiguous.  I don't know what "its" refers to, and
I'm not sure what "safe" means in this context.

The patch looks okay to me.

Reviewed-by: Jeff Moyer <jmoyer@redhat.com>
Fabio M. De Francesco Oct. 19, 2022, 6:52 p.m. UTC | #2 
On Wednesday, October 19, 2022 5:41:21 PM CEST Jeff Moyer wrote:
> "Fabio M. De Francesco" <fmdefrancesco@gmail.com> writes:
> 
> > The use of kmap() and kmap_atomic() are being deprecated in favor of
> > kmap_local_page().
> >
> > There are two main problems with kmap(): (1) It comes with an overhead as
> > the mapping space is restricted and protected by a global lock for
> > synchronization and (2) it also requires global TLB invalidation when the
> > kmap’s pool wraps and it might block when the mapping space is fully
> > utilized until a slot becomes available.
> >
> > With kmap_local_page() the mappings are per thread, CPU local, can take
> > page faults, and can be called from any context (including interrupts).
> > It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
> > the tasks can be preempted and, when they are scheduled to run again, the
> > kernel virtual addresses are restored and still valid.
> >
> > Since its use in fs/aio.c is safe everywhere, it should be preferred.
> 
> That sentence is very ambiguous.  I don't know what "its" refers to, and
> I'm not sure what "safe" means in this context.

I'm sorry for not being clearer.

"its use" means "the use of kmap_local_page()". Few lines above you may also 
see "It is faster", meaning "kmap_local_page() is faster".

The "safety" is a very concise way to assert that I've checked, by code 
inspection and by testing (as it is better detailed some lines below) that 
these conversions (1) don't break any of the rules of use of local mapping 
when converting kmap() (please read highmem.rst about these) and (2) the call 
sites of kmap_atomic() didn't rely on its side effects (pagefaults disable and 
potential preemption disables). 

Therefore, you may read it as it was: "The use of kmap_local_page() in fs/
aio.c has been carefully checked to assure that the conversions won't break 
the code, therefore the newer API is preferred".

I hope it makes my argument clearer.

> 
> The patch looks okay to me.
> 
> Reviewed-by: Jeff Moyer <jmoyer@redhat.com>
> 

Thank you so much for the  "Reviewed-by" tag.

Regards,

Fabio
Jeff Moyer Oct. 19, 2022, 7:07 p.m. UTC | #3 
"Fabio M. De Francesco" <fmdefrancesco@gmail.com> writes:

> On Wednesday, October 19, 2022 5:41:21 PM CEST Jeff Moyer wrote:
>> "Fabio M. De Francesco" <fmdefrancesco@gmail.com> writes:
>> 
>> > The use of kmap() and kmap_atomic() are being deprecated in favor of
>> > kmap_local_page().
>> >
>> > There are two main problems with kmap(): (1) It comes with an overhead as
>> > the mapping space is restricted and protected by a global lock for
>> > synchronization and (2) it also requires global TLB invalidation when the
>> > kmap’s pool wraps and it might block when the mapping space is fully
>> > utilized until a slot becomes available.
>> >
>> > With kmap_local_page() the mappings are per thread, CPU local, can take
>> > page faults, and can be called from any context (including interrupts).
>> > It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
>> > the tasks can be preempted and, when they are scheduled to run again, the
>> > kernel virtual addresses are restored and still valid.
>> >
>> > Since its use in fs/aio.c is safe everywhere, it should be preferred.
>> 
>> That sentence is very ambiguous.  I don't know what "its" refers to, and
>> I'm not sure what "safe" means in this context.
>
> I'm sorry for not being clearer.
>
> "its use" means "the use of kmap_local_page()". Few lines above you may also 
> see "It is faster", meaning "kmap_local_page() is faster".

Got it, thanks.

> The "safety" is a very concise way to assert that I've checked, by code 
> inspection and by testing (as it is better detailed some lines below) that 
> these conversions (1) don't break any of the rules of use of local mapping 
> when converting kmap() (please read highmem.rst about these) and (2) the call 
> sites of kmap_atomic() didn't rely on its side effects (pagefaults disable and 
> potential preemption disables). 

OK, good.  I agree that the aio code wasn't relying on the side effects of
kmap_atomic.

> Therefore, you may read it as it was: "The use of kmap_local_page() in fs/
> aio.c has been carefully checked to assure that the conversions won't break 
> the code, therefore the newer API is preferred".
>
> I hope it makes my argument clearer.

Yes, thank you for explaining!

-Jeff

>
>> 
>> The patch looks okay to me.
>> 
>> Reviewed-by: Jeff Moyer <jmoyer@redhat.com>
>> 
>
> Thank you so much for the  "Reviewed-by" tag.
>
> Regards,
>
> Fabio
Fabio M. De Francesco Nov. 26, 2022, 4:51 p.m. UTC | #4 
On domenica 16 ottobre 2022 17:06:56 CET Fabio M. De Francesco wrote:
> The use of kmap() and kmap_atomic() are being deprecated in favor of
> kmap_local_page().
> 
> There are two main problems with kmap(): (1) It comes with an overhead as
> the mapping space is restricted and protected by a global lock for
> synchronization and (2) it also requires global TLB invalidation when the
> kmap’s pool wraps and it might block when the mapping space is fully
> utilized until a slot becomes available.
> 
> With kmap_local_page() the mappings are per thread, CPU local, can take
> page faults, and can be called from any context (including interrupts).
> It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
> the tasks can be preempted and, when they are scheduled to run again, the
> kernel virtual addresses are restored and still valid.
> 
> Since its use in fs/aio.c is safe everywhere, it should be preferred.
> 
> Therefore, replace kmap() and kmap_atomic() with kmap_local_page() in
> fs/aio.c.
> 
> Tested with xfstests on a QEMU/KVM x86_32 VM, 6GB RAM, booting a kernel
> with HIGHMEM64GB enabled.
> 
> Cc: "Venkataramanan, Anirudh" <anirudh.venkataramanan@intel.com>
> Suggested-by: Ira Weiny <ira.weiny@intel.com>
> Reviewed-by: Ira Weiny <ira.weiny@intel.com>
> Signed-off-by: Fabio M. De Francesco <fmdefrancesco@gmail.com>
> ---
> 
> I've tested with "./check -g aio". The tests in this group fail 3/26
> times, with and without my patch. Therefore, these changes don't introduce
> further errors. I'm not aware of any further tests I may run, so that
> any suggestions would be precious and much appreciated :-)
> 
> I'm resending this patch because some recipients were missing in the
> previous submissions. In the meantime I'm also adding some more information
> in the commit message. There are no changes in the code.
> 
>  fs/aio.c | 32 ++++++++++++++++----------------
>  1 file changed, 16 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/aio.c b/fs/aio.c
> index 3c249b938632..343fea0c6d1a 100644
> --- a/fs/aio.c
> +++ b/fs/aio.c
> @@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned 
int
> nr_events) ctx->user_id = ctx->mmap_base;
>  	ctx->nr_events = nr_events; /* trusted copy */
> 
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	ring->nr = nr_events;	/* user copy */
>  	ring->id = ~0U;
>  	ring->head = ring->tail = 0;
> @@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned 
int
> nr_events) ring->compat_features = AIO_RING_COMPAT_FEATURES;
>  	ring->incompat_features = AIO_RING_INCOMPAT_FEATURES;
>  	ring->header_length = sizeof(struct aio_ring);
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
>  	flush_dcache_page(ctx->ring_pages[0]);
> 
>  	return 0;
> @@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct
> mm_struct *mm) * we are protected from page migration
>  					 * changes ring_pages by -
>ring_lock.
>  					 */
> -					ring = kmap_atomic(ctx-
>ring_pages[0]);
> +					ring = kmap_local_page(ctx-
>ring_pages[0]);
>  					ring->id = ctx->id;
> -					kunmap_atomic(ring);
> +					kunmap_local(ring);
>  					return 0;
>  				}
> 
> @@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx
> *ctx) * against ctx->completed_events below will make sure we do the
>  		 * safe/right thing.
>  		 */
> -		ring = kmap_atomic(ctx->ring_pages[0]);
> +		ring = kmap_local_page(ctx->ring_pages[0]);
>  		head = ring->head;
> -		kunmap_atomic(ring);
> +		kunmap_local(ring);
> 
>  		refill_reqs_available(ctx, head, ctx->tail);
>  	}
> @@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)
>  	if (++tail >= ctx->nr_events)
>  		tail = 0;
> 
> -	ev_page = kmap_atomic(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
> +	ev_page = kmap_local_page(ctx->ring_pages[pos / 
AIO_EVENTS_PER_PAGE]);
>  	event = ev_page + pos % AIO_EVENTS_PER_PAGE;
> 
>  	*event = iocb->ki_res;
> 
> -	kunmap_atomic(ev_page);
> +	kunmap_local(ev_page);
>  	flush_dcache_page(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
> 
>  	pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,
> @@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb)
> 
>  	ctx->tail = tail;
> 
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	head = ring->head;
>  	ring->tail = tail;
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
>  	flush_dcache_page(ctx->ring_pages[0]);
> 
>  	ctx->completed_events++;
> @@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx *ctx,
>  	mutex_lock(&ctx->ring_lock);
> 
>  	/* Access to ->ring_pages here is protected by ctx->ring_lock. */
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	head = ring->head;
>  	tail = ring->tail;
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
> 
>  	/*
>  	 * Ensure that once we've read the current tail pointer, that
> @@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx *ctx,
>  		avail = min(avail, nr - ret);
>  		avail = min_t(long, avail, AIO_EVENTS_PER_PAGE - pos);
> 
> -		ev = kmap(page);
> +		ev = kmap_local_page(page);
>  		copy_ret = copy_to_user(event + ret, ev + pos,
>  					sizeof(*ev) * avail);
> -		kunmap(page);
> +		kunmap_local(ev);
> 
>  		if (unlikely(copy_ret)) {
>  			ret = -EFAULT;
> @@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx,
>  		head %= ctx->nr_events;
>  	}
> 
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	ring->head = head;
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
>  	flush_dcache_page(ctx->ring_pages[0]);
> 
>  	pr_debug("%li  h%u t%u\n", ret, head, tail);
> --
> 2.36.1

Al, Benjamin,

I'm sending a gentle ping for this old patch too (and thanking Al again for 
pointing me out how fs/ufs and fs/sysv conversions must be reworked and 
mistakes fixed).

About this I've had Ira's and Jeff's "Reviewed-by:" tags. I also responded in 
this thread to a couple of objections from Jeff which were regarding some 
ambiguities in the commit message.

Please let me know if here too there are mistakes which must be fixed and code 
to be reworked.

I'm currently just a little more than a pure hobbyist, therefore please be 
patient for the time it takes because until mid February 2023 I'll only be 
able to work few hours per weeks using my spare time. 

I'm looking forward to hearing from you.

Regards,

Fabio
Fabio M. De Francesco Dec. 1, 2022, 2:29 p.m. UTC | #5 
On domenica 16 ottobre 2022 17:06:56 CET Fabio M. De Francesco wrote:
> The use of kmap() and kmap_atomic() are being deprecated in favor of
> kmap_local_page().
> 
> There are two main problems with kmap(): (1) It comes with an overhead as
> the mapping space is restricted and protected by a global lock for
> synchronization and (2) it also requires global TLB invalidation when the
> kmap’s pool wraps and it might block when the mapping space is fully
> utilized until a slot becomes available.
> 
> With kmap_local_page() the mappings are per thread, CPU local, can take
> page faults, and can be called from any context (including interrupts).
> It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
> the tasks can be preempted and, when they are scheduled to run again, the
> kernel virtual addresses are restored and still valid.
> 
> Since its use in fs/aio.c is safe everywhere, it should be preferred.
> 
> Therefore, replace kmap() and kmap_atomic() with kmap_local_page() in
> fs/aio.c.
> 
> Tested with xfstests on a QEMU/KVM x86_32 VM, 6GB RAM, booting a kernel
> with HIGHMEM64GB enabled.
> 
> Cc: "Venkataramanan, Anirudh" <anirudh.venkataramanan@intel.com>
> Suggested-by: Ira Weiny <ira.weiny@intel.com>
> Reviewed-by: Ira Weiny <ira.weiny@intel.com>
Reviewed-by: Jeff Moyer <jmoyer@redhat.com>
> Signed-off-by: Fabio M. De Francesco <fmdefrancesco@gmail.com>
> ---

I'm sorry to resend again. Last time I forgot to forward the "Reviewed-by:" 
tag from Jeff (thanks!).

> 
> I've tested with "./check -g aio". The tests in this group fail 3/26
> times, with and without my patch. Therefore, these changes don't introduce
> further errors. I'm not aware of any further tests I may run, so that
> any suggestions would be precious and much appreciated :-)
> 
> I'm resending this patch because some recipients were missing in the
> previous submissions. In the meantime I'm also adding some more information
> in the commit message. There are no changes in the code.
> 
>  fs/aio.c | 32 ++++++++++++++++----------------
>  1 file changed, 16 insertions(+), 16 deletions(-)
> 
> diff --git a/fs/aio.c b/fs/aio.c
> index 3c249b938632..343fea0c6d1a 100644
> --- a/fs/aio.c
> +++ b/fs/aio.c
> @@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned 
int
> nr_events) ctx->user_id = ctx->mmap_base;
>  	ctx->nr_events = nr_events; /* trusted copy */
> 
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	ring->nr = nr_events;	/* user copy */
>  	ring->id = ~0U;
>  	ring->head = ring->tail = 0;
> @@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned 
int
> nr_events) ring->compat_features = AIO_RING_COMPAT_FEATURES;
>  	ring->incompat_features = AIO_RING_INCOMPAT_FEATURES;
>  	ring->header_length = sizeof(struct aio_ring);
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
>  	flush_dcache_page(ctx->ring_pages[0]);
> 
>  	return 0;
> @@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct
> mm_struct *mm) * we are protected from page migration
>  					 * changes ring_pages by -
>ring_lock.
>  					 */
> -					ring = kmap_atomic(ctx-
>ring_pages[0]);
> +					ring = kmap_local_page(ctx-
>ring_pages[0]);
>  					ring->id = ctx->id;
> -					kunmap_atomic(ring);
> +					kunmap_local(ring);
>  					return 0;
>  				}
> 
> @@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx
> *ctx) * against ctx->completed_events below will make sure we do the
>  		 * safe/right thing.
>  		 */
> -		ring = kmap_atomic(ctx->ring_pages[0]);
> +		ring = kmap_local_page(ctx->ring_pages[0]);
>  		head = ring->head;
> -		kunmap_atomic(ring);
> +		kunmap_local(ring);
> 
>  		refill_reqs_available(ctx, head, ctx->tail);
>  	}
> @@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)
>  	if (++tail >= ctx->nr_events)
>  		tail = 0;
> 
> -	ev_page = kmap_atomic(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
> +	ev_page = kmap_local_page(ctx->ring_pages[pos / 
AIO_EVENTS_PER_PAGE]);
>  	event = ev_page + pos % AIO_EVENTS_PER_PAGE;
> 
>  	*event = iocb->ki_res;
> 
> -	kunmap_atomic(ev_page);
> +	kunmap_local(ev_page);
>  	flush_dcache_page(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
> 
>  	pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,
> @@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb)
> 
>  	ctx->tail = tail;
> 
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	head = ring->head;
>  	ring->tail = tail;
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
>  	flush_dcache_page(ctx->ring_pages[0]);
> 
>  	ctx->completed_events++;
> @@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx *ctx,
>  	mutex_lock(&ctx->ring_lock);
> 
>  	/* Access to ->ring_pages here is protected by ctx->ring_lock. */
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	head = ring->head;
>  	tail = ring->tail;
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
> 
>  	/*
>  	 * Ensure that once we've read the current tail pointer, that
> @@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx *ctx,
>  		avail = min(avail, nr - ret);
>  		avail = min_t(long, avail, AIO_EVENTS_PER_PAGE - pos);
> 
> -		ev = kmap(page);
> +		ev = kmap_local_page(page);
>  		copy_ret = copy_to_user(event + ret, ev + pos,
>  					sizeof(*ev) * avail);
> -		kunmap(page);
> +		kunmap_local(ev);
> 
>  		if (unlikely(copy_ret)) {
>  			ret = -EFAULT;
> @@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx,
>  		head %= ctx->nr_events;
>  	}
> 
> -	ring = kmap_atomic(ctx->ring_pages[0]);
> +	ring = kmap_local_page(ctx->ring_pages[0]);
>  	ring->head = head;
> -	kunmap_atomic(ring);
> +	kunmap_local(ring);
>  	flush_dcache_page(ctx->ring_pages[0]);
> 
>  	pr_debug("%li  h%u t%u\n", ret, head, tail);
> --
> 2.36.1
Fabio M. De Francesco Jan. 9, 2023, 6:12 p.m. UTC | #6 
On giovedì 1 dicembre 2022 15:29:17 CET Fabio M. De Francesco wrote:
> On domenica 16 ottobre 2022 17:06:56 CET Fabio M. De Francesco wrote:
> > The use of kmap() and kmap_atomic() are being deprecated in favor of
> > kmap_local_page().
> > 
> > There are two main problems with kmap(): (1) It comes with an overhead as
> > the mapping space is restricted and protected by a global lock for
> > synchronization and (2) it also requires global TLB invalidation when the
> > kmap’s pool wraps and it might block when the mapping space is fully
> > utilized until a slot becomes available.
> > 
> > With kmap_local_page() the mappings are per thread, CPU local, can take
> > page faults, and can be called from any context (including interrupts).
> > It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
> > the tasks can be preempted and, when they are scheduled to run again, the
> > kernel virtual addresses are restored and still valid.
> > 
> > Since its use in fs/aio.c is safe everywhere, it should be preferred.
> > 
> > Therefore, replace kmap() and kmap_atomic() with kmap_local_page() in
> > fs/aio.c.
> > 
> > Tested with xfstests on a QEMU/KVM x86_32 VM, 6GB RAM, booting a kernel
> > with HIGHMEM64GB enabled.
> > 
> > Cc: "Venkataramanan, Anirudh" <anirudh.venkataramanan@intel.com>
> > Suggested-by: Ira Weiny <ira.weiny@intel.com>
> > Reviewed-by: Ira Weiny <ira.weiny@intel.com>
> 
> Reviewed-by: Jeff Moyer <jmoyer@redhat.com>
> 
> > Signed-off-by: Fabio M. De Francesco <fmdefrancesco@gmail.com>
> > ---
> 
> I'm sorry to resend again. Last time I forgot to forward the "Reviewed-by:"
> tag from Jeff (thanks!).
> 
> > I've tested with "./check -g aio". The tests in this group fail 3/26
> > times, with and without my patch. Therefore, these changes don't introduce
> > further errors. I'm not aware of any further tests I may run, so that
> > any suggestions would be precious and much appreciated :-)
> > 
> > I'm resending this patch because some recipients were missing in the
> > previous submissions. In the meantime I'm also adding some more 
information
> > in the commit message. There are no changes in the code.
> > 
> >  fs/aio.c | 32 ++++++++++++++++----------------
> >  1 file changed, 16 insertions(+), 16 deletions(-)
> > 
> > diff --git a/fs/aio.c b/fs/aio.c
> > index 3c249b938632..343fea0c6d1a 100644
> > --- a/fs/aio.c
> > +++ b/fs/aio.c
> > @@ -567,7 +567,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned
> 
> int
> 
> > nr_events) ctx->user_id = ctx->mmap_base;
> > 
> >  	ctx->nr_events = nr_events; /* trusted copy */
> > 
> > -	ring = kmap_atomic(ctx->ring_pages[0]);
> > +	ring = kmap_local_page(ctx->ring_pages[0]);
> > 
> >  	ring->nr = nr_events;	/* user copy */
> >  	ring->id = ~0U;
> >  	ring->head = ring->tail = 0;
> > 
> > @@ -575,7 +575,7 @@ static int aio_setup_ring(struct kioctx *ctx, unsigned
> 
> int
> 
> > nr_events) ring->compat_features = AIO_RING_COMPAT_FEATURES;
> > 
> >  	ring->incompat_features = AIO_RING_INCOMPAT_FEATURES;
> >  	ring->header_length = sizeof(struct aio_ring);
> > 
> > -	kunmap_atomic(ring);
> > +	kunmap_local(ring);
> > 
> >  	flush_dcache_page(ctx->ring_pages[0]);
> >  	
> >  	return 0;
> > 
> > @@ -678,9 +678,9 @@ static int ioctx_add_table(struct kioctx *ctx, struct
> > mm_struct *mm) * we are protected from page migration
> > 
> >  					 * changes ring_pages by -
> >
> >ring_lock.
> >
> >  					 */
> > 
> > -					ring = kmap_atomic(ctx-
> >
> >ring_pages[0]);
> >
> > +					ring = kmap_local_page(ctx-
> >
> >ring_pages[0]);
> >
> >  					ring->id = ctx->id;
> > 
> > -					kunmap_atomic(ring);
> > +					kunmap_local(ring);
> > 
> >  					return 0;
> >  				
> >  				}
> > 
> > @@ -1024,9 +1024,9 @@ static void user_refill_reqs_available(struct kioctx
> > *ctx) * against ctx->completed_events below will make sure we do the
> > 
> >  		 * safe/right thing.
> >  		 */
> > 
> > -		ring = kmap_atomic(ctx->ring_pages[0]);
> > +		ring = kmap_local_page(ctx->ring_pages[0]);
> > 
> >  		head = ring->head;
> > 
> > -		kunmap_atomic(ring);
> > +		kunmap_local(ring);
> > 
> >  		refill_reqs_available(ctx, head, ctx->tail);
> >  	
> >  	}
> > 
> > @@ -1132,12 +1132,12 @@ static void aio_complete(struct aio_kiocb *iocb)
> > 
> >  	if (++tail >= ctx->nr_events)
> >  	
> >  		tail = 0;
> > 
> > -	ev_page = kmap_atomic(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
> > +	ev_page = kmap_local_page(ctx->ring_pages[pos /
> 
> AIO_EVENTS_PER_PAGE]);
> 
> >  	event = ev_page + pos % AIO_EVENTS_PER_PAGE;
> >  	
> >  	*event = iocb->ki_res;
> > 
> > -	kunmap_atomic(ev_page);
> > +	kunmap_local(ev_page);
> > 
> >  	flush_dcache_page(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
> >  	
> >  	pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,
> > 
> > @@ -1151,10 +1151,10 @@ static void aio_complete(struct aio_kiocb *iocb)
> > 
> >  	ctx->tail = tail;
> > 
> > -	ring = kmap_atomic(ctx->ring_pages[0]);
> > +	ring = kmap_local_page(ctx->ring_pages[0]);
> > 
> >  	head = ring->head;
> >  	ring->tail = tail;
> > 
> > -	kunmap_atomic(ring);
> > +	kunmap_local(ring);
> > 
> >  	flush_dcache_page(ctx->ring_pages[0]);
> >  	
> >  	ctx->completed_events++;
> > 
> > @@ -1214,10 +1214,10 @@ static long aio_read_events_ring(struct kioctx 
*ctx,
> > 
> >  	mutex_lock(&ctx->ring_lock);
> >  	
> >  	/* Access to ->ring_pages here is protected by ctx->ring_lock. */
> > 
> > -	ring = kmap_atomic(ctx->ring_pages[0]);
> > +	ring = kmap_local_page(ctx->ring_pages[0]);
> > 
> >  	head = ring->head;
> >  	tail = ring->tail;
> > 
> > -	kunmap_atomic(ring);
> > +	kunmap_local(ring);
> > 
> >  	/*
> >  	
> >  	 * Ensure that once we've read the current tail pointer, that
> > 
> > @@ -1249,10 +1249,10 @@ static long aio_read_events_ring(struct kioctx 
*ctx,
> > 
> >  		avail = min(avail, nr - ret);
> >  		avail = min_t(long, avail, AIO_EVENTS_PER_PAGE - pos);
> > 
> > -		ev = kmap(page);
> > +		ev = kmap_local_page(page);
> > 
> >  		copy_ret = copy_to_user(event + ret, ev + pos,
> >  		
> >  					sizeof(*ev) * avail);
> > 
> > -		kunmap(page);
> > +		kunmap_local(ev);
> > 
> >  		if (unlikely(copy_ret)) {
> >  		
> >  			ret = -EFAULT;
> > 
> > @@ -1264,9 +1264,9 @@ static long aio_read_events_ring(struct kioctx *ctx,
> > 
> >  		head %= ctx->nr_events;
> >  	
> >  	}
> > 
> > -	ring = kmap_atomic(ctx->ring_pages[0]);
> > +	ring = kmap_local_page(ctx->ring_pages[0]);
> > 
> >  	ring->head = head;
> > 
> > -	kunmap_atomic(ring);
> > +	kunmap_local(ring);
> > 
> >  	flush_dcache_page(ctx->ring_pages[0]);
> >  	
> >  	pr_debug("%li  h%u t%u\n", ret, head, tail);
> > 
> > --
> > 2.36.1

Please disregard this patch because I just sent a v2 with some additional 
information in the commit message and added Jeff's "Reviewed-by" tag.

Thanks,

Fabio

[1] https://lore.kernel.org/lkml/20230109175629.9482-1-fmdefrancesco@gmail.com/
Kent Overstreet Jan. 19, 2023, 9:41 a.m. UTC | #7 
On Sun, Oct 16, 2022 at 05:06:56PM +0200, Fabio M. De Francesco wrote:
> The use of kmap() and kmap_atomic() are being deprecated in favor of
> kmap_local_page().
> 
> There are two main problems with kmap(): (1) It comes with an overhead as
> the mapping space is restricted and protected by a global lock for
> synchronization and (2) it also requires global TLB invalidation when the
> kmap’s pool wraps and it might block when the mapping space is fully
> utilized until a slot becomes available.
> 
> With kmap_local_page() the mappings are per thread, CPU local, can take
> page faults, and can be called from any context (including interrupts).
> It is faster than kmap() in kernels with HIGHMEM enabled. Furthermore,
> the tasks can be preempted and, when they are scheduled to run again, the
> kernel virtual addresses are restored and still valid.
> 
> Since its use in fs/aio.c is safe everywhere, it should be preferred.
> 
> Therefore, replace kmap() and kmap_atomic() with kmap_local_page() in
> fs/aio.c.
> 
> Tested with xfstests on a QEMU/KVM x86_32 VM, 6GB RAM, booting a kernel
> with HIGHMEM64GB enabled.

I was just looking over this code and made the same kmap -> kmap_local
change, but you've done a more complete version - nice.

For context, I was the one who added the kmap() call, because
copy_to_user() can sleep - anything else at the time would've been more
awkward, and highmem machines were already on the way out. But
kmap_local is perfect here :)

Reviewed-by: Kent Overstreet <kent.overstreet@linux.dev>
diff mbox series

Patch

diff --git a/fs/aio.c b/fs/aio.c
index 3c249b938632..343fea0c6d1a 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -567,7 +567,7 @@  static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events)
 	ctx->user_id = ctx->mmap_base;
 	ctx->nr_events = nr_events; /* trusted copy */
 
-	ring = kmap_atomic(ctx->ring_pages[0]);
+	ring = kmap_local_page(ctx->ring_pages[0]);
 	ring->nr = nr_events;	/* user copy */
 	ring->id = ~0U;
 	ring->head = ring->tail = 0;
@@ -575,7 +575,7 @@  static int aio_setup_ring(struct kioctx *ctx, unsigned int nr_events)
 	ring->compat_features = AIO_RING_COMPAT_FEATURES;
 	ring->incompat_features = AIO_RING_INCOMPAT_FEATURES;
 	ring->header_length = sizeof(struct aio_ring);
-	kunmap_atomic(ring);
+	kunmap_local(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
 	return 0;
@@ -678,9 +678,9 @@  static int ioctx_add_table(struct kioctx *ctx, struct mm_struct *mm)
 					 * we are protected from page migration
 					 * changes ring_pages by ->ring_lock.
 					 */
-					ring = kmap_atomic(ctx->ring_pages[0]);
+					ring = kmap_local_page(ctx->ring_pages[0]);
 					ring->id = ctx->id;
-					kunmap_atomic(ring);
+					kunmap_local(ring);
 					return 0;
 				}
 
@@ -1024,9 +1024,9 @@  static void user_refill_reqs_available(struct kioctx *ctx)
 		 * against ctx->completed_events below will make sure we do the
 		 * safe/right thing.
 		 */
-		ring = kmap_atomic(ctx->ring_pages[0]);
+		ring = kmap_local_page(ctx->ring_pages[0]);
 		head = ring->head;
-		kunmap_atomic(ring);
+		kunmap_local(ring);
 
 		refill_reqs_available(ctx, head, ctx->tail);
 	}
@@ -1132,12 +1132,12 @@  static void aio_complete(struct aio_kiocb *iocb)
 	if (++tail >= ctx->nr_events)
 		tail = 0;
 
-	ev_page = kmap_atomic(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
+	ev_page = kmap_local_page(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
 	event = ev_page + pos % AIO_EVENTS_PER_PAGE;
 
 	*event = iocb->ki_res;
 
-	kunmap_atomic(ev_page);
+	kunmap_local(ev_page);
 	flush_dcache_page(ctx->ring_pages[pos / AIO_EVENTS_PER_PAGE]);
 
 	pr_debug("%p[%u]: %p: %p %Lx %Lx %Lx\n", ctx, tail, iocb,
@@ -1151,10 +1151,10 @@  static void aio_complete(struct aio_kiocb *iocb)
 
 	ctx->tail = tail;
 
-	ring = kmap_atomic(ctx->ring_pages[0]);
+	ring = kmap_local_page(ctx->ring_pages[0]);
 	head = ring->head;
 	ring->tail = tail;
-	kunmap_atomic(ring);
+	kunmap_local(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
 	ctx->completed_events++;
@@ -1214,10 +1214,10 @@  static long aio_read_events_ring(struct kioctx *ctx,
 	mutex_lock(&ctx->ring_lock);
 
 	/* Access to ->ring_pages here is protected by ctx->ring_lock. */
-	ring = kmap_atomic(ctx->ring_pages[0]);
+	ring = kmap_local_page(ctx->ring_pages[0]);
 	head = ring->head;
 	tail = ring->tail;
-	kunmap_atomic(ring);
+	kunmap_local(ring);
 
 	/*
 	 * Ensure that once we've read the current tail pointer, that
@@ -1249,10 +1249,10 @@  static long aio_read_events_ring(struct kioctx *ctx,
 		avail = min(avail, nr - ret);
 		avail = min_t(long, avail, AIO_EVENTS_PER_PAGE - pos);
 
-		ev = kmap(page);
+		ev = kmap_local_page(page);
 		copy_ret = copy_to_user(event + ret, ev + pos,
 					sizeof(*ev) * avail);
-		kunmap(page);
+		kunmap_local(ev);
 
 		if (unlikely(copy_ret)) {
 			ret = -EFAULT;
@@ -1264,9 +1264,9 @@  static long aio_read_events_ring(struct kioctx *ctx,
 		head %= ctx->nr_events;
 	}
 
-	ring = kmap_atomic(ctx->ring_pages[0]);
+	ring = kmap_local_page(ctx->ring_pages[0]);
 	ring->head = head;
-	kunmap_atomic(ring);
+	kunmap_local(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
 	pr_debug("%li  h%u t%u\n", ret, head, tail);