[v1,3/7] rust: sync: add `Arc::{from_raw, into_raw}`

  From: Wedson Almeida Filho <walmeida@microsoft.com>

These methods can be used to turn an `Arc` into a raw pointer and back,
in a way that preserves the metadata for fat pointers.

This is done using the unstable ptr_metadata feature [1]. However, it
could also be done using the unstable pointer_byte_offsets feature [2],
which is likely to have a shorter path to stabilization than
ptr_metadata.

Link: https://github.com/rust-lang/rust/issues/81513 [1]
Link: https://github.com/rust-lang/rust/issues/96283 [2]
Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
---
 rust/kernel/lib.rs      |  1 +
 rust/kernel/sync/arc.rs | 44 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)
  

On 5/17/23 17:31, Alice Ryhl wrote:
> From: Wedson Almeida Filho <walmeida@microsoft.com>
> 
> These methods can be used to turn an `Arc` into a raw pointer and back,
> in a way that preserves the metadata for fat pointers.
> 
> This is done using the unstable ptr_metadata feature [1]. However, it
> could also be done using the unstable pointer_byte_offsets feature [2],
> which is likely to have a shorter path to stabilization than
> ptr_metadata.
> 
> Link: https://github.com/rust-lang/rust/issues/81513 [1]
> Link: https://github.com/rust-lang/rust/issues/96283 [2]
> Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
> Co-developed-by: Alice Ryhl <aliceryhl@google.com>
> Signed-off-by: Alice Ryhl <aliceryhl@google.com>
> ---
> [...]

Reviewed-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com>
  

On Wed, 17 May 2023 20:31:15 +0000
Alice Ryhl <aliceryhl@google.com> wrote:

> From: Wedson Almeida Filho <walmeida@microsoft.com>
> 
> These methods can be used to turn an `Arc` into a raw pointer and back,
> in a way that preserves the metadata for fat pointers.
> 
> This is done using the unstable ptr_metadata feature [1]. However, it
> could also be done using the unstable pointer_byte_offsets feature [2],
> which is likely to have a shorter path to stabilization than
> ptr_metadata.
> 
> Link: https://github.com/rust-lang/rust/issues/81513 [1]
> Link: https://github.com/rust-lang/rust/issues/96283 [2]
> Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
> Co-developed-by: Alice Ryhl <aliceryhl@google.com>
> Signed-off-by: Alice Ryhl <aliceryhl@google.com>
> ---
>  rust/kernel/lib.rs      |  1 +
>  rust/kernel/sync/arc.rs | 44 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 45 insertions(+)
> 
> diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
> index cdf9fe999328..82854c86e65d 100644
> --- a/rust/kernel/lib.rs
> +++ b/rust/kernel/lib.rs
> @@ -22,6 +22,7 @@
>  #![feature(generic_associated_types)]
>  #![feature(new_uninit)]
>  #![feature(pin_macro)]
> +#![feature(ptr_metadata)]
>  #![feature(receiver_trait)]
>  #![feature(unsize)]
>  
> diff --git a/rust/kernel/sync/arc.rs b/rust/kernel/sync/arc.rs
> index e6d206242465..7c55a9178dfb 100644
> --- a/rust/kernel/sync/arc.rs
> +++ b/rust/kernel/sync/arc.rs
> @@ -210,6 +210,50 @@ impl<T: ?Sized> Arc<T> {
>          }
>      }
>  
> +    /// Convert the [`Arc`] into a raw pointer.
> +    ///
> +    /// The raw pointer has ownership of the refcount that this Arc object owned.
> +    pub fn into_raw(self) -> *const T {
> +        let ptr = self.ptr.as_ptr();
> +        core::mem::forget(self);
> +        // SAFETY: The pointer is valid.
> +        unsafe { core::ptr::addr_of!((*ptr).data) }
> +    }
> +
> +    /// Recreates an [`Arc`] instance previously deconstructed via [`Arc::into_raw`].
> +    ///
> +    /// This code relies on the `repr(C)` layout of structs as described in
> +    /// <https://doc.rust-lang.org/reference/type-layout.html#reprc-structs>.
> +    ///
> +    /// # Safety
> +    ///
> +    /// `ptr` must have been returned by a previous call to [`Arc::into_raw`]. Additionally, it
> +    /// can only be called once for each previous call to [`Arc::into_raw`].
> +    pub unsafe fn from_raw(ptr: *const T) -> Self {
> +        // SAFETY: The safety requirement ensures that the pointer is valid.
> +        let val_align = core::mem::align_of_val(unsafe { &*ptr });
> +        let refcount_size = core::mem::size_of::<Opaque<bindings::refcount_t>>();
> +
> +        // Use the `repr(C)` algorithm to compute the offset of `data` in `ArcInner`.
> +        //
> +        // Pseudo-code for the `#[repr(C)]` algorithm can be found here:
> +        // <https://doc.rust-lang.org/reference/type-layout.html#reprc-structs>
> +        let mut val_offset = refcount_size;
> +        let val_misalign = val_offset % val_align;
> +        if val_misalign > 0 {
> +            val_offset += val_align - val_misalign;
> +        }

Given the layout of the whole ArcInner can be calculated as

	Layout::new::<bindings::refcount_t>().extend(Layout::for_value(&*ptr)).unwrap_unchecked().0.pad_to_align()

The offset of `data` could be more intuitively calculated by

	Layout::new::<bindings::refcount_t>().extend(Layout::for_value(&*ptr)).unwrap_unchecked().1

or

	Layout::new::<bindings::refcount_t>().align_to(val_align).unwrap_unchecked().pad_to_align().size()

Best,
Gary
  

Alice Ryhl <aliceryhl@google.com> writes:

> From: Wedson Almeida Filho <walmeida@microsoft.com>
>
> These methods can be used to turn an `Arc` into a raw pointer and back,
> in a way that preserves the metadata for fat pointers.
>
> This is done using the unstable ptr_metadata feature [1]. However, it
> could also be done using the unstable pointer_byte_offsets feature [2],
> which is likely to have a shorter path to stabilization than
> ptr_metadata.
>
> Link: https://github.com/rust-lang/rust/issues/81513 [1]
> Link: https://github.com/rust-lang/rust/issues/96283 [2]
> Signed-off-by: Wedson Almeida Filho <walmeida@microsoft.com>
> Co-developed-by: Alice Ryhl <aliceryhl@google.com>
> Signed-off-by: Alice Ryhl <aliceryhl@google.com>
> ---
>  rust/kernel/lib.rs      |  1 +
>  rust/kernel/sync/arc.rs | 44 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 45 insertions(+)
>
> diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
> index cdf9fe999328..82854c86e65d 100644
> --- a/rust/kernel/lib.rs
> +++ b/rust/kernel/lib.rs
> @@ -22,6 +22,7 @@
>  #![feature(generic_associated_types)]
>  #![feature(new_uninit)]
>  #![feature(pin_macro)]
> +#![feature(ptr_metadata)]
>  #![feature(receiver_trait)]
>  #![feature(unsize)]
>  
> diff --git a/rust/kernel/sync/arc.rs b/rust/kernel/sync/arc.rs
> index e6d206242465..7c55a9178dfb 100644
> --- a/rust/kernel/sync/arc.rs
> +++ b/rust/kernel/sync/arc.rs
> @@ -210,6 +210,50 @@ impl<T: ?Sized> Arc<T> {
>          }
>      }
>  
> +    /// Convert the [`Arc`] into a raw pointer.
> +    ///
> +    /// The raw pointer has ownership of the refcount that this Arc object owned.
> +    pub fn into_raw(self) -> *const T {
> +        let ptr = self.ptr.as_ptr();
> +        core::mem::forget(self);
> +        // SAFETY: The pointer is valid.
> +        unsafe { core::ptr::addr_of!((*ptr).data) }
> +    }
> +
> +    /// Recreates an [`Arc`] instance previously deconstructed via [`Arc::into_raw`].
> +    ///
> +    /// This code relies on the `repr(C)` layout of structs as described in
> +    /// <https://doc.rust-lang.org/reference/type-layout.html#reprc-structs>.
> +    ///
> +    /// # Safety
> +    ///
> +    /// `ptr` must have been returned by a previous call to [`Arc::into_raw`]. Additionally, it
> +    /// can only be called once for each previous call to [`Arc::into_raw`].
> +    pub unsafe fn from_raw(ptr: *const T) -> Self {
> +        // SAFETY: The safety requirement ensures that the pointer is valid.
> +        let val_align = core::mem::align_of_val(unsafe { &*ptr });
> +        let refcount_size = core::mem::size_of::<Opaque<bindings::refcount_t>>();
> +
> +        // Use the `repr(C)` algorithm to compute the offset of `data` in `ArcInner`.
> +        //
> +        // Pseudo-code for the `#[repr(C)]` algorithm can be found here:
> +        // <https://doc.rust-lang.org/reference/type-layout.html#reprc-structs>
> +        let mut val_offset = refcount_size;
> +        let val_misalign = val_offset % val_align;
> +        if val_misalign > 0 {
> +            val_offset += val_align - val_misalign;
> +        }
> +
> +        // This preserves the metadata in the pointer, if any.
> +        let metadata = core::ptr::metadata(ptr as *const ArcInner<T>);

I can't follow this. How does this work? `ptr` was for field
`inner.data: T`, but we are casting to `ArcInner<T>`.

> +        let ptr = (ptr as *mut u8).wrapping_sub(val_offset) as *mut ();
> +        let ptr = core::ptr::from_raw_parts_mut(ptr, metadata);

Metadata was obtained from a pointer pointing to `inner.data`, we then
move it back to beginning of `ArcInner<T>` and then reconstruct the
potentially fat pointer with metadata from the pointer to `T`? How can
this be right?

BR Andreas

> +
> +        // SAFETY: By the safety requirements we know that `ptr` came from `Arc::into_raw`, so the
> +        // reference count held then will be owned by the new `Arc` object.
> +        unsafe { Self::from_inner(NonNull::new_unchecked(ptr)) }
> +    }
> +
>      /// Returns an [`ArcBorrow`] from the given [`Arc`].
>      ///
>      /// This is useful when the argument of a function call is an [`ArcBorrow`] (e.g., in a method
  

Andreas Hindborg <nmi@metaspace.dk> writes:
> Alice Ryhl <aliceryhl@google.com> writes:
>> +        // This preserves the metadata in the pointer, if any.
>> +        let metadata = core::ptr::metadata(ptr as *const ArcInner<T>);
> 
> I can't follow this. How does this work? `ptr` was for field
> `inner.data: T`, but we are casting to `ArcInner<T>`.
> 
>> +        let ptr = (ptr as *mut u8).wrapping_sub(val_offset) as *mut ();
>> +        let ptr = core::ptr::from_raw_parts_mut(ptr, metadata);
> 
> Metadata was obtained from a pointer pointing to `inner.data`, we then
> move it back to beginning of `ArcInner<T>` and then reconstruct the
> potentially fat pointer with metadata from the pointer to `T`? How can
> this be right?

The metadata of a struct is always the metadata of its last field, so
both `*mut T` and `*mut ArcInner<T>` have the same metadata. Because of
that, moving the metadata over from one type to the other is ok.

The reason that I cast to an `ArcInner<T>` pointer before calling
`metadata` is because I get a type mismatch otherwise for the metadata,
since the compiler doesn't unify the metadata types when the type is
generic.

Alice
  

Gary Guo <gary@garyguo.net> writes:
> On Wed, 17 May 2023 20:31:15 +0000
> Alice Ryhl <aliceryhl@google.com> wrote:
>> +    /// Recreates an [`Arc`] instance previously deconstructed via [`Arc::into_raw`].
>> +    ///
>> +    /// This code relies on the `repr(C)` layout of structs as described in
>> +    /// <https://doc.rust-lang.org/reference/type-layout.html#reprc-structs>.
>> +    ///
>> +    /// # Safety
>> +    ///
>> +    /// `ptr` must have been returned by a previous call to [`Arc::into_raw`]. Additionally, it
>> +    /// can only be called once for each previous call to [`Arc::into_raw`].
>> +    pub unsafe fn from_raw(ptr: *const T) -> Self {
>> +        // SAFETY: The safety requirement ensures that the pointer is valid.
>> +        let val_align = core::mem::align_of_val(unsafe { &*ptr });
>> +        let refcount_size = core::mem::size_of::<Opaque<bindings::refcount_t>>();
>> +
>> +        // Use the `repr(C)` algorithm to compute the offset of `data` in `ArcInner`.
>> +        //
>> +        // Pseudo-code for the `#[repr(C)]` algorithm can be found here:
>> +        // <https://doc.rust-lang.org/reference/type-layout.html#reprc-structs>
>> +        let mut val_offset = refcount_size;
>> +        let val_misalign = val_offset % val_align;
>> +        if val_misalign > 0 {
>> +            val_offset += val_align - val_misalign;
>> +        }
> 
> Given the layout of the whole ArcInner can be calculated as
> 
> 	Layout::new::<bindings::refcount_t>().extend(Layout::for_value(&*ptr)).unwrap_unchecked().0.pad_to_align()
> 
> The offset of `data` could be more intuitively calculated by
> 
> 	Layout::new::<bindings::refcount_t>().extend(Layout::for_value(&*ptr)).unwrap_unchecked().1
> 
> or
> 
> 	Layout::new::<bindings::refcount_t>().align_to(val_align).unwrap_unchecked().pad_to_align().size()

I'm not a big fan of the `pad_to_align` version (which is also what
the rust branch uses), but I like the version you posted with
`extend`, and I agree that it is clear and intuitive. I will use that
in the next version of the patchset. Thanks for the suggestion.

Alice
  

Alice Ryhl <aliceryhl@google.com> writes:

> Andreas Hindborg <nmi@metaspace.dk> writes:
>> Alice Ryhl <aliceryhl@google.com> writes:
>>> +        // This preserves the metadata in the pointer, if any.
>>> +        let metadata = core::ptr::metadata(ptr as *const ArcInner<T>);
>> 
>> I can't follow this. How does this work? `ptr` was for field
>> `inner.data: T`, but we are casting to `ArcInner<T>`.
>> 
>>> +        let ptr = (ptr as *mut u8).wrapping_sub(val_offset) as *mut ();
>>> +        let ptr = core::ptr::from_raw_parts_mut(ptr, metadata);
>> 
>> Metadata was obtained from a pointer pointing to `inner.data`, we then
>> move it back to beginning of `ArcInner<T>` and then reconstruct the
>> potentially fat pointer with metadata from the pointer to `T`? How can
>> this be right?
>
> The metadata of a struct is always the metadata of its last field, so
> both `*mut T` and `*mut ArcInner<T>` have the same metadata. Because of
> that, moving the metadata over from one type to the other is ok.
>
> The reason that I cast to an `ArcInner<T>` pointer before calling
> `metadata` is because I get a type mismatch otherwise for the metadata,
> since the compiler doesn't unify the metadata types when the type is
> generic.

OK, cool. In that case, since this is common knowledge (is it?),
could you maybe include a link to the relevant documentation, or a
comment indicating why this is OK?

BR Andreas
  

On Thu, 25 May 2023 09:45:29 +0200
Andreas Hindborg <nmi@metaspace.dk> wrote:

> Alice Ryhl <aliceryhl@google.com> writes:
> 
> > Andreas Hindborg <nmi@metaspace.dk> writes:  
> >> Alice Ryhl <aliceryhl@google.com> writes:  
> >>> +        // This preserves the metadata in the pointer, if any.
> >>> +        let metadata = core::ptr::metadata(ptr as *const ArcInner<T>);  
> >> 
> >> I can't follow this. How does this work? `ptr` was for field
> >> `inner.data: T`, but we are casting to `ArcInner<T>`.
> >>   
> >>> +        let ptr = (ptr as *mut u8).wrapping_sub(val_offset) as *mut ();
> >>> +        let ptr = core::ptr::from_raw_parts_mut(ptr, metadata);  
> >> 
> >> Metadata was obtained from a pointer pointing to `inner.data`, we then
> >> move it back to beginning of `ArcInner<T>` and then reconstruct the
> >> potentially fat pointer with metadata from the pointer to `T`? How can
> >> this be right?  
> >
> > The metadata of a struct is always the metadata of its last field, so
> > both `*mut T` and `*mut ArcInner<T>` have the same metadata. Because of
> > that, moving the metadata over from one type to the other is ok.
> >
> > The reason that I cast to an `ArcInner<T>` pointer before calling
> > `metadata` is because I get a type mismatch otherwise for the metadata,
> > since the compiler doesn't unify the metadata types when the type is
> > generic.  
> 
> OK, cool. In that case, since this is common knowledge (is it?),
> could you maybe include a link to the relevant documentation, or a
> comment indicating why this is OK?
> 
> BR Andreas

This is documented in the doc of Pointee trait:

https://doc.rust-lang.org/std/ptr/trait.Pointee.html

> For structs whose last field is a DST, metadata is the metadata for the last field

Best,
Gary
  

Gary Guo <gary@garyguo.net> writes:

> On Thu, 25 May 2023 09:45:29 +0200
> Andreas Hindborg <nmi@metaspace.dk> wrote:
>
>> Alice Ryhl <aliceryhl@google.com> writes:
>> 
>> > Andreas Hindborg <nmi@metaspace.dk> writes:  
>> >> Alice Ryhl <aliceryhl@google.com> writes:  
>> >>> +        // This preserves the metadata in the pointer, if any.
>> >>> +        let metadata = core::ptr::metadata(ptr as *const ArcInner<T>);  
>> >> 
>> >> I can't follow this. How does this work? `ptr` was for field
>> >> `inner.data: T`, but we are casting to `ArcInner<T>`.
>> >>   
>> >>> +        let ptr = (ptr as *mut u8).wrapping_sub(val_offset) as *mut ();
>> >>> +        let ptr = core::ptr::from_raw_parts_mut(ptr, metadata);  
>> >> 
>> >> Metadata was obtained from a pointer pointing to `inner.data`, we then
>> >> move it back to beginning of `ArcInner<T>` and then reconstruct the
>> >> potentially fat pointer with metadata from the pointer to `T`? How can
>> >> this be right?  
>> >
>> > The metadata of a struct is always the metadata of its last field, so
>> > both `*mut T` and `*mut ArcInner<T>` have the same metadata. Because of
>> > that, moving the metadata over from one type to the other is ok.
>> >
>> > The reason that I cast to an `ArcInner<T>` pointer before calling
>> > `metadata` is because I get a type mismatch otherwise for the metadata,
>> > since the compiler doesn't unify the metadata types when the type is
>> > generic.  
>> 
>> OK, cool. In that case, since this is common knowledge (is it?),
>> could you maybe include a link to the relevant documentation, or a
>> comment indicating why this is OK?
>> 
>> BR Andreas
>
> This is documented in the doc of Pointee trait:
>
> https://doc.rust-lang.org/std/ptr/trait.Pointee.html

Nice. I think I forgot a _not_ in my last message. I think it would be
nice to add a comment with a link to this documentation and perhaps a
note as to why this works.

BR Andreas