Thoughts on methods to (de)construct fat pointers

[WaffleLapkin]

I'm currently working on a custom DSTs. In my opinion, std lacks some APIs for working with fat pointers which makes coding DSTs hard.

e.g. the one way to create *const Dst from the data part (*const ()) and the meta (vtable/length) part (usize) is to use core::slice::from_raw_parts and then cast it's result (the idea is stolen from dtolnay):

let slice = unsafe { core::slice::from_raw_parts(data, meta) };
slice as *const [()] as *const Dst

This works, though

• using slice seems like a hack
• while the layout of *const [T] seems to be defined, the layout of *const Dst doesn't seem to be defined, so the code doesn't seem right
• it doesn't allow to construct ptr to DST which contain dyn Trait

Another thing that I personally miss is extracting meta from a fat pointer.

More precisely I'm looking for similar APIs:

impl<T: ?Sized> *const T {
    // I'm bad at naming but hopefully you've got the idea :)
    
    /// ignores meta field for `T: Sized`
    pub fn fat_from_parts(data: *const (), meta: usize) -> Self;

    /// returns None for `T: Sized`
    pub fn meta(self) -> Option<usize>;
}

This could make my code a lot clearer

Though, design like this has some downsides:

• this more or less disallows fat pointers with meta that is not an usize (or pointer)
• this approach is much simpler/more dump/less typed than RFC for custom DSTs that I could find:
  • https://github.com/strega-nil/rust-rfcs/blob/custom-dst/text/0000-custom-dst.md
  • https://github.com/SimonSapin/rfcs/blob/ptr-meta/text/0000-ptr-meta.md

Is it worth creating an RFC? Or this is too restrictive?

[steffahn]

There is also the (still unstable / nightly-only) option of transmuting to/from TraitObject which probably doesn't solve all your problems, but it is a way to create pointers to dyn types and you didn't mention it.

[CAD97]

@WaffleLapkin

while the layout of *const [T] seems to be defined, the layout of *const Dst doesn't seem to be defined, so the code doesn't seem right

The layout of *const [T] is unspecified. The layout of *const T for T: Sized is what is guaranteed by the language.

However, given Dst that is unsized due to having a slice tail, *const [T] and *const Dst are defined to, while not necessarily be the same layout, to be compatible and cast correctly via as.

You can get the length of an arbitrary pointer-to-slice-tailed-dst by casting to *const [()] and asking for the length of that. So long as the pointer is nonnull, it's sound to call <[()]>::len on it, and iirc there's a pointer version of slice::len on nightly.

At this point I need to mention my crate, slice-dst, which is designed to make working with slice-tailed DSTs a bit easier.

Working with arbitrary DSTs is difficult, because the type of the metadata on fat pointers may not match up. The size of a slice is usize, the vtable for dyn trait objects is &'static TraitVtable, there's certainly interest in pointers-to-dsts with no metadata ("fixing CStr"), and vague interest in user-defined fat pointers with arbitrary metadata (e.g. &MatrixSlice -- width, height, and stride (width of the full thing for row-major)).

[WaffleLapkin]

@CAD97

The layout of *const [T] is unspecified.

But unsafe guidelines says that it's specified:

The layout of &[T] is the same as that of:

#[repr(C)]
struct Slice<T> {
  ptr: *const T,
  len: usize,
}

---

@CAD97

So long as the pointer is nonnull, it's sound to call <[()]>::len on it

It should be sound for null pointers too, the len is near the pointer, not behind it

@CAD97

At this point I need to mention my crate, slice-dst, which is designed to make working with slice-tailed DSTs a bit easier.

Sounds interesting, thanks, I'll check it out!

@CAD97

Working with arbitrary DSTs is difficult

Yeah, I wonder if there will be a solution to this sometime...

[dhm]

@WaffleLapkin

But unsafe guidelines says that it's specified:

The layout of &[T] is the same as that of: ...

The UCG are not normative, they represent a an intent of what Rust may aim to guarantee at some point.

@https://rust-lang.github.io/unsafe-code-guidelines/introduction.html

Rust's Unsafe Code Guidelines Reference

This document is produced by the UCG WG to provide a "guide" for writing unsafe code that "recommends" what kinds of things unsafe code can and cannot do, and that documents which guarantees unsafe code may rely on. It is largely a work-in-progress right now.

Unless we state otherwise, the information in the guide is mostly a "recommendation" and still subject to change. The main goal for the WG in 2019 is to RFC parts of the guide into the language specification, and move those parts into the Rust Language Reference proper.

So, what we actually have, is rather:

The layout of $[T] is is likely to end up guaranteed to be that of: ...

That is, any verb inside the UCG currently should be prefixed with is likely to end up guaranteed to:

• "end up" to express that this is not yet guaranteed,

• "likely", to express that this is not fixed in stone, and some design / proposal could argue against that stabilization.

---

But in a way, that's all the more reason to try and push some of these "layout-agnostic" APIs!

[CAD97]

@WaffleLapkin

It should be sound for null pointers too, the len is near the pointer, not behind it

You have to use the raw pointer version if the pointer is null or unaligned. <[()]>::len takes &[()], so the reference must be nonnull and aligned.

[RalfJung]

@dhm

That is, any verb inside the UCG currently should be prefixed with is likely to end up guaranteed to

oops usually we also say that explicitly on pages that contain non-RFC'd information, but looks like we forgot that here. Sorry for this.

Opened add note that pointer layout information is preliminary by RalfJung · Pull Request #249 · rust-lang/unsafe-code-guidelines · GitHub to fix this.

[tcsc]

@WaffleLapkin

Is it worth creating an RFC? Or this is too restrictive?

I think this prevents ever implementing something like Custom Dynamically Sized Types for Rust by strega-nil · Pull Request #1524 · rust-lang/rfcs · GitHub, without solving most of the problems it attempts to address, unfortunately.

In particular I think it would be good to keep the possibility for other kinds of non-Sized types than ones with two pointers (e.g. metadata shouldn't always be assumed to be either None or Some(usize). IMO it's better to just avoid defining this at all if possible, so it can be addressed later).

IMO the functionality you're describing should be done for concrete cases: e.g:

1. We need raw parts APIs for constructing/deconstructing *const [T]/*mut [T].

2. Trait objects need an unsafe API for constructing/deconstructing them (to/from *const dyn Foo/*mut dyn Foo).

3. Structs which have a unsized trailing member should guarantee that any metadata they have is equivalent to the metadata the trailing member would have on its own:

   In particular, code exists in the wild which assumes that given struct MySlice(Stuff, [T]), &*(slice::from_raw_parts(pointer_to_myslice_instance, length_of_trailing_slice) as *const MySlice) is legal. In practice this works, but I believe it's not guaranteed (although it seems hard to imagine it wouldn't be implemetned some other way since other ways would mean as on pointers would need to make updates to len, I think).

   It would be good if this were guaranteed, since then the APIs in 1 and 2 + casting would be enough for these cases.

I'd really prefer we didn't cut off the ability to do something better here in the future, there are a lot of really great use cases for custom DSTs, and so we should try not to block that path off with whatever we do here.

[CAD97]

@tcsc

Structs which have a unsized trailing member should guarantee that any metadata they have is equivalent to the metadata the trailing member would have on its own:

I think this is actually guaranteed. If you try to as cast between cases where the metadata isn't guaranteed to be the same, you get a "pointer metadata may not be the same" error.

[WaffleLapkin]

I've re-read the RFCes and now I agree on the fact that we should leave space for them

Hope some RFC will be accepted in the (near) future.