[Pre-RFC] Custom DSTs

[RalfJung]

Just some minor editorial notes:

// size_of_val(&foo) returns size_of_val(&foo.y) // same with align_of_val

I assume it returns size_of::<usize>() + size_of_val(&foo.y)?

we will also introduce four functions

You are listing five.

[gnzlbg]

• The size_of_val and align_of_val declarations are now incorrect; they should take T: DynamicallySized , as opposed to T: ?Sized .

Is this a breaking change ?

All other points are just nitpicks that you might want to improve for the RFC. Thank you a lot for working on this. This is great.

---

In the Foo example, size_of_val(&foo) result appears incorrect (the size of usize is missing) - would be cool to work out this example fully for the RFC.

---

Are std::raw::ptr and std::raw::ptr_mut necessary? You can get this behavior with as.

You just said they aren’t, so why add them ? People should be using as for this already, so I think it would be better to just add an example that shows that as just works.

---

How should these thick pointers be passed, if they are larger than two pointers?

IIUC these don’t have to be C FFI safe for the short-term goals of the RFC, so we would be talking here about repr(Rust) only. I don’t see any reason why these couldn’t be passed as aggregates with an unspecified layout. After the RFC, we could specify these further as part of the unsafe code guidelines, but this does not have to be done right now.

---

we will also introduce four functions into the standard library, in std::raw:

I don’t see any reason to put these in libstd and not libcore, so that should probably be core::raw

---

Flexible Array Members

In the introduction you might want to mention “(similar to C VLAs)” since VLAs is what people know, but FAM is probably a too technical term, for the introduction at least. Linking the FAM paper (and C VLAs) from the introduction might be worth it.

indexing and slicing 2-d arrays

If you need more examples, you can add one for writing a DST for 2d arrays - that shouldn’t be too hard. If you want to mention more use cases, multi-dimensional slices in general, and maybe “growable” slices (e.g. a slice into a growable Vec/ArrayVec/SmallVec with fixed-capacity).

---

Prior art

I know you are aware of all previous RFCs on this topic, so you might want to reference some of that here. In particular, the last RFC about this, and @nikomatsakis comments about it at the end, and maybe address what has changed.

[gnzlbg]

@kennytm @ubsan is there a need for a DST working group ? It might make sense to just open an RFC repo for this given the amount of work that has been done already on this subject (many RFCs, pre-RFCs, discussions, etc.) so that those interested can work together in a proposal that has consensus at least among those trying to solve this problem.

[ubsan]

In the introduction you might want to mention “(similar to C VLAs)” since VLAs is what people know

To be clear, this does nothing for VLAs. FAMs and VLAs are very, very different features, the first of which is super useful and awesome, the second of which is widely agreed to be a mistake and a massive security risk.

Is this a breaking change ?

no, because I don’t actually recommend making this change

I’m just saying it would now be more correct. It would be a breaking change since ?Sized would no longer imply you could call size_of_val on this type.

[Ixrec]

the second of which is widely agreed to be a mistake and a massive security risk

Maybe slightly off-topic, but I somehow completely missed this memo; what’s wrong with VLAs?

[ubsan]

Fairly good write-up here: https://stackoverflow.com/a/21519062

I may have overstated it a bit

But yeah, this proposal has nothing to do with VLAs.

[gnzlbg]

It would be a breaking change since ?Sized would no longer imply you could call size_of_val on this type.

Is this behavior worth deprecating so that we can make this breaking change in the future ?

To be clear, this does nothing for VLAs. FAMs and VLAs are very, very different features, the first of which is super useful and awesome, the second of which is widely agreed to be a mistake and a massive security risk.

I don’t disagree, my point was only that FAMs is something that most people haven’t heard about, but VLAs is. I don’t know how to word this, but something like “similar to the good parts of VLAs” or something like that might give people some context to know what this is about.

[ubsan]

Is this behavior worth deprecating so that we can make this breaking change in the future ?

I strongly believe that it is. The existing proposed behavior of panicking/aborting when called with an invalid type is, frankly, not great.

[ubsan]

If anybody would like to write examples, I’m looking for more examples, as well

[comex]

That’s good, thanks. I don’t think I really understand why that would be the case, though. This RFC draft doesn’t go into much detail on what size_of_val and align_of_val would be used for. What sort of code would you expect to call those functions, and what guarantees can that code depend on? It might be useful to add that information to the RFC, since there seems to be some confusion about it.

One prominent user of size_of_val is Rc::from_box, which converts a Box<T> to Rc<T> by making a new heap allocation and copying the contents (because Rc needs extra space for the reference count), and works for unsized types. It’s also used by Layout::for_value, which is indirectly used in some other places in the standard library for similar purposes.

This functionality has been a headache in general; see also:

Issue: `extern type` cannot support `size_of_val` and `align_of_val`

IIRC there was at least one scenario where existing code could call it on borrowed references, not just owned smart pointers… or maybe it was just hypothetical; I can’t remember.

[gnzlbg]

This RFC might want to explicitly include what’s guaranteed about the layout of custom DSTs, if anything at all.

[ubsan]

There are no guarantees about the layouts of pointers to custom DST, that’s the unsafe code guidelines team’s job.

[gnzlbg]

Your comments in the unsafe code guidelines discussions appear to suggest that the pointer and the metadata need to be disjoint, maybe hinting that unsafe code should be able to rely on, e.g., that the first field of a custom DST is a pointer value.

Do you think it might make sense to support encoding the metadata in the pointer ? For example, implementing a “short slice” where the slice length is encoded in the pointer ?

[ubsan]

If that were to be implemented, one would have to use Metadata = (). The question is then, is the pointer allowed to be invalid; not the representation of the DST.

[gnzlbg]

The question is then, is the pointer allowed to be invalid; not the representation of the DST.

In general we wanted to support encoding information in unused pointer bits, but I don’t think that discussions considered custom DSTs.

Unsafe code can already assume that a &T always points to a valid T. For DSTs we can specify where in the layout the DST pointer lives. If we allow custom DSTs to contain invalid pointers (which is what encoding information in unused pointer bits produces), then unsafe code cannot assume that just because they get the pointer from a &T this pointer is, in general, dereferenceable. It might do so for concrete DSTs that guarantee that this is always the case though.

[ubsan]

There’s no way to, given α : !Sized, β : Sized safely go from &α → &β, in general. Therefore, the pointer that the DST pointer contains doesn’t necessarily need to be valid under any specific &β.

[rkruppe]

I don’t really follow. It seems to me custom DSTs should, in fact, follow some discipline wrt the “data pointer” part, because the whole point of custom DSTs is that they are agnostic to the kind of pointer that refers to them and just enrich it with additional metadata in some way.

For example, if we want ThinBox<CustomDST> to work, it doesn’t make any sense to let a custom DST stash something extra in “the pointer” – ThinBox is just a newtype around a plain old *mut (), pointing to an allocation that contains first the metadata and then (after suitable padding) the actual unsized data.

[ubsan]

When I say “doesn’t necessarily need to be valid”, I don’t mean “we won’t require it to be valid”, just that the design doesn’t require it.

[rkruppe]

I’m not even talking about invariants right now, I don’t even see how “a custom DST that packs some metadata into the ‘data pointer’ portion of references to itself” could even work in the first place. I guess the idea is to rely on those references being produced and consumed by the impl DynamicallySized for TheDST (and thereby special-casing &T-references over all other kinds of pointers)?

But then what about consumers that need to get the address of the size_of_val()-sized, align_of_val()-aligned chunk of memory that is the DST value? To give just one example, Box::drop needs that address (along with size and align) to deallocate memory, and currently it gets it by extracting the “pointer” portion (discarding the metadata).

There’s a way out of this problem: making this conversion from “a &DST what means whatever DST wants it to mean” to “the actual address of the referent” another method of the trait, but at that point it stops being just about dynamic size and starts being closer to overloading the reference type entirely (e.g. tagged pointers that pretend to be regular references are just as attractive for some sized types).

---

BTW, about this:

There’s no way to, given α : !Sized, β : Sized safely go from &α → &β , in general. Therefore, the pointer that the DST pointer contains doesn’t necessarily need to be valid under any specific &β .

This is a quite fragile property. For example, once we decide what to do about padding bytes by e.g. allowing &MaybeUninit<u8> to point to padding bytes (and that this is not only valid but also upholds the safety invariant, in @RalfJung’s terminology), the only reason one can’t cast every &T where T: !Sized to a &[MaybeUninit<u8>; N] is that one doesn’t know a priori what the right N is. But one can guess and check at runtime, so for every N there’s at least a sound &T -> &Option<[MaybeUninit<u8>; N]> conversion which returns Some whenever size_of_val >= N. Admittedly there’s not a lot one can do with that (because who knows which of those bytes are initialized) but I hope it illustrates that this is quite subtle.

[comex]

But then what about consumers that need to get the address of the size_of_val() -sized, align_of_val() -aligned chunk of memory that is the DST value?

Isn’t that what as casts (search for cast_to_thin) are for?

but at that point it stops being just about dynamic size and starts being closer to overloading the reference type entirely (e.g. tagged pointers that pretend to be regular references are just as attractive for some sized types).

That’s not what this is proposing, but… would that be such a bad thing?