[Pre-RFC] Custom DSTs

[gnzlbg]

@kennytm @ubsan is there a need for a DST working group ? It might make sense to just open an RFC repo for this given the amount of work that has been done already on this subject (many RFCs, pre-RFCs, discussions, etc.) so that those interested can work together in a proposal that has consensus at least among those trying to solve this problem.

[ubsan]

In the introduction you might want to mention “(similar to C VLAs)” since VLAs is what people know

To be clear, this does nothing for VLAs. FAMs and VLAs are very, very different features, the first of which is super useful and awesome, the second of which is widely agreed to be a mistake and a massive security risk.

Is this a breaking change ?

no, because I don’t actually recommend making this change

I’m just saying it would now be more correct. It would be a breaking change since ?Sized would no longer imply you could call size_of_val on this type.

[Ixrec]

the second of which is widely agreed to be a mistake and a massive security risk

Maybe slightly off-topic, but I somehow completely missed this memo; what’s wrong with VLAs?

[ubsan]

Fairly good write-up here: https://stackoverflow.com/a/21519062

I may have overstated it a bit

But yeah, this proposal has nothing to do with VLAs.

[gnzlbg]

It would be a breaking change since ?Sized would no longer imply you could call size_of_val on this type.

Is this behavior worth deprecating so that we can make this breaking change in the future ?

To be clear, this does nothing for VLAs. FAMs and VLAs are very, very different features, the first of which is super useful and awesome, the second of which is widely agreed to be a mistake and a massive security risk.

I don’t disagree, my point was only that FAMs is something that most people haven’t heard about, but VLAs is. I don’t know how to word this, but something like “similar to the good parts of VLAs” or something like that might give people some context to know what this is about.

[ubsan]

Is this behavior worth deprecating so that we can make this breaking change in the future ?

I strongly believe that it is. The existing proposed behavior of panicking/aborting when called with an invalid type is, frankly, not great.

[ubsan]

If anybody would like to write examples, I’m looking for more examples, as well

[comex]

That’s good, thanks. I don’t think I really understand why that would be the case, though. This RFC draft doesn’t go into much detail on what size_of_val and align_of_val would be used for. What sort of code would you expect to call those functions, and what guarantees can that code depend on? It might be useful to add that information to the RFC, since there seems to be some confusion about it.

One prominent user of size_of_val is Rc::from_box, which converts a Box<T> to Rc<T> by making a new heap allocation and copying the contents (because Rc needs extra space for the reference count), and works for unsized types. It’s also used by Layout::for_value, which is indirectly used in some other places in the standard library for similar purposes.

This functionality has been a headache in general; see also:

Issue: `extern type` cannot support `size_of_val` and `align_of_val`

IIRC there was at least one scenario where existing code could call it on borrowed references, not just owned smart pointers… or maybe it was just hypothetical; I can’t remember.

[gnzlbg]

This RFC might want to explicitly include what’s guaranteed about the layout of custom DSTs, if anything at all.

[ubsan]

There are no guarantees about the layouts of pointers to custom DST, that’s the unsafe code guidelines team’s job.

[gnzlbg]

Your comments in the unsafe code guidelines discussions appear to suggest that the pointer and the metadata need to be disjoint, maybe hinting that unsafe code should be able to rely on, e.g., that the first field of a custom DST is a pointer value.

Do you think it might make sense to support encoding the metadata in the pointer ? For example, implementing a “short slice” where the slice length is encoded in the pointer ?

[ubsan]

If that were to be implemented, one would have to use Metadata = (). The question is then, is the pointer allowed to be invalid; not the representation of the DST.

[gnzlbg]

The question is then, is the pointer allowed to be invalid; not the representation of the DST.

In general we wanted to support encoding information in unused pointer bits, but I don’t think that discussions considered custom DSTs.

Unsafe code can already assume that a &T always points to a valid T. For DSTs we can specify where in the layout the DST pointer lives. If we allow custom DSTs to contain invalid pointers (which is what encoding information in unused pointer bits produces), then unsafe code cannot assume that just because they get the pointer from a &T this pointer is, in general, dereferenceable. It might do so for concrete DSTs that guarantee that this is always the case though.

[ubsan]

There’s no way to, given α : !Sized, β : Sized safely go from &α → &β, in general. Therefore, the pointer that the DST pointer contains doesn’t necessarily need to be valid under any specific &β.

[rkruppe]

I don’t really follow. It seems to me custom DSTs should, in fact, follow some discipline wrt the “data pointer” part, because the whole point of custom DSTs is that they are agnostic to the kind of pointer that refers to them and just enrich it with additional metadata in some way.

For example, if we want ThinBox<CustomDST> to work, it doesn’t make any sense to let a custom DST stash something extra in “the pointer” – ThinBox is just a newtype around a plain old *mut (), pointing to an allocation that contains first the metadata and then (after suitable padding) the actual unsized data.

[ubsan]

When I say “doesn’t necessarily need to be valid”, I don’t mean “we won’t require it to be valid”, just that the design doesn’t require it.

[rkruppe]

I’m not even talking about invariants right now, I don’t even see how “a custom DST that packs some metadata into the ‘data pointer’ portion of references to itself” could even work in the first place. I guess the idea is to rely on those references being produced and consumed by the impl DynamicallySized for TheDST (and thereby special-casing &T-references over all other kinds of pointers)?

But then what about consumers that need to get the address of the size_of_val()-sized, align_of_val()-aligned chunk of memory that is the DST value? To give just one example, Box::drop needs that address (along with size and align) to deallocate memory, and currently it gets it by extracting the “pointer” portion (discarding the metadata).

There’s a way out of this problem: making this conversion from “a &DST what means whatever DST wants it to mean” to “the actual address of the referent” another method of the trait, but at that point it stops being just about dynamic size and starts being closer to overloading the reference type entirely (e.g. tagged pointers that pretend to be regular references are just as attractive for some sized types).

---

BTW, about this:

There’s no way to, given α : !Sized, β : Sized safely go from &α → &β , in general. Therefore, the pointer that the DST pointer contains doesn’t necessarily need to be valid under any specific &β .

This is a quite fragile property. For example, once we decide what to do about padding bytes by e.g. allowing &MaybeUninit<u8> to point to padding bytes (and that this is not only valid but also upholds the safety invariant, in @RalfJung’s terminology), the only reason one can’t cast every &T where T: !Sized to a &[MaybeUninit<u8>; N] is that one doesn’t know a priori what the right N is. But one can guess and check at runtime, so for every N there’s at least a sound &T -> &Option<[MaybeUninit<u8>; N]> conversion which returns Some whenever size_of_val >= N. Admittedly there’s not a lot one can do with that (because who knows which of those bytes are initialized) but I hope it illustrates that this is quite subtle.

[comex]

But then what about consumers that need to get the address of the size_of_val() -sized, align_of_val() -aligned chunk of memory that is the DST value?

Isn’t that what as casts (search for cast_to_thin) are for?

but at that point it stops being just about dynamic size and starts being closer to overloading the reference type entirely (e.g. tagged pointers that pretend to be regular references are just as attractive for some sized types).

That’s not what this is proposing, but… would that be such a bad thing?

[RalfJung]

That’s not what this is proposing, but… would that be such a bad thing?

It’s somewhat off-topic in this RFC, which is about DST specifically.

Moreover, we already have newtypes, so if you want a reference-like thing, just make your own type and implement Deref and DerefMut. Maybe add some arbitrary-self for extra measure. Why would we want more than this?

[rkruppe]

Isn’t that what as casts (search for cast_to_thin ) are for?

Yes, my point is that (at minimum) that operation would have to become overloadable as well to make this line of thinking work.

That’s not what this is proposing, but… would that be such a bad thing?

While I’d have to see a specific fleshed-out proposal to say anything about its merits, anything I can imagine in this space suffers from an unconvincing reward vs complexity ratio. In addition to what @RalfJung said, there’s no clear way to generalize this sort of pointer bit fiddling to library-defined smart pointers such as Box (well, almost library-defined) or Rc even when one can in principle imagine what a “tagged variant” of such a pointer would look like (which isn’t true for all smart pointers).