pre-RFC FromBits/IntoBits

[rkruppe]

You can use union MaybeInit<T> { uninit: (), x: T } to get uninitialized memory and then get a &[u8] from that if FromBits<MaybeInit<Foo>> is implemented for [u8] .

Ah, right, thanks.

But chances are that this is safe for all types if you implement it properly, preferably by just using ptr::{read,write} . These functions might be special in the memory model if they can read uninitialized memory and copy it into a destination without making the memory in the destination initialized.

These functions are not special in this sense, any loads and stores should have this property. ptr::{read,write} are at most special in that they are the primitive way to express “write without dropping old contents first” (in contrast to *p = x; which drops) and “read without concern for move semantics” (in contrast to let x = *p; which only works for Copy types). But this isn’t really true either, at least of ptr::read (check out its implementation, you could write that in stable Rust today).

---

Aside from that, the IRC discussion @gnzlbg alludes to was about raw pointer loads and stores. When you add references in the mix, it gets a lot more complicated. While it’s probably fine to cast a *const T to a *const [u8; size_of::<T>()] and copy around any of those bytes even if they’re padding or uninitialized, a &[u8] to the same memory is a rather different story. References have quite strong invariants both about their address (+ metadata, if any) and about the concents of the memory they point at – the main open questions are about when exactly these invariants are asserted (e.g., “all the time”, “at function call boundaries”, “when you load or store through the reference”, etc.).

To say that a &[u8] to padding or unintialized memory is fine amounts to saying there is no such thing as uninitialized or padding memory, every byte of memory always one of 256 possible values and it’s safe to do anything with any of these bytes. I don’t want to rehash that debate here but this option is very radical and, I believe, unacceptable for Rust because of how much it constrains the optimizer.

Thus, I don’t think a &T -> &[u8] conversion could be safe for all T even if there was then no way to reinterpret the &[u8] as any other type (e.g. imagine it was a trait object for trait Blob { fn get_byte(idx: usize) -> u8; }). Or, put differently, there needs to be some way to correctly handle padding and uninitalized memory in unsafe code, but I don’t see any (desirable) way to give safe code such capabilities.

[joshlf]

Thus, I don’t think a &T -> &[u8] conversion could be safe for all T even if there was then no way to reinterpret the &[u8] as any other type (e.g. imagine it was a trait object for trait Blob { fn get_byte(idx: usize) -> u8; } ). Or, put differently, there needs to be some way to correctly handle padding and uninitalized memory in unsafe code, but I don’t see any (desirable) way to give safe code such capabilities.

Would it be safe to have a function like this?

fn move_into<T>(t: T, dst: &mut [u8]) {
    // verify that dst is large enough
    // write t's bytes into *dst and forget t
}

[rkruppe]

Would it be safe to have a function like this?

If we take the quoted belief for granted, no, it wouldn’t be safe. It’s basically the same thing, after all.

[joshlf]

My understanding from the previous discussion is that it’s safe to copy uninitialized bytes into a [u8], at which point the [u8] is considered initialized? Maybe I misread that.

[rkruppe]

I don’t know where you read that, but I would argue against it. It still overly constrains the optimizer (you either have to stop doing many memcpy and load-store optimizations, or you get all the same restrictions that “there’s no such thing as uninitialized memory” implies).

[joshlf]

OK, so it sounds like a) Rust differs from C/LLVM in that reading uninitialized memory into “character types” is UB and, b) we should just go full bore on disallowing going from T -> U if U has data where T has padding or other possibly uninitialized bytes?

[gnzlbg]

But can’t you do that via a pointer instead of a reference?

fn move_into<T>(t: T, dst: *mut [u8]) {
    // verify that dst is not null
    // verify that *dst is large enough
    // write t's bytes into *dst and forget t
}

[rkruppe]

a) Rust differs from C/LLVM in that reading uninitialized memory into “character types” is UB and

No! As I said before, unsafe code needs to be able to do some things with uninitialized memory and especially with padding. Simply loading from it and storing the loaded value elsewhere should definitely be among the things possible. The rationale for that is that it needs to be possible to implement e.g. memcpy in Rust. It’s just that doing many other things with padding/uninitalized bytes should probably be UB, or otherwise dangerous enough that we can’t allow safe code to do it arbitrarily.

But can’t you do that via a pointer instead of a reference?

This function isn’t safe because dst can be non-null and dangling, but supposing it’s okay to write size_of::<T> bytes through dst, then yes, my gut feeling is that this function would be fine. Turning the *mut [u8] into a reference to [u8] or some other type might be unsound, though.

[joshlf]

But can’t you do that via a pointer instead of a reference?

This function isn’t safe because dst can be non-null and dangling, but supposing it’s okay to write size_of::<T> bytes through dst , then yes, my gut feeling is that this function would be fine. Turning the *mut [u8] into a reference to [u8] or some other type might be unsound, though.

In that case, couldn’t you just implement my proposed function by first converting the &mut [u8] into a *mut [u8]?

[rkruppe]

In that case, couldn’t you just implement my proposed function by first converting the &mut [u8] into a *mut [u8] ?

Having the &mut reference probably implies a lot more than having a raw pointer to the same memory. See @RalfJung’s Types as Contracts work for one specific proposal for what (and when) this means, though it doesn’t talk a lot about initialized-ness specifically. I don’t think those semantics would actually have a problem with this specific example, but might have a problem with e.g. a caller that does move_into(t, &mut buffer); and then passes buffer to another function.

To be clear, there will necessarily be some loopholes for taking a reference to some invalid memory and quickly turning it into a raw pointer (since in current Rust you can’t create a raw pointer without first going through references). but having a reference for an extended period of time, across (e.g.) multiple crates, as the safe APIs discussed in this thread will allow, will certainly require the reference to point to “valid” memory. What that entails is up for discussion but as I said I really hope & expect it will exclude treating uninitialized or padding bytes as u8 values.

[joshlf]

Yeah, I’m definitely leaning more and more towards just banning it outright and requiring manual (or derived) impls for the subset of types for which it’s safe.

[rkruppe]

If I rediscovered this link earlier I would have brought it up somewhere appropriate, but it’s still relevant so I’ll just drop it here: https://github.com/nikomatsakis/rust-memory-model/issues/42

[joshlf]

Related question: if we’re going to say that the uninitialized bytes of a given type must not correspond to the data bytes of another type, what exactly does that mean? For example, the following is probably fine:

#[repr(C)] struct T { a: u8, /* padding byte */, b: u16 }
#[repr(C)] struct U { a: u8, /* padding byte */, b: u16 }
unsafe impl FromBits<T> for U {}

because the padding byte is in the same position in both types. However, is it valid to convert an enum or union to a struct so long as the possibly-uninitialized bytes of the former correspond to padding bytes of the latter? In other words, are all possibly-uninitialized bytes alike, and converting between them is fine, or are different “types” of possibly-uninitialized bytes distinct, and so converting between them is not necessarily safe in the general case?

[joshlf]

OK, I’ve started working on a draft RFC. The first draft is not complete, but you folks may have feedback nonetheless. https://github.com/joshlf/rfcs/blob/joshlf/from-bits/text/0000-from-bits.md

A few things to note:

• I’ve decided to include SizeLeq and AlignLeq in the proposal because I think they give us a lot more power, but I could be convinced to remove them if folks think the proposal encompases too much. A good compromise might be to just remove SizeLeq since it’s less useful than AlignLeq, although my preference is to keep them both.
• I’ve opted to include a derive for FromBits in addition to it being an auto trait. This is so that authors of types with private fields can still opt in if they want.

[newpavlov]

Can’t we utilize From trait for this problem by making a wrapper type Bits<T> with appropriate From implementations for numeric types? So instead of f32::from_bits(0x0_i32) we will write f32::from(Bits(0x0i32)).

On a side note I would really like to see as operator to be a sugar for Into trait, for example it could be really convenient to use it with units of measure: let dist_meters = dist_miles as Meter;. What is the main reason for not doing it?

[gnzlbg]

Can’t we utilize From trait for this problem by making a wrapper type Bits<T> with appropriate From implementations for numeric types?

I think that might work. Users would need to impl From<Bits<T>> for U and the transitivity magic would need to work on that.

On a side note I would really like to see as operator to be a sugar for Into trait, […] What is the main reason for not doing it?

These are just different types of conversions: as performs fallible non-value-preserving zero/sign extending and truncating “conversions”, while From and Into perform value-preserving infallible conversions.

[joshlf]

Can’t we utilize From trait for this problem by making a wrapper type Bits<T> with appropriate From implementations for numeric types? So instead of f32::from_bits(0x0_i32) we will write f32::from(Bits(0x0i32)) .

Hmmm interesting. You could do something like #[repr(transparent)] struct Bits<T>(T) and then give Bits<T> two from_ref(&T) -> &Bits<T> and from_mut(&mut T) -> &mut Bits<T> constructors so it’d work for references too.

The big question I’d have is: how do you construct the From impl? One of the things that I like about this proposal is that, if we go with having either compiler assistance or a custom derive, the user doesn’t have to reason about the (very subtle and complex) memory safety themselves.

Also, since From and From::from are safe, there’s nothing stopping somebody from implementing From<Bits<T>>::from in a way that doesn’t actually depend on the argument, but instead produces some default value. That, in turn, means that you can no longer use U: From<T> as a signal that it’s safe to coerce a reference to T into a reference to U.

[gnzlbg]

The big question I’d have is: how do you construct the From impl?

If coerce would work with references then maybe something like:

impl From<Bits<T>> for U where U: FromBits<T> {
    #[inline]
    fn from(x: Bits<T>) -> Self {
        coerce(x.0)  // EDIT: fixed bug, had coerce(x) before
    }
}

[joshlf]

If coerce would work with references then maybe something like:

impl From<Bits<T>> for U where U: FromBits<T> {
    #[inline]
    fn from(x: Bits<T>) -> Self {
        coerce(x)
    }
}

So how does coerce work then? It looks like it’s a safe function, which means that there’s some mechanism for deciding whether coerce::<T, U> is valid for T and U.

[gnzlbg]

So how does coerce work then? It looks like it’s a safe function, which means that there’s some mechanism for deciding whether coerce::<T, U> is valid for T and U .

The same way that it is currently implemented:

fn coerce<T, U>(x: T) -> U where U: FromBits<T>

For references one needs to provide different blanket impls of From. I don’t know if that can be done with specialization or not.

EDIT: had a bug in the way coerce is called, but like this it should work because the constraints are the same:

impl From<Bits<T>> for U where U: FromBits<T> {
    #[inline]
    fn from(x: Bits<T>) -> Self {
        coerce(x.0)
    }
}