pre-RFC FromBits/IntoBits

[Centril]

As @ExpHP wrote, an unsafe trait should be used when and means that there are some documented non-executable invariants that are being relied upon in unsafe { .. }. This does not seem to be the case here, wherefore the traits should not be unsafe.

[gnzlbg]

Why are the traits unsafe? Normally an unsafe trait provides additional guarantees that can be relied on by unsafe code. The only guarantees I see here are “the methods are safe to call,” which is already guaranteed by the fact that the methods are not unsafe.

Makes sense, I’ve removed these. FWIW in stdsimd these traits are safe.

---

@mark-i-m the trait methods are safe, so their implementation must maintain safety.

---

@scottmcm

Everything listed is already available in safe code via either as

as is not implementable for user-defined types. An alternative would be to provide one trait that allows using as with user-defined types.

Why not also do things like &'a i32 to &'a u32?

I just implemented the trait for the types I cared about, but the list isn’t comprehensive. Nothing prevents the addition of more implementation later. If you want to propose more, ideally, you would specify exactly for which types you want this implemented. If it makes sense I’ll edit the list.

This like:

impl<'a> FromBits<&'a i32> for &'a u32 { }
impl<'a> FromBits<&'a mut i32> for  &'a mut u32 { }

sound good to me.

---

@jmst

A much more general design seems to have an auto trait that signals that all bits pattern are valid for a data type

These traits support cases for which not all bit-patterns are valid, like the reference above (a null reference would be invalid). In any case, I don’t think I fully understand what you proposed, so maybe you could reformulate it in a different way? AFAICT you propose a trait that says whether transmuting into [u8; size_of::<T>()] is safe, and therefore, all types for which this is safe, can be safely transmuted into one another. But I am not sure if this is what I meant.

---

@zackw I will add some examples of this for the RFC. This is a part of the portable SIMD rfc that’s not written yet, so I couldn’t just copy it over here.

[gnzlbg]

Right now all usages in stdsimd require that the size of both types must be equal. I left that requirement provisionally out here, but I think these traits should maintain them.

The example of i32 to u64 shows the issue: there are many ways to do that, two of them often useful: zext and sext.

These two traits would, however, only be able to express one of them. AFAICT if these traits require the same size, because the bit-pattern preserving requirement, there is only one way to do this, and this way must be “invertible”, so one only needs to implement the trait for one permutation and a blanket impl could be used to provide the other.

[gnzlbg]

This also seems like a similar problem statement to these crates, which work totally differently… https://docs.rs/plain/ https://docs.rs/pod/

So plain looks larger in scope than this, allowing also to convert to/from u8 arrays and slices, and pod does look even larger in scope than plain. @scottmcm I think that this meshes also with the comment that @jmst made, and it does make sense to consider a path forward here that can be seamlessly extended to those use cases, or that at least those use cases can build on.

I find the approach of having a single unsafe trait appealing:

/// Marker trait that indicates that all 
/// bit-patterns of a type are valid. 
unsafe trait Raw {
   fn from_raw<T: Raw>(x: T) -> Self 
     where mem::size_of::<T>() == mem::size_of::<Self>() {
       unsafe { mem::transmute(x) }
   }
   fn into_raw<T: Raw>(self) -> T 
     where mem::size_of::<T>() == mem::size_of::<Self>() {
       unsafe { mem::transmute(self) }
   }
}

[jmst]

These traits support cases for which not all bit-patterns are valid, like the reference above (a null reference would be invalid).

If that’s necessary, then a T: Compatible<U> “auto” trait seems better (the idea is that the compiler automatically implements T: Compatible<U> if T can be safely transmuted to U and obviously viceversa - perhaps “Transmutable” could be an option as a name as well).

It would work like FromBits (but with an external safe transmute method conditional to the trait being implemented), except the compiler would automatically provide it.

The upside is that unlike FromBits, which seems completely infeasible to implement for all possible pairs, it would always be available when it’s appropriate. The downside is that it requires a compiler change.

[gnzlbg]

@zackw I’ve added an example to the RFC.

[kornel]

This doesn’t work for slices/arrays/vectors.

When I really need casting/transmuting is for conversions between [T] and [u8].

So I’d like a trait such as trait Primitive: Copy that is (auto?) implemented for all types that can be harmlessly cased to and from a bunch of bits, and then on top of that it will be safe to implement helper functions converting T, as well as [T], etc.

[gnzlbg]

The upside is that unlike FromBits, which seems completely infeasible to implement for all possible pairs, it would always be available when it’s appropriate.

I don’t understand how that would work. Suppose I define the following type:

// This struct can only contain one value (42, 42)
struct Foo(i32, i32);

impl Foo {
    pub fn new(x: i32, y: i32) -> Self {
        assert!(x == 42, y == 32);
        Foo(x, y)
    }
}

which as you see can only contain one value Foo(42, 42), an invariant maintained by the new constructor, which can be relied by unsafe code for correctness.

How would the compiler know that this type is not “compatible” with (i32, i32) ?

[gnzlbg]

How would you transmute two regions of memory of different sizes into each other?

[kornel]

I do that with slice.as_ptr() and slice_from_raw_parts with length adjusted by size_of

T to u8 AFAIK would always be safe (the other way round can fail due to required alignment).

I’m also interested in conversion between [T] and [Newtype<T>], and [T] to [[T; 3]]. I don’t mind using unsafe and upholding invariants of slices, but I want to support arbitrary user-supplied T type for which this is safe.

[gnzlbg]

@kornel What I meant is, how do you deal with this (assuming alignment would be right):

pub struct Foo(u8, u8);
let xs = [0_u8; 1];
let Foos: &[Foo] = (&xs).into_bits();   // ?

That is, I don’t understand how safe infallible conversions between slices could work because the length of a slice is not part of its type, so at best such a conversion would need to be fallible and return an Option (and would need to be accounted for by TryFromBits, or similar traits, not discussed here). In the case above, I don’t see a clear way of safely “transmuting” a slice of 1 u8 to a slice of Foo where each Foo has a size of 2 u8s. So what am I missing?

[jmst]

The compiler sees that the fields are not public, and types with any private field would not be compatible with anything except trivially their own exact type (and maybe the same type with different generic parameters, if the private fields had compatible types and if an attribute to enable that was manually placed on the type).

[gnzlbg]

The compiler sees that the fields are not public, and types with any private field would not be compatible with anything (except maybe with same type with different generic parameters, if the private fields had compatible types and if an attribute to enable that was manually placed on the type).

I see. The question then becomes: how do we offer this functionality for types with private fields? FromBits/IntoBits allow this. That is, given the previous example:

// This struct can only contain one value (42, 42)
struct Foo(i32, i32);
impl Foo {
    pub fn new(x: i32, y: i32) -> Self {
        assert!(x == 42, y == 42);
        Foo(x, y)
    }
}

// This struct can contain all permutation pairs of 42, 7: (42, 42)
// (7, 7), (7, 42), (42, 7)
struct Bar(u32, u32);
impl Bar {
    pub fn new(x: u32, y: u32) -> Self {
        assert!(x == 42 || x == 7, y == 42 || y == 7);
        Foo(x, y)
    }
}

impl FromBits<Foo> for Bar { ... } // This is safe and correct
// impl FromBits<Bar> for Foo { ... } // can't be implemented

Transmuting from Foo to Bar is ok while the opposite is not true.

We use this property of FromBits/IntoBits in stdsimd a lot actually. For example, transmuting from b8x8 (a boolean mask of 8 8-bit integers) into i8x8 is ok, but the opposite isn’t (not al i8x8 bit patterns are valid b8x8 ones). Also, all fields of SIMD types are private. You might use a native 128-bit type on one architecture, but emulate it using 4x 32-bit types in another, so the exact internal representation isn’t something we want to expose.

[jmst]

I think both the Foo/Bar and SIMD cases can be addressed by allowing to specify compatibility explicitly and then automatically closing the relation transitively (i.e. if A is manually declared as compatible with B, and B with C, then A becomes compatible with C).

The compatibility specification could either by an explicit trait impl, an attribute on the type or a macro-like syntax.

Could also have a way to say to do as if fields were not public, as a shorthand for declaring compatibility with an identical struct but with public fields.

bool -> int but no int -> bool seems orthogonal, and seems addressable by making Compatible not be reflexive.

Also note that structs with multiple fields without a layout-fixing repr should probably not be compatible with anything other than maybe structs with the exact same contents (and even this would mean giving up the option to do profile-guided struct layout differently for each type, so probably not a good idea).

[gnzlbg]

So how would this system fit with your Compatible trait proposed above? Could you summarize your whole proposal?

[gnzlbg]

@jmst I’ll try to summarize your proposal as good as I can. You propose to add an auto trait:

auto trait Compatible<T> {
    fn from_compatible(T) -> Self; // needed for the explicit impls
}

that:

• is automatically derived for all permutations of two equally-sized equally-aligned types with whose fields are all public,
• allows coercing T as Self, e.g., using Self::from_compatible(T) (or some other syntax)
• is transitive: allows coercing T as V, if Compatible<T> for U and Compatible<U> for V

For types with private fields, the users would need to specify this trait manually for the relations they care about and due to transitivity the compiler would “fill in the blanks” from the relations the users specifies to the relations the target types support.

Is that it?

[jmst]

The idea would that something like this is added to the core crate:

pub unsafe trait Compatible<T> {}

pub fn safe_transmute<T, U>(x: T) -> U where U: Compatible<T> {unsafe{transmute(x)}}

A first version could just be this, requiring manual implementation of the trait.

A second version could add compiler support for automatically deriving the trait in addition to doing so manually and for performing transitive closure so that if A: Compatible<B> and B: Compatible<C> (whether manually or automatically), then A: Compatible<C>.

A further addition could be to provide a safe way to manually implement the trait for private types in the current crate.

The compiler would generally derive it when doing so would not result in safe_transmute having undefined behavior, changing its result depending on compiler optimization strategy or backwards-compatible crate changes or creating values of U that could not be created otherwise.

The details of the exact rules to accomplish that would be take quite some work and probably some trial-and-error to specify precisely: for example, it needs to account for private field preservation, alignment preservation, the possibility of struct/enum layout optimization changes, non-exhaustive types like char and enum, and so on.

[gnzlbg]

How does coherence interact with your approach?

For example, with the FromBits/IntoBits approach if you have two crates, A and B, exposing two types A::AT and B::BT, a third crate C cannot implement FromBits<A::AT> for B::BT.

Maybe your approach could “fill in this blank”, but I don’t know whether doing so is brittle. For example if crate B adds a new private field to BT, which used to have only public fields. This already is a breaking change, so this breaking might be ok.

[jmst]

Now that I think of it one could have a fallible extension, like:

pub unsafe trait MaybeCompatible<T> {
    fn is_compatible(&self) -> bool {true}
}

pub unsafe trait Compatible<T> : MaybeCompatible<T> {}

pub fn safe_transmute<T, U>(x: T) -> U where U: Compatible<T> {unsafe{transmute(x)}}

pub fn try_safe_transmute<T, U>(x: T) -> Option<U> where U: MaybeCompatible<T> {if(x.is_compatible) {Some(unsafe{transmute(x)})} else {None}}

which would allow to convert int -> char when int happens to be a valid Unicode code point, or &[u8] -> &[u32] when it happens to be 4-byte-aligned.

[jmst]

In the minimalist version, that’s impossible.

With automatic transitive closure crate C could define a type CT and (using “unsafe”) declare that AT is compatible with CT and CT with BT.

But in general this shouldn’t be an issue if the trait is derived by the compiler, since if AT and BT are not automatically compatible, crate C usually cannot safely declare them compatible (not sure if there can be any cases where that would be safe and compatible with changes in crates A/B).