pre-RFC FromBits/IntoBits

This could be extremely useful for use with C-like enums if the compiler is smart enough to recognize valid ranges, much like it does when checking for exhaustiveness in match statements.

The simplest case is a #[repr(T)] enum where every possible value in T is defined in the enum. In practice this would be useful mainly for repr(u8) because of the current require to specify enum items for each individual value.

A slightly more advanced case is a #[repr(T)] enum where every value between 0 and N are defined. Currently it is only possible to convert between enums with the same ranges either through exhaustive match statements or using transmute. Compiler-generated implementations of FromBits / IntoBits or Compatible would eliminate a huge amount of unsafe boilerplate code and be the first step into turning C-like enums into a general-purpose ranged value type.

For instance, bobbin-bits defines enums for U1 to U32 (covering 1-bit to 32-bit values) and ranges R1 to R32 (covering ranges for 1…N). These are used for modeling bit fields and for type-checked indexing of small fixed arrays, but using them with enums requires manually implementing From and To traits to do conversions, which is tedious and mistake-prone.

It would be far more useful if I could do something like this and have the compiler check that Compatible<U4> is valid:

fn takes_u4<T: Compatible<U4>>(v: T) {
   let value: U4 = v.safe_transmute();
   do_something(value)
}

enum ThingWithSixteenItems {
    Item0 = 0,
    Item1 = 1,
    ...
    Item15 = 15,
}

impl Compatible<U4> for ThingWithSixteenItems {}

or even better, have the Compatible<U4> trait automatically derived for any enum matching the valid range of U4.

@jmst One thing that came out during the preparation of the portable SIMD vectors RFC is that a safe Compatible<T> would probably need to produce the same results in both little endian and big endian platforms.

That is, currently unsafe { mem::transmute } produces this behavior (playground):

let x: [i8; 16] = [
    0, 1, 2, 3, 4, 5, 6, 7,
    8, 9, 10, 11, 12, 13, 14, 15
];
let t: [i16; 8] = unsafe { mem::transmute(x) };
if cfg!(target_endian = "little") {
    let t_el: [i16; 8] = [256, 770, 1284, 1798, 2312, 2826, 3340, 3854];
    assert_eq!(t, t_el);  // OK on LE
} else if cfg!(target_endian = "big") {
    let t_be: [i16; 8] = [1, 515, 1029, 1543, 2057, 2571, 3085, 3599];
    assert_eq!(t, t_be);  // OK on BE
}

It would be nice if a safe_transmute operation could produce the same result in both architectures:

let x: [i8; 16] = [
    0, 1, 2, 3, 4, 5, 6, 7
    8, 9, 10, 11, 12, 13, 14, 15
];
let t: [i16; 8] = safe_transmute(x);
let el: [i16; 8] = [???];
assert_eq!(t, e);  // OK on LE and on BE

Maybe this might not only be nice, but actually a necessary condition to make safe_transmute safe.

AFAIK the only way to achieve this would be to swap bytes inside safe_transmute on either little endian or big endian architectures.

That would be impossible when converting slices, and in any case, endianness dependence is not unsafe.

All of the transmute_copy() calls in simd_funcs.rs of encoding_rs are actually safe and I'd like to write them so.

That's news to me, but happy news.

For ergonomic portable SIMD, it's essential that we have convenient safe syntax for the SIMD type conversions that are zero-cost reinterpretations in the little-endian case but produce different results in the big-endian case.

Since, thanks to WebGL, big endian is not coming back, I don't care much what Rust does for SIMD in the big endian case (compute different results, inject shuffles to match little-endian results or make the conversion unavailable when compiling for a big-endian target), but I really want to have safe wrappers for the little-endian SIMD transmutes.

In addition to just ignoring the problem, a way to handle endianness could be to add a trait variant of Compatible that is guaranteed to give the same results regardless of endianness.

To do so, one would introduce a #[repr(ecC)] for “endian-corrected C repr” (could maybe find a better name for this) which would be like repr© except that fields are laid out in reverse order on big-endian machines.

In addition, endian-corrected tuples, slices and arrays need to be introduced, where on big-endian machines items are laid out backwards and indexing is implemented by subtracting from the end (the syntax maybe being something like &'a #[repr(ecC)] [u8]).

Then cases where the conversion would give different results would be defined only for repr(ecC) when using the endian-corrected version of the trait.

Having to duplicate tuples, slices and arrays is annoying, but those types would probably only be used for small sections of code.

Alternatively one could lay out everything backwards by default, but this would be incompatible with C FFI and performance might be reduced since the CPU instruction sequence for backwards indexing is often less efficient and the backward direction might not trigger hardware prefetching if the CPU is not sophisticated enough.

As a further extension, one could add an “always-reversed repr” and an “reversed only on little-endian repr” and provide a ByteSwap trait that could convert between the versions (either implemented with custom derive or again by the compiler).

An even further extension would be allowing structs and code to be generic over the repr (without using macros), although I guess this would not be worth it.

[not sure about the WebGL argument, since it seems to me that only the GPU needs to be configurable to run in little-endian mode, which is true for all PCIe GPUs since they must work in PCs, and the JS/Wasm CPU code could just byteswap on all memory accesses - it’s not ideal, but CPUs are not normally designed to primarily run WebGL or WebAssembly code]

1 Like

I recently proposed Pre-RFC: Trait for deserializing untrusted input, which is similar to some of the arbitrary bits stuff (like the pod crate) discussed here.

One important difference is that we propose having a custom derive. So instead of requiring your fields to be public or requiring you to use an unsafe impl, you can voluntarily do #[derive(ArbitraryBytesSafe)] (that’s the trait name we propose), and the custom derive will verify whether all of your fields, and their composition, are safe for deserializing from arbitrary byte patterns. It works for private fields too, but because you have to explicitly use the derive, it won’t just automatically apply to your type, invalidating your invariants.

@joshlf is the trait transitive ? That is, is there a way to safely transmute between two unrelated types deriving ArbitraryBytesSafe ?

Well you can copy the bytes of any T into something which is ArbitraryBytesSafe because those bytes are just bytes. You could even transmute a T into an ArbitraryBytesSafe as long as you were willing to mem::forget or drop the original T. So I guess it’s transitive in the trivial sense that anything can be converted into an ArbitraryBytesSafe.

It’s a bit hard for me to visualize it, could you show a code example?

Sure.

// NOTE: need to verify that size_of::<T>() == size_of::<U>().
// How to do that is an open question in the pre-RFC.
fn safe_transmute<T, U: ArbitraryBytesSafe>(t: T) -> U {
    unsafe {
        // First, convert t to its underlying bytes. This effectively
        // mem::forget's it. Could also drop first. Now we just have
        // a meaningless pile of bytes.
        let bytes = mem::transmute::<T, [u8; mem::size_of::<T>()]>(t);
        // Second, convert the pile of bytes to a U. We know this is
        // safe because U: ArbitraryBytesSafe. We could have done both
        // of these steps at once (transmuting T to U), but this is more
        // illustrative of why it's safe to do the conversion.
        mem::transmute::<[u8; mem::size_of::<T>()], U>(bytes)
    }
}

I see. So there are some pairs of types for which safe_transmute is not bidirectional. For example, m32x4 and m16x8 can be safely transmuted into i32x4 but i32x4 cannot be safely transmuted into either m32x4 nor m16x8. I guess that would be handled by making i32x4 derive ArbitraryBytesSafe and leaving m32x4 and m16x8 without an implementation. Since all these types have the same size, m32x4 and m16x8 can be safely transmuted into i32x4, but since m32x4 and m16x8 do not derive ArbitraryBytesSafe, i32x4 cannot be safely transmuted into any of these. So far so good.

However, m32x4 can be safely transmuted into a m16x8 while m16x8 cannot be safely transmuted into m32x4. I wonder how that could be handled by ArbitraryBytesSafe without allowing any safe transmutes of m32x4 or m16x8 to i32x4.

So my thinking was that ArbitraryBytesSafe would be somewhat related to but also somewhat orthogonal to FromBits/IntoBits. In particular, you could imagine the following holding:

  • If something is ArbitraryBytesSafe, then it is FromBits<T> for arbitrary T
  • If something is ArbitraryBytesSafe, then immutable references to it are FromBits<&T> for arbitrary T
  • If two things, T and U, are both ArbitraryBytesSafe, then mutable references to T are FromBits<&mut U> and vice versa
  • If something is FromBits<[u8; size_of::<Self>()]>, then it is ArbitraryBytesSafe.

And I could also imagine other blanket/default (I'm bad with terminology) impls holding for other combinations. But the example you mentioned would be naturally handled by only writing blanket/default impls that are definitely sound. For example, it sounds like m32x4 shouldn't be ArbitraryBytesSafe; if it were, then it would be safe to transmute m16x8 into m32x4.

Note that there are also concerns around alignment that are trickier when you're trying to transmute references than when you're only transmuting values. My proposal focuses on references because the goal is to enable safe zero-copy deserializing, but I don't think you have to deal with those concerns here because you're doing everything by value. There are also concerns, when consuming by value, around whether you drop or forget the input.

Ah! I thought that ArbitraryBytesSafe should solve the whole problem!

I don't know how I feel about having 2 mechanisms to achieve almost the same thing, but not quite.

In particular, @jmst proposed a Compatible trait (read from here downwards: pre-RFC FromBits/IntoBits - #23 by gnzlbg) that appears to solve all problems better than FromBits/IntoBits and ArbitraryBytesSafe.

So I wonder why can't ArbitraryBytesSafe just derive Compatible<[u8; mem::size_of::<T>()]> or similar instead?

I don't think it can because it's strictly less powerful than FromBits/IntoBits. In particular, the latter can express the idea that "any valid instance of this type has a bit pattern which is also valid for this type" even if the latter type is not ArbitraryBytesSafe. For my particular use cases, I don't need anything that powerful, but it sounds like you do.

Have you thought about having a custom derive to cover the gaps? If I've written a type that I'd like to be Convert<T>, but it has private fields, then I'm forced to unsafe impl Convert<T> for MyType {}. The thing is, it's not actually memory unsafety that's the issue here, but invariants on the values of my fields, so having to invoke unsafe here feels wrong and dangerous since it might let you not only break your contract, but actually introduce memory unsafety. A custom derive would give us something like "dear custom derive, I know that I'm OK with converting from T for the purposes of my invariants, so could you verify for me that it would be memory safe?"

Also, it looks like there hasn't been any discussion about converting references. I'm interested in zero-copy deserializing, so being able to convert references would be huge. In particular, I can imagine the following:

  • pub fn safe_transmute_ref<T, U>(x: &T) -> &U where U: Compatible<T> { ... }
  • pub fn safe_transmute_mut<T, U>(x: &mut T) -> &mut U where U: Compatible<T>, T: Compatible<U> { ... }

As discussed in Pre-RFC: Trait for deserializing untrusted input, there are still issues with verifying alignment, but it'd be a very powerful feature to have in general.

Not really, but why can't that be done with a custom derive? If it can, then it can just be done on a crate, without having to write any kind of RFC for it. The proposal ensures transitivity, so this derive doesn't really need any kind of compiler support, it is purely syntactic.

Also, it looks like there hasn’t been any discussion about converting references.

One might be able to solve this with some blanket impls for references and raw pointers:

impl<T, U> Compatible<&T> for &U where U: Compatible<T> {}
impl<T, U> Compatible<&T> for &mut U where U: Compatible<T> {}
impl<T, U> Compatible<&mut T> for &mut U where U: Compatible<T> {}
// ... and for *T ...

there are still issues with verifying alignment

Which issues? For practical purposes transmute is just a memcpy, so if the source type is properly aligned and the destination type is properly aligned, which they must be, then there aren't any alignment issues AFAICT. The same applies to "endianness": since transmute is just a memcpy, the bytes will just be copied from the source to the destination. If you don't take endianness into account when reinterpreting the bytes on the destination, you will get different results on big endian and little endian systems, but that is just how memcpy works, so that's "working as intended".

Yeah, and I know that there was a proposal to have v0 of this just require manual impls. If you can have a custom derive, then v0 could use the custom derive (so that folks don't have to unsafe impl manually, which introduces a risk of unsafety), and have v1 be the move from custom derive to compiler-supported auto trait.

I'm referring to alignment of references. impl<T, U> Compatible<&T> for &U where U: Compatible<T> {} isn't safe in general because U may have higher alignment requirements than T, so it's not actually guaranteed that any valid &T is also a valid &U. Pre-RFC: Trait for deserializing untrusted input discusses some options here, but if you don't have compiler support, then your options are either somewhat unergonomic (use a macro that uses static_assert! under the hood) or unsafe (since the caller needs to verify alignment manually).

This makes sense.

At this point doing this in the compiler looks like the most appealing solution to me, and is something that Compatible<T> could do for references, for example: if T is compatible with U and some alignment conditions hold, then &T is compatible with &U.

Maybe one day we will be able to do where mem::align_of::<T>() >= mem::align_of::<U>() in the language, but this won't be the case in the near future since we need more than just const generics for this.

Yeah, my stopgap idea was to use macros to do something like:

pub unsafe fn transmute_ref<T, U>(x: &T) -> &U where U: Compatible<T> { ... }

macro_rules! transmute_ref {
    ($x:expr, $T:ty, $U:ty) => (
        static_assert!(::std::mem::align_of::<$T>() >= ::std::mem::align_of::<$U>());
        unsafe { transmute_ref::<$T, $U>($x) }
    );
}

Another question: What about DSTs? In particular:

  • If T is a DST and U: Sized, then what does U: Compatible<T> mean?
  • If T and U are both DSTs, then what does U: Compatible<T> mean?

Some ideas:

  • If T is a DST and U: Sized, then
    • size_of_val(t) == size_of::<U>() implies that t's bits correspond to a valid U.
    • size_of_val(t) > size_of::<U>() might imply that t's bits correspond to a valid U? Is this always safe?
  • If T and U are both DSTs, then
    • If you have an existing u: U, then any t: T of size size_of_val(u) is a valid U (in other words, the existence of t is proof that size_of_val(t) is a valid size for T)
    • Maybe it’s the case that any t: T whose size corresponds to a valid size for U is a valid U? What does "valid size for U" even mean? Can we query it at compile or run time?

There’s a caveat here, which is that since we’re defining what Compatible means, the questions of “is this safe?” are somewhat up to how we define Compatible. I have a vague intuition for how this would work with [T] and composite types ending in [T]. I have essentially no idea how this would work for trait objects. I’d love to hear some thoughts on all of this.

Is it possible to mem::transmute a DST into a Sized type and vice-versa ? Is it possible to mem::transmute two different DSTs to each other?