I’ve ran into an issue with FFI that I believe requires clarification, or maybe some additions to the language. Many C APIs have functions with output value types defined as C enumerations. In Rust FFI, these are “naturally” represented by
#[repr(C)] enums with C-style integer members. This is all good while we receive values that fit our FFI definition. But if the foreign library grows some new enumeration members and is used without changing the Rust-side definition, attempting to match an unknown value of a Rust enum brings undefined behavior.
In the C world it’s normal to update dynamic libraries and expect binaries built against older versions of those libraries to work as long as binary compatibility is preserved, and adding new enum members is not normally considered an API or ABI break (the standard says enums can be backed by implementation-defined integer types, but in practice they are int-sized unless overridden by compiler-specific means). The soname of GLib has been kept the same for a dozen years, and recent releases of the library probably still work with GNOME 2.0 binaries. I think binaries built from Rust sources should be kept to the same expectations.
Some C APIs are pushing it further by defining bit flags with enums and then using the enum types directly in function signatures and structures. It’s apparently legal in C, where enums are full integral types with some value constants sprinkled over them as a vague hint on the expected domain. This brings issues on the input side as well: input parameters and struct members should be declared in FFI with their corresponding enum types for repr compatibility, but the actual values are composed from bit flags and so need to be transmuted.
So how can we deal with that heathen land of C? Can we rely on a tacit assumption that the actual underlying integer value can be received intact by transmuting the enum value into an appropriately repr-sized int, and vice versa? In the long term, would it be worthwhile to add more ergonomic means for passing “vaguely enum” values through FFI?
There is a long-standing Rust issue on the lack of support for C unions. I think that is a very similar problem to the one discussed here, and as such both can be solved by adding a single new Rust type:
unsafe enum as described in the discussion on rust-lang/rust#5492. C-style unsafe enums would be used to unsafely match C enum values, while the struct-membered variety would stand in for unions. What do you think?