The point being made is that relying on layout details already is not language UB[1]. Making more layout guarantees for #[repr(Rust)] makes the hacks more portable between compilers, but it still remains library UB, in that the library author can change the layout and interpretation of the type's private fields without breaking compatibility.
You can claim that private fields could get OSSified into being layout stable, but I don't think making more guarantees on #[repr(Rust)] consistency meaningfully changes the calculus here. #[repr(Rust)] is already consistent in practice[2] (though you aren't supposed to rely on it, just like you're not supposed to rely on being able to transmute your way to private fields) as the PGO field reordering optimization is still purely theoretical and -Zrandomize-layout is an opt-in (partly due to not wanting to break code that wrongly assumes #[repr(Rust)] is consistent without reason).
It's in the strange place between language UB and just library UB. Relying on layout choices made by the compiler is relying on implementation details, so nonportable (including between compiler versions), but it is not a violation of any rules of the Abstract Machine to read bytes out of a type's representation and then use them at whatever type you want. âŠī¸
Disclaimer: niche availability is also factored into field ordering IIRC, so it's not just based on field size/align. Also don't rely on this until if/when a guarantee is provided, because the flag to break you already exists and could theoretically be turned on by default. âŠī¸
UB is not a security feature. In practice you can successfully perform operations that the language declares to be UB. In the end it's just a bunch of bytes in your own process and you have full access to all of it.
It would be a bad practice and ugly not-future-proof code, but that's not a security boundary.
I would very much want -Zrandomize-layout to be the default. It should have been the default from Rust 1.0, specifically to avoid future problems with people depending on it. Regardless, with respect to program correctness I have about as much compassion for depending on #[repr(Rust)] as for depending on double free and reading freed memory being OK because the allocator should just recycle memory pages for the same process. Which is, absolutely zero compassion. The language should help to enforce correctness and portability, not encourage the can of worms like C++ accidental ABI stabilization.
There is also no good reason why Rust couldn't optimize layouts in a type-specific way regardless of field similarity, even if it doesn't do so today.
It never works this way. Either you enforce that some property cannot be depended upon, or all of your observable behaviour is your public contract. Considering Rust's stability guarantees, this is very disturbing. It's not like we're talking about some weird corner case of unsafe semantics, it's something that is explicitly documented as unspecified and arbitrarily varying behaviour.
Rust goes to great length to enforce the privacy boundaries of libraries and modules, to enforce soundness and the possibility of API evolution. A stable #[repr(Rust)] means that subverting all those guarantees is a single google search with a stackoverflow question at the top: "how do I access private fields in Rust". At that point it will be copied everywhere, the most by the people who understand the consequences the least. Why must I even bother with all boilerplate and draconian restrictions of orphan rules, privacy, const correctness and explicit type declarations if I can't rely on it when I need it most?
We're not talking about security features, we're talking about which semantics the language guarantees, what counts as backwards compatibility and which way the APIs must be structured. I change a private field and the consumers of my library break. Is it their problem or mine?
This wastes both memory and execution time. -Z randomize-layout is intended as a debugging/fuzzing tool, and is not good for general use.
Rust never Promises that it will break #[repr(Rust)] layout, only that it's allowed to.
You still don't know what fields might be present in the struct. Private fields can be removed, change, and (as long as at least one private field existed from the start) added. They can also be maintaining arbitrary invariants, and have any meaning whatsoever. This can further change in any update whatsoever. This is true whether it's an unstable #[repr(Rust)] under -Z randomize-layout, or straight up #[repr(C)], as long as the actual layout of the structure isn't specified by the API of the type.
Their's.
From a standard perspective, yes. Individual implementations do specify a reliable ABI (and bend over backwards to preserve it).
That doesn't need to be true. For example, you could pick from one of the possible layout randomizations that doesn't increase the size of the struct from from the minimum. Especially for things like (Box<T>, bool, bool, bool) there's, what, a dozen different layouts that are all minimum size?
And I don't think execution time is affected once it's minimum size, because things like cache locality are already not guaranteed because even without randomization you don't know what the order will be.
Hmm, okay. Personally I don't think one should allow code to mix different stable APIs and hardcode them in a single crate (which would be possible with you notation.) In C++ the implementation of the stable API can be selected with a compiler flag. I believe that's the way to go for Rust (where stable API might not be the repr(Rust) but the repr(Portable) API). Hence there shouldn't #[repr(InteroperableRust_2027)] that can be attached to individual items.
CoerceUnsized is about coercing pointer types. You can use it on nightly to make your own custom pointer types support unsizing coercion like e.g. &T or Box<T> do.
The coercion for S<T: ?Sized>(u8, u8, T) of something like S<[u32; 4]> into S<[u32]> is codified by the Unsize trait instead, which is always implemented/fulfilled automatically and implicitly by the compiler. E.g. this compiles
It's important to still be able to consume multiple ABIs, so you can use libraries compiled on older ABIs. Otherwise having a stable ABI isn't useful in the first place.
You could argue that splitting the declaration into multiple compilation units (crates) is reasonable, so the ABI is specified once per compilation unit rather than once per item is possible, but requiring it seems unnecessary.
In MSVC at least, the calling convention of individual function declarations can be selected with the __cdecl, __fastcall, __stdcall, __vectorcall, and __thiscall. The compiler flags /Gd, /Gr, /Gv, and /Gz select the default calling convention if unannotated. Being able to annotate declarations is extra important in C++ since headers are textual inclusion and thus need to say which calling convention to expect.
The equivalent for Rust would be extern "stable-20XX" to specify a specific calling convention, and compiler/cargo flags to make the default unannotated functions use the specified stable calling convention. (Whether extern "Rust" means use the unstable convention or is equivalent to no annotation is a decision to be made if this path is taken.)
A similar case can be made for a #[repr(Stable20XX)] layout.
I really don't think we should have any kind of edition-tied stable ABIs. Rather, I think we should just have a completely separate extern "safe" ABI, which is not just edition-independent but language-independent, and which supports a subset of Rust functionality. We can expand that over time, and if we ever need to expand it incompatibly we can provide an extern "safe-2".
I actually agree that we don't need an edition-tied calling convention or layout (or the other bits of ABI that I'm sure I'm forgetting).
Calling convention is at least somewhat interesting, since the ideal registers to use may differ between targets, even compatible ones. (See e.g. MSVC x64 __fastcall versus __vectorcall, which enables use of vector registers.)
Layout without automatic niching is basically a solved packing problem (when you have the restriction that size is a multiple of align, anyway) â just stable sort the field order by alignment high to low.
Niching is more complicated, but there's ways of incrementally making niching available in a controlled manner, the first of which would just be "has zero niche" and "uses zero niche" for Option.
The ideal would be a truly language independent description of semantic ABI, but we're a long way from making that a reality.
I was more thinking about gcc's -fabi-version flag (C++ Dialect Options (Using the GNU Compiler Collection (GCC))), when thinking about different ABIs here. __cdecl, __fastcall are like major variant layouts and it makes somewhat sense to use them alongside each other as they all have their pros and cons, whereas I believe the rust-stable ABI, will have a new version mainly because someone found a shortcoming of the current design, just like with new editions.
Of course this will make older ABIs more tricky, but the compiler could still target an older ABI for a specific libary after inspecting it's metadata. The idea would be that a libary will stick to one ABI version until an new major library version is released, which will then upgrade everything to the new version.
Regarding niching, given that we could still rely on precompiled metadata to link to code, even with a stable API, instead of stabilizing the algorithm to lay out struct/enum fields, we could instead use a description of the layout, the compiler came up with when compiling the library. This would be agnostic, or at least backward compatible, over the specific algorithm.
I think there's two slightly different definitions of "stable ABI" going on here.
The weaker one is just "rustc can link to an rlib compiled by a previous rustc." This doesn't actually require a stable ABI at all, though! What this requires is a forwards compatible rlib format that says which internal version of the Rust ABI is in use, such that new rustc versions can read it and know what ABI to expect from symbols in that library. This would be simplified by using a slower cadence than every six weeks, but the details can be entirely private to rustc and the rlib format so long as other tools aren't expected to injest the information.
The stronger one is "the ABI is predictable and can be linked from other languages." This is required to be able to call Rust from e.g. Zig just from type and function header information and not require the external language tooling to consume the rlib format, or to provide source "import headers" for a binary blob library rather than a full rlib.
While the former isn't truly a stable ABI, it gives much of the benefit of it when just using rustc. It's a useful stepping stone towards a stable ABI, though, and perhaps is all that's really necessary if we expect other tooling to consume it through the rlib descriptor.
Even with a versioned ABI with only minor updates, though, it's still useful to mix ABI versions in a single crate (in C++ you could do so via splitting .cxx files per ABI for your .o which you link as your lib) such that a new version of your library only requires knowledge of the new ABI format for new functions.