I'm working on an embedded project which is introducing more and more Rust. This is the kind of environment where debugging often means poring over hex dumps to work out why the system got into a certain state. As one does this, one learns various patterns, like the layout of a common structure, or when reading a disassembly which registers are args, and which are scratch in the ABI.
(Risc V target for concrete context.)
One concern by C-familiar engineers is that if Rust doesn't have a stable ABI, then how can they become accustomed to these patterns? Does it mean that for every compiler release all the structures are going to have a dramatically different layout, or that it will use arbitrary different register conventions?
My gut feeling is that while there are no stability guarantees, these kinds of things don't change in arbitrary ways from version to version, only if there's a good reason. Specifically:
Structs layouts are fairly predicable, and we can always add repr(C) if we need to nail things down
"Simple" function params will be pretty much as C passes things (using the same ABI conventions?); things will get a bit more complex with complex types, but still manageable
It seems to me that niche optimizations change more than most, so enum representations might be a bit hard to eyeball, at least not without a fairly detailed understanding of their actual types
For parameter passing: today it's lowered to LLVM using the default calling convention, so it's almost always going to look like passing something in C, though the details of whether, say, by-value [u16; 2] is passed as u32, as ptr, or as two u16s might not match exactly what you're used to in C, and may well change from time to time. But for x64 linux, for example, you'll still get the first two things in rdi+rsi most of the time, so it'll look pretty familiar. Realistically, rustc will never have some random ABI because it's always emitting LLVM, so it'll be something that LLVM generates.
For niche optimizations, remember that if you can get &mut to a field, that field has to exist exactly as normal in-memory (because you could pass that &mut to something that doesn't know it's inside an enum and it still needs to be able to read and write it). So if you have a Foo::Bar(0xDEADBEEF_u32) in memory, you're still going to see 0xDEADBEEF in the hexdump for sure. It's just that where inside the object and how the discriminant is represented might be subtle and change. But here too you can repr(C) to get a predictable layout if you need it, just like with structs.
The layout algorithm isn't trivial, so predictable isn't the word I'd pick. It doesn't change frequently, maybe once or twice per year.
And then there's -Zrandomize-layout which might get enabled in debug builds at some point in the future to weed out accidental dependencies on struct ordering.
Now I'm curious about the alternative backends such as cg_cranelift and cg_gcc. Do they use different or the same ABI in practice? Without dynamic linking it doesn't matter of course in practice.
Without -Zbuild-std they appear to link against the llvm-compiled distributed standard library. I'm not sure if they independently implement the exact same ABI, or if the rmeta encodes the ABI of all the exported functions for them to match against.
The memory layout is implemented in rustc itself with the layout_of query. Using a different layout in your backend is not an option as const eval needs to produce abi compatible values. The calling convention is implemented in the FnAbi type. Avoiding this one in a custom codegen backend is possible, but reusing the main one is easier as it also avoids having to implement the C calling convention yourself and as such both cg_clif and cg_gcc use it just like the default cg_llvm.
The debuginfo emitted with -Cdebuginfo=2 should contain the precise locations of all function arguments as well as the full memory layout of all types.
We would use a debugger where possible, but sometimes the debugging process is literally interpreting a hex dump in some kind of crash/panic message, using some other low-level memory inspection tool, or even reconstructed from transactions captured off a bus.
Yeah we'd definitely want to disable randomized layouts in this case. Though it's quite likely that we'd decide to repr(c) everything significant anyway.
In principle you could change codegen backend options on a target-by-target basis, so if that's supported you'd need to make sure all the codegens use the same register conventions. But I could easily imagine we'd want to just leave that unsupported.
For cg_clif I fully intend it to be ABI compatible with cg_llvm to allow compiling dependencies with cg_llvm and full optimizations and then use cg_clif for the user code. Compiling dependencies with optimizations is pretty important for games.