A Stable Modular ABI for Rust

compiler
68

I thought back about what I wrote, and I think it can be made even easier.

Inside a crate, the compiler should be able to use whatever ABI it considers to be the best. There is only one place where the ABI is important: the interface between two libraries/executable. The compiler could choose to use the Rust ABI or the one of the interface, or a mix, whatever is best for the code that isn't part of the interface.

There are two things that cross the interface between two crates: functions and structs.

If the consumed library is in source code form, then there is no need to specify the ABI. The compiler should be able to use whatever ABI it consider best, just like if both the consumer and the consumed were a single crate.

If the consumed library is a binary, the consumer wants to match it's ABI of the consumed library. I don't see any motivation to have different entry points that would have different ABI¹. So I think the consumer should be able to select the ABI of the consumed library with an additional flag in Cargo.toml in the dependency section. I think this should be required for consuming pre-compiled libraries (but maybe defaulting to the C abi is better). All struct that would cross the ABI (aka all struct that are used as an argument of a function of the external library in binary form) should be explicitly marked with #repr(...) and the ABI should match the one specified in the Cargo.toml (otherwise it should be a compilation error).

¹ Note: if for some reason, a library libA must have some of its public function consumed using the ABI B, and the rest using the ABI C, it is always possible to split it in 3: libA_core would contains all the functionalities, but shouldn't be consumed directly, libB would be a thin wrapper over the part of libA_core that would be consumed with the calling convention B (and depends on libA_core), while libC would be a thin wrapper over the part of libA_core that would be consumed with the calling convention C (and likewise would depends on libA_core).

Finally, when creating a binary (either a static or dynamic library) from the source code of a library, we should be able to specify the ABI of all entry points (the public functions and the public struct) as a cargo/rustc flag. If a struct has a #repr(...) it should be a compiler error. This make it possible to create different binaries, each with their entry points using a different calling convention, without having to change anything the source code.

To sum-up:

  • the only places where the ABI is important is the interface
  • the ABI of the function shouldn't be specified in the source code. Only the ABI of struct consumed by other libraries pre-compiled in a static or dynamic library
  • the compiler should be allowed to use whatever ABI it wants internally (either match the ABI of the interface, using the Rust ABI, or a mix…)
  • the ABI of a pre-compiled library (static and dynamic library) should be set with a rustc/cargo flag when compiling the library, and would be common for all entry points of the library (you can always split a library to have different ABI for different entry points). This flags would control both the ABI of the public function and the public structs.
  • the ABI when consuming a library in source form, should be left to the compiler (it is not considered an interface)
  • the ABI when consuming a pre-compiled library (static or dynamic library) should be set in the Cargo.toml, as an additional option in the dependency section, and all structs passed as arguments to those function should have a matching #[repr(...)] (otherwise it's a compilation error).
  • the only place where it is allowed (and required) to add #[repr(...)] is on struct that are used as arguments of pre-compiled libraries.
6 Likes