A Stable Modular ABI for Rust

@kornel I do want to see a "safe ABI" subset, which is bigger than C and smaller than fully general Rust.

However, having a shared library for cases like _join would require making the internal _join interface stable, which would increase our stability surface area. And in practice, I don't think most people who want shared library support specifically want it to save disk storage space. They do sometimes want it to save RAM, and a shared library might help a little with that when multiple Rust programs are running simultaneously. But mostly, shared libraries make distribution maintenance easier: you can upgrade a library without rebuilding the world. This wouldn't necessarily solve that problem. And I think it would in practice make distribution of Rust binaries harder for many people, because then they'd need to supply the matching library.

1 Like

The interesting ABI question when talking about kernels, to my mind, is: Suppose a kernel written entirely in Rust + assembly, instead of C + assembly. What subset of Rust types can safely appear in the signatures of system calls, such that user space programs written in Rust can use them with no friction (and ideally without unsafe), but it's still possible to write user space programs in other languages?

It's important to think about both passing and returning values here. For instance, passing &str and &[u8] is straightforward, but being able to return str and [u8] is also desirable and might be a huge pain.

4 Likes

This got me thinking about other possibilities; Erlang has the ability to hotswap code. For servers, kernels, and other very long running processes, this could be a Good Thing™. Can the current ABI support this kind of use? If not, what would be required to make this a standard?

I want to toss i128/u128 into the mix here, not because they are all that hard, but because they illustrate that rust has been extended in the past, and could be again in the future (I don't remember when those values became stable, but I do remember them not being available when I started programming in rust). If this happens again in the future, a stable and immutable ABI will become a burden.

Thinking about all of this really got me thinking about what we're trying to solve. In my mind, we're trying to turn software into a bolt or a screw; a utilitarian object whose interface and guarantees are easy to determine. The ABI is a method of determining the interface and guarantees of the object under examination (library, application, whatever) in a forwards-compatible manner, so that you can decide at runtime if two 'things' are actually able to communicate, and decide if you can replace one instance with another. The closest analogy I can think of is how I can swap one bolt for another bolt in an engine, provided I know both the size and threading of the bolt (the interface), and the yield strength required (the version). If I try to put in the wrong size bolt, it's immediately obvious that the interfaces don't work; likewise, a loader that can't find certain symbols in a library that an application is asking for will fail to load the application.

But finding symbols is the easy part; the hard part is determining if the intent of the interface is unchanged. Semver exists in part to let us know if the intent of an interface has changed; e.g., if fn foo() printed Hello, World in 1.0, but it now prints Frobnicate, the version number tells me if the intent has changed, something the loader won't be able to determine (e.g., was the change a fix for a spelling error, or will the change have a major impact on how the function is used), the versioning information is actually more important because computers can't decide intent, only the programmer can. So we need a machine parseable versioning interface that is guaranteed to remain stable across all versions of the ABI. Ordinary SemVer may not be sufficient (you need a total ordering).

But that is only half the story. If I continue with the machine analogy, think of a large machine like a ship. It is common for a vessel to be undergoing some kind of maintenance while it is in active use. Erlang recognizes this, and has a method for performing maintenance while the machine is in use. But as far as I know, there is no common ABI specification that permits this happen. We do have something similar in that we start and stop applications, and in some cases different applications are able to save their state in a form that a later version of the application is able to pick up where the first stopped, but it would be nice to be able to replace the security module of a running webserver without having to stop and restart the server.

1 Like

Most languages use C ABI to do FFI, so we have to gurantee that our structs and enums fit it. If we want to use dynamic sized types, for example &str or &[u8], we might want to do C's struct Dynamic{int size,void* data} analog, but this is obvious. Things become more complicated if we do trait objects in syscalls: C doesn't have vtbls, but it has function pointer; we could invent some method calling schema, for example, provide void* (*call)(int,void*) C function pointer which call the method from vtbl with tuple from second argument, method is indetified by index, which is first argument of function which we provide.

If we want to do non-FFI trait objects, then ABI can be more complex, but i'm not sure how it could look like.

In case of dynamically sized values, i don't think that we can do that with plain stack, think of segmented stack.

1 Like

I'm actually against this. To quote @matklad:

If this is the route that we're taking (and I really, really hope it is!), then we should ignore C ABI compatibility completely, and develop a new, clean ABI that addresses current needs. Furthermore, if you do need C ABI compatibility, we already came up with a solution; see A Stable Modular ABI for Rust above.

I tent to disagree, mainly because C already have all primitives we actually need, we can do both struct and union in C, that is sufficient to describe anything we need. Slices are repesented like a struct of size and pointer to data. I've wrote the definition of slice above.

Next point is how to deal with representation optimizations, current rust ABI is unspecified intentionaly, to allow this optimizations. We have a bunch of optimizations that change the representation, all of them could be configurable by special macro, effect of which is determined by input params only, note, the compiler might be forced to do pointed optimizations, as well as give gurantees on what they produce, leaving little to no room for their use.

I think that cargo have to support limitation on which ABIs can be used, i.e. a new section like [ABIs] with all used in crate ABIs except for rust,C, maybe more, the ABI item then must specify provider - crate of special type, which is used to produce Layout for types and specify calling convention, this means exposing compiler internals or creating semi-stable API solely for that.

Custom #[repr(...)] items are then not quite special as they are today. Back to ABI section, user could specify which ABIs he use, providers, possibly versions. Then, all information from ABIs section should be bundled in the dylib, giving consumer ability to use lib with ease as well as constraing the ABI version (#[link(...)] parameters?).

Aditionally #[repr(...)] items can be allowed on modules, selecting default ABI for its items. This enables easy maintence of code which has to work with different ABIs. The open questions on this is exact rules for the feature.

Yes, you're right in the sense that given enough effort we can lower all calls to fit within the C ABI, but my concern is that if we make C ABI compatibility a priority, then we can't move beyond it either. That said, I'm not an expert in what Rust's ABI is like. @josh, @hanna-kruppe, @kornel, @CAD97, @comex, @matklad, @Tom-Phinney, I see all of you on posts everywhere, and I know you all are deeply involved in the internals of rust, so I'd like to know your opinions. What can the C ABI not support in rust? What is missing?

IIUC, everything layout-wise can be described in terms of the C ABI. In fact, if anyone develops a new ABI, it's typically first described in terms of how it maps to the C ABI.

When you get to tweaking the calling conventions rather than just layout is where just using the C ABI no longer works. E.g. if you want to reserve registers for e.g. an out parameter for copy elision, that cannot be described solely with the C ABI.

Also, the C standard technically doesn't allow for zero-sized anything.

(To be honest: I'd probably suggest any pluggable/custom/stable ABI work to start by just focusing on the layout part, as that's the main barrier for dynamic code reuse is. Changing functions to extern "C" instead of extern "Rust" (or generating shims) is not all that impactful (currently).)

While it is not Rust, the Swift layout algorithm cannot be represented with either the C or (current) Rust layout algorithms.