A Stable Modular ABI for Rust

compiler
1

This topic has grown into quite the discussion, which is why there is now a repo to organize the discussion thus far and create an easy starting point for those who want to contribute. PRs welcome :slight_smile:

Proposing a stable modularizable ABI interface for Rust

Based on the points from the discussion here.

Introduction

Rust is a powerful systems programming with strong memory guarantees. Rust allows for concise expression at a high-level, while still producing fast low-level code. However, Rust does not guarantee the calling conventions and layout of structures in memory, which makes it difficult to write external applications that interface with Rust; Rust lacks a standardized ABI. Standardizing Rust's ABI has been brought up before, but has usually gone nowhere due to the difficulty of the task. In this post, we outline the benefits and stumbling-blocks of a stable ABI, as well as suggest a semi-novel technique as to how such an ABI could be implemented.

Benefits

There are many benefits a standardized ABI would bring to Rust. A stable ABI enables dynamic linking between Rust crates, which would allow for Rust programs to support dynamically loaded plugins (a feature common in C/C++). Dynamic linking would result in shorter compile-times and lower disk-space use for projects, as multiple projects could link to the same dylib. For example, imagine having multiple CLIs all link to the same core library crate.

Although this use case is already rather well covered by abi-stable-crates, there are still many more benefits beyond linking crates dynamically. A stable ABI would allow Rust libraries to be loaded by other languages (such as Swift), and would allow Rust to interop with libraries defined in other programming languages. Non-Rust crates could be integrated with Rust toolchains; providing an ABI would also allow outside code to rely on Rust for performance-intensive tasks. Cross-language compatibility would increase the diversity of Rust's package ecosystem.

Quote: Imho one of the biggest mistakes C++ ever made was not stabilizing its abi; swift just stabilized theirs and is already reaping the benefits, swift system libraries, the swift runtime, swift UI libraries, all dynamically linked and backwards abi compatible.

Stabilizing the Rust's ABI would allow for cross language interop and dynamic linking. " extern "C" as the lowest common denominator is too low for Rust" (Quote).

Recently, the Fuschia OS Team at Google decided to ban Rust's for use in Fuschia microkernel, citing C as an alternative because of its stable ABI. Not providing a stable ABI ultimately hurts Rust when getting down to the metal. Given similar languages like C and Swift have a stable ABI, I see no reason why a stable ABI would not be implementable for Rust. As discussed here, some ABIs/FFIs have already been written using proc macro and the like.

Potential Issues

However, a stable ABI is not all peaches and roses. Having to standardize the memory layout of data can limit the number of optimizations the compiler can perform.There has been a lot of work on optimizing laying out fields in structs in reliable and ABI-compliant ways. There are a large class of optimizations that can be done in compliance with an ABI; since an ABI solidifies the layout of data, more reliable bit-twiddling and the like can occur.

While discussing the matter, a point was brought up that the ABI could be modularized. A modularized ABI would be optional while compiling. This modular ABI could be published as a versioned crate. If the ABI ever needs a backward-compatibility breaking change, the change could be made within Semver. Alternatively, a new ABI-compliant compiler backend could be developed, or the current compiler backend could be extended to support an ABI feature flag that would toggle ABI compliant builds.

However:

Quote: Depending on the implementation, if we want to make ABI plugins, to avoid stabilizing the compiler's built-in ABI, we might run into another problem because we have to stabilize the plugin interface, which could be another can of worms.

Standardizing the ABI would take a lot of work. A poorly designed ABI is worse than not having an ABI at all. And as we all know, the right solution is often the hardest one.

Another downside is that allowing ABI crates might not stabilize Rust's ABI, there'd just be ABI fragmentation. Although this is a genuine concern, a 'master' ABI crate with Rust's 'official' ABI could be developed. This would standardize Rust's ABI, while still allowing other crates with other ABI's to be written for interop with other ABIs, like Swift's. Additionally, because modular ABIs are opt-in, ABIs would be used only where explicitly necessary.

Implementation Proposal

So, what might this modularized ABI look like? Roughly speaking, an ABI would be defined by a series of macros in a crate which specify the layout and calling conventions of data structures according to that ABI. During compilation, while determining the layout of the data, the layout information provided by the ABI macros would be used. The end-goal would be for something like #[repr(RustABI)] or $ cargo build --release --abi rust-abi to be plausible.

Let's get into more detail. Right now, the closest analogue to a stable Rust ABI is the abi_stable crate. abi_stable uses #[repr(C)] to create ABI-compatible data structures. This is a step in the right direction, but every ABI-complaint type has to pass through abi_stable's mechanisms. These data-structures are also more expressively limited. For example, every abi_stable ABI struct has to contain ABI compatible fields - and some Rust types, like Result, aren't compatible at all.

A modular ABI could solve this issue. An "ABI" Rust crate is a proc-macro-like crate that determines exactly how each byte of a data-structure should be laid out in memory. To do this, the "ABI" crate should provide a macro each standard Rust data-structure (struct, enum, tuple, etc.) When a data-structure is marked as ABI-compliant (either through a #[repr(ABI)] proc macro or compiler flag), the compiler calls out the "ABI" crate which recursively lays out said data-structure in an ABI-compliant manner.

There are a few issues that still need to be addressed. How do pointers and memory management work across FFI boundaries? We propose that when ABI-compliant data is transferred across an FFI boundary, it should be either copied or moved. Once some data has moved across an FFI boundary, the only way to reference that data is to use the copy, or have the program the data was transferred to transfer it back. This copy/move borrowing technique is merely a suggestion, as there is probably a better way to do it (semi-related post).

To determine the layout of data for Rust's own ABI, a minimum API would have to be found. Rust currently provides many niche optimizations and field ordering techniques to increase performance - a stable ABI might interrupt or prevent some of this. However, as mentioned in the Potential Issues section, there are ways to work around this. Different calling conventions could be supported through a proxy assemble stub or the like, but the devil's always in the details.

Multiple ABI crates would be able to be defined—for example, there could be an abi_swift crate for interop with Swift's ABI—Rust itself could have it's own ABI in an ABI crate titled abi_rust or the like.

Quote: The potential to have different ABIs (e.g., abi_rust, abi_swift) that are used concurrently in the same compilation would permit Rust programs to act as the "glue" between external components that use incompatible ABIs.

Closing Thoughts

We hope that this outline of a very rough specification will provide a launching point for the ultimate development of a stable modularized ABI interface for Rust. Such an ABI would expand the number of applications that Rust could be used for. A stable ABI would standardize dynamic linking between Rust crates, minimize the amount of space-time used during compilation, allow for cross-compatibility between Rust and other programming languages, and increase the plausibility of Rust as a kernel-level language. Something like this takes hard work and good communication, so if you have any questions, comments, concerns, feedback, or other ideas, please don't hesitate to share.

45 Likes
2

I would want to keep the ABI zero cost so that there is no runtime performance disadvantage to going over the stable ABI compared to going over a C ABI for instance. It would be best if we wouldn't have to copy anything.

How does this sort of stuff work with the C ABI and repr(C)? There isn't memory copying over C interfaces is there? How much can we keep the way that things currently work with things like repr(C) while ( probably over simplified ) just allowing you to create your own reprs and optionally apply them to the entire crate?

2 Likes
3

The ABI representations would most likely have to be generated during compile time. An ABI crate would describe the layout of a data structure, then that datastructure would be used during runtime. Of course, only #[repr(ABI)] data structures would be transferable across that ABI's FFI.

AFAICT, #[repr(C)] generates a C compliant layout for the data structure at compile time. The proposal for modular ABIs would be to abstract this behaviour out of the compiler to allow new ABIs to be developed. These ABI's could be implementations of existing ABIs, like Swift or C, or they could be completely new ABIs.

5 Likes
4

TL;DR is that a shared-nothing ABI is easier to spec initially, and still super useful. (In fact, WASM interface types is effectively a WASM ABI in a very similar manner, in that it's a translation to a known middle representation (that can be optimized by the VM).)

The ABI proposal and plan should support forward compatibility to "at rest, internal" specification of layout to allow extending to a shared representation in the future, but a shared-nothing starting place is a decent one.

(Disclaimer: haven't actually read the OP in depth yet.)

6 Likes
5

Yeah, it's quite long as I wanted to try to cover as much surface area as possible. Here's a summary:

  • A stable ABI would be really nice for Rust
  • But it would also be difficult to do
  • We propose a modular ABI, where compiler-time macros can determine the layout of datastructures so that they are ABI-compliant.
  • We discuss caveats
  • We ask for feedback.

I'll take a look at everything tomorrow,

4 Likes
6

I applaud any attempt to address some of the use cases commonly associated with "ABI stability" without affecting code that doesn't actually need "ABI stability". Sadly I am too short in time to delve into implementation details, but I can (and feel the need to) give a feedback that I often give then the topic of "stable ABI" comes up: this is a overloaded term, and it's important to be precise about which of the several meanings we have in mind at any given point, because they differ greatly in what they enable and what they require from the compiler and from the user.

More concretely: there are broadly two clusters of meanings for "stable ABI", the first referring to how language concepts are mapped machine code (e.g. data structure layout, calling conventions) and the latter to the ability to change (upgrade) a software component without rebuilding all the software that interacts with this component. The second is ultimately the responsibility of programmers (e.g. deleting a function from a library breaks both source compatibility and ABI compatibility), but since it's required for many of the benefits ascribed to "stable ABI" and requires compiler support too, anyone pondering this subject should either explicitly acknowledge that aspect (and everything that it enables) as out of scope, or think about what it entails -- which goes far beyond "freezing" data structure layouts, calling conventions, etc.

In the C world, there is a relatively simple relation between these two aspects: everything you put into ("public") headers is part of the ABI, and if you want the ability to e.g. change the layout of a type without breaking ABI compatibility, then you need to make that struct an opaque type in the headers and e.g. expose getter/setter functions for fields whose existence you want to guarantee.

In a richer language, however, it's more difficult. The way Rust compiles generics, for example, necessarily inlines lots of "internal" library code (full of hard-coded field offsets, type sizes, etc.) into consumers of the library. Even in non-generic code, innocent changes such as adding an extra field to a struct often invalidate all code that deals with that struct in any way. In contrast, Swift has similar features but achieves dynamic linking anyway by adopting different compilation strategies and a host of neat tricks that Rust lacks. There is a huge design space here: what subset of Rust can we make usable for libraries with stable ABIs? what sorts of changes can be made non-ABI-breaking? What changes can we make to the compiler to allow a bigger subset of Rust and more kinds of changes? How can we communicate these rules to programmers who author libraries with stable ABIs, and how can we make it easy for them to adhere to the rules? What escape hatches should we have for trading off resilience and performance (like Swift's @frozen structs)? etc. etc.

These kinds of questions are crucial if you want to reap the benefits Swift and C are reaping from their "stable ABI", but are only apparent if you go beyond viewing ABI stability as just about the choices the compiler makes while compiling code. You can focus exclusively on that aspect if you want, but in that case you still need to think very carefully to avoid over-stating what use cases your proposal will actually enable.

47 Likes
7

Addendum: when I wrote this sentence I also fell into the trap of focusing just on the "ABI" in the narrow sense of choices made while mapping Rust to machine code. Just as important is the fact that the library's code is copied into the consumer at all. So if a bug is fixed in the library, even if you can recompile that library and re-link everything (because you went through great effort to make type sizes and calling conventions and so on stable), the bug fix still won't reach the applications that use the library. Or will reach it inconsistently depending on where a fresh copy of the generic code was instantiated and where an existing copy of the code was reused (see: -Zshare-generics flag).

21 Likes
8

Thanks for taking the time to provide feedback. I'll try to provide clarifications and specify in more exact terms what meanings were meant.

I'll call the first definition of ABI the 'ABI specification', and the latter definition an 'ABI interface'.

An ABI specification isn't really programming-language determined, rather it's determined by largely the OS (and toolchain). An ABI specification provides guarantees as to how types are laid out in memory, how functions are called, and how things are named. The definition of 'ABI' used in this proposal largely falls into the ABI specification category, as the proposal discusses modularizing the code that defines Rust's ABI specification.

An ABI interface defines how a user of a ABI specification can upgrade existing code without invalidating the ABI. The challenges of cementing an ABI interface difficult: because ABI specifications are modularized, more than one ABI interface can exist, meaning that each ABI specification has to describe its own interface. This is a complex topic and I'll need some more time to think of an elegant solution, but for now we'll assume that this is out-of-scope and two ABI interface data types are only compatible if they have the same byte-level layout.

AFAICT, Swift's ABI largely works by generating 'header files' (swiftmodule + interface summary) for the library at compile time. When a library is dynamically linked, the 'header files' are used to communicate with the library, regardless of version. However, because Swift is dynamic, it has much more free reign as to what stability guarantees its ABI is able to provide. For example, adding a field to a struct in C will invalidate the struct's ABI, but doing the same in Swift won't, necessarily.

I don't know the specifics of the Rust linker, so any clarification on how Rust currently works in this regard would be appreciated.

I'll try to answer these questions, but they're fairly tough, so forgive me if I provide a nonsensical answer.

What subset of Rust can we make usable for libraries with stable ABIs? This proposal proposed a modular system-ABI where each basic Rust structure has a macro associated with it that generates layouts for it. This was discussed in the original thread, but the result was inconclusive. I think that the core rust data types, namely structs, tuples, enums, and basic pointers could all be supported.

What sorts of changes can be made that are non-ABI breaking? Any changes to types that retain the same layout are ABI-specification compliant. Changes to API interfaces are more difficult to ascertain, so for now we'll say that an API interface change is non-breaking if an ABI-specification change is non-breaking.

Note: I actually spent a long time writing out a huge answer to this question with code examples, discussion of layouts, how traits can be used to provide more concrete interfaces, how the compiler might integrate with the ABI to detect these changes. After writing it all out, I figured that it was quite long-winded and I wasn't fully sure if it was a fully-baked idea, so I removed it in favor of this shorter answer. I'll try to rewrite and repost the original answer as it's quite interesting, but it still needs some work.

What changes can we make to the compiler to allow a bigger subset of Rust and more kinds of changes? To support modularizable ABIs, the compiler would have to be extended to use information provided by proc macros (layout, etc.) to determine the ABI specification. At compile time, macros would be passed data about structs and produce layout information for the compiler to use. For example, consider:

#[repr(abi_rust)]
pub enum Numbers {
    Float(f64),
    Int(usize),
}

The enum macro specified by abi_rust would use information about the Numbers enum to produce a layout. For example, following the conventions in this post, the generated Layout might be:

Enum {
    size: 16,
    discriminant: (0, Discriminant { size: 1 }),
    variants: vec![
        (8, Variant { size: 8, layout: Some(Box::new(USize)) }),
        (8, Variant { size: 8, layout: Some(Box::new(F64))   }),
    ]
}

Of course, this representation is far from perfect (Which variant do we mean?), but it shows the general idea. The compiler can use the provided layout in conjunction with the information about the enum to correctly determine layouts. The compiler would also have to be modified to support dynamic linking (and should be able to report errors when ABI interfaces don't align).

When I started the proposal, I was worried about trying to make a point overstating it. AFAICT, Once an modular ABI specification system as proposed this is put in place, just about everything I mentioned in the proposal is should be possible. The main caveat is that although a modular ABI specification might enable certain things, like ABI interop with Swift, implementing a modular ABI will not immediately guarantee such a thing will immediately work out of the box - much more work will still have to be done to bring Swift ABI support to Rust.

Thank you again for the feedback. It's taken a lot of challenging thinking, writing, and discussion to put this proposal coherently because the underlying complexity of the problem of developing a modular ABI - I'd be lying if I said I understood everything completely. Any forward progress we make on this problem is good; although the problem at hand is difficult to solve elegantly, I hope that through the insights and feedback from the many smart people in the Rust community an ultimatum to this problem may be found, whether it be a solution or a concrete consensus as to why such a thing is impossible.

4 Likes
9

Really good points @hanna-kruppe.

Whew, that's a difficult one for sure. We may have to, at least initially, limit the ABI stable constructs that you are able to use such as blacklisting generics or something. Obviously some of Rusts most awesome features such as Result<T> and Option<T> come from generics, so to say that you couldn't use generics anywhere in your code obviously wouldn't work. So then there are restrictions on the public interfaces only maybe?

As a library designer, most naturally, I would think that my ABI stays compatible if I don't make any public API changes. Essentially if the Rustdoc doesn't look any different, the ABI stayed the same, but that is just what is intuitive. Maybe when compiling in ABI compatible mode is makes sure that all public interfaces only use types that are ABI compatible ( so if that meant no-generics, your public functions couldn't return Result ).

Also, there could be a tool that compares two commits of your application and makes sure that the resulting ABIs are the same.

Absolutely. For this proposal I would think that we should definitely keep our minds open and think about the problem of how we serve ABI stability to the programmer, not just to the binary representation of the ABI, which, like you stated, is a much bigger problem.

Still, if we could achieve just a step in the right direction that would also be useful, even if it doesn't bring in the whole solution.

I would rather that we don't design something that satisfies only the ABI specification ( as @isaac defined the term ) without providing a lot of value to the programmer who really has to deal with the ABI interface. Still, if we have to find a way to work out a more stable ABI specification before we can tackle a stable ABI interface, then it is still useful work.

3 Likes
10

One super rough idea I am thinking about recently is that maybe we don‘t need „Rust“ ABI, but „System ABI“? Today, the interop language of software components is C ABI, and it is wholly inadequate: it doesn’t even have slices, strings are slow and error-prone, etc.

It seems like developing language-independent ABI which significantly improves over C, without being Rust specific, is possible. Slices, tagged unions, utf8 strings, borrowed closures are features which immediately come to mind and have obvious-ish implementations.

Distinction between borrowed and owned (callee must invoke destructor) data and simple lifetimes are somewhat more Rust/C++ specific, but don’t seem too controversial either.

Support for dynamic dispatch (where fat vs thin pointer is a big tradeoff) and templates seems pretty hard, but also desirable and plausible.

It seems like if we had this „C ABI, but you also can have nice things“ , then interoperability between various software components would be easier and far less error prone.

51 Likes
11

That actually sounds like a really cool idea. I'm thinking we would still try to work out this proposal for a modular ABI, then we would try to implement this new ABI, that was designed not only for Rust, as an ABI crate.

The spec for that ABI would be a separate proposal, but maybe this proposal would seek to satisfy the requirements for being able to provide that "System ABI" for Rust.

2 Likes
12

Generics are not a problem as long as all types are known in the interface exposed through the ABI. So it wouldn't be a problem to return Result<i32, String> or even Result<(), Box<dyn Error>> (assuming trait objects have defined ABI).

They're only an issue for functions like pub fn generic<T, E>() -> Result<T, E> that can't be monomorphized on the library side. For these functions the options are:

  • Just forbid them. Library interfaces would have to use dyn Trait or concrete types instead. IMHO this is quite sensible limitation, especially for an MVP.

  • Require defining ahead of time which parameters can be used, and compile monomorphic versions just for these types (similar to template instantiations in C++)

  • Do what Swift does and compile a universal version of the generic function that uses run time type information to support arbitrary types. It's a very clever approach from ABI perspective, but it's equivalent of changing everything into dyn Trait, so it may be a poor fit for Rust.

25 Likes
13

That's nice! :slight_smile:

I think that is rather reasonable.

Maybe this could be optional behavior?

4 Likes
14

That's not actually too far fetched: Optionally: declare the ffi-safe traits with #[sabi_trait] , used as trait objects in the public interface.

That seems like a reasonable limitation.

IMO, I feel like that might be a comparatively significant extension to the compiler.

Perhaps for only non-monomorphizable functions, explicitly annotated.

1 Like
15

The Swift ABI isn't specific to Swift, and is an impressive feat of engineering that could fill this slot.

Witness table indirection obviously adds some overhead over static linking or "just repr(Rust)" linking, but it successfully maintains the ability to evolve private implementation details, and it's a necessary cost to do so. (Also, it supports freezing the ABI to remove (most) witness table overhead.)

The one thing I don't recall off the top of my head is whether it requires any shared heap object to be managed by Swift atomic reference counting. I think it just uses opaque "clone pointers", though.

"Add repr(Swift) to Rust" is an interesting research topic independent of "Add user-defined repr to Rust".

But, yes, an initial MVP would handle concretized generics fine as "just another type" and punt on unconcrete generics, as you can "just" expose a dyn API yourself.

13 Likes
16

Since the Swift ABI would be included in any reasonable set of modular Rust ABIs, it probably makes sense to subset the modular ABI task to first just develop the Swift ABI, noting during the development process those places where additional effort would be needed to generalize to additional ABIs beyond the obvious three: Rust's native unstable/unspecified ABI, the stable# C ABI, and the new stable# Swift ABI.

# Of course those externally-specified ABIs are subject to change through their own language specification maintenance processes.

7 Likes
17

That saves us the development of the proposal, which would be large in-and-of-itself, for a „System ABI“. I think that makes a lot of sense.

So the goal for this proposal would be to create a modular ABI system that could support a Swift ABI crate for Rust.

:+1:

6 Likes
18

The problem with that is that the Swift ABI is much more complicated than the Rust or C ABIs, as it has e.g. different behavior inside the defining compilation static linking and things using it via dynamic linking.

And another thing is that IIRC, the Swift ABI requires alloca, which Rust doesn't have yet (and is decently low priority IIUC).

But if a modular ABI is powerful enough to support the Swift ABI, it's probably good enough to support basically any feasible ABI. And it would be nice if the definition of repr(Rust) and/or repr(C) could be separate from rustc, I suppose.

Personally, I'd target the MVP at being able to specify repr(C), stage 2 at (most of?) repr(Rust) (~niche filling), and stage 3 at the (most of?) nongeneric subset of repr(Swift) (witness tables outside the main static link).

6 Likes
19

IIRC, in some situations where alloca isn't appropriate, another technique is used.

20

Would it be good to create some Wiki pages somewhere maybe to outline a more formal specification of the proposal or a couple different versions of the proposal that could be critiqued?

It seems like the general consensus is that it is an overall good idea, but the details need to be worked out. In order to work out the details we probably need to start spec-ing stuff out, even if it needs to be heavily revised so that we can work out specific points that need to be addressed.

Also, before even attempting to spec this out we should be sure that we are clear on the goal or "vision" of the proposal. What are we trying to accomplish with it? If necessary we could modify the vision later, if it makes sense because of technical limitations on what we can accomplish in a reasonable proposal, but we should come up with an initial "what is the goal of the proposal?". What should the proposal accomplish?

I think @CAD97's suggestion to target this in stages might be a good idea:

The only potential problem I see of taking the proposal in stages from simpler to more complicated ABIs is that we might have to refactor the plugin API each time we need it to be able to support a more complicated ABI, and I'm not sure if it would be better just to try and take into account the complication necessary for the Swift API in the first place.

I am leaning towards doing it in stages, but others might know better what would be the most efficient way to tackle that problem.

5 Likes