A Stable Modular ABI for Rust

I'm glad you already thought of it, and mentioned it!

1 Like

Just forbid them. Library interfaces would have to use dyn Trait or concrete types instead. IMHO this is quite sensible limitation, especially for an MVP

I think doing that is very reasonable. Then the library developer could still get the second and third behaviors in many cases by doing something like:

pub fn int_generic() -> Result<i32, Box<dyn Error>> {
  return generic<i32, Box<dyn Error>>();
}
pub fn dyn_generic() -> Result<Box<dyn Any>, Box<dyn Error>> {
  return generic<Box<dyn Any>, Box<dyn Error>>();
}
2 Likes

This is something I'd love to see released as part of the 2024 edition. Hear me out:

Rust 2018 was released with Futures to big hurrah, but the implementation hasn't been completed until just recently. The push to get something feature-complete out the door led to a lot of corner-cuts that are still being felt by those trying to implement generators. Now it was a good choice to stabilize Futures early because that's been a very good selling point for developers. ABIs are different though, you don't need to stabilize ABIs to build a Rust library. It's perfectly valid to build your product on stable Rust and then a Swift version on nightly Rust if need be.

Rust's growth has been significantly based on the fact that we are statically linked. I have no proof of this, this is just an opinion, but hear me out. With #[repr(ABI)] feature-gated, companies using Rust can export as Go or Swift libraries but they won't be able to import libraries without committing to nightly. Once we can import DLLs our barrier to entry skyrockets. Large Rust projects like Amethyst or Servo will no longer be buildable by just installing Cargo. Fragment code, partially release source, binary blobs etc will become much more common simply due to the highway effect[1].

A configurable ABI is essential to Rust's long-term growth however. The programming language barrier-to-entry is growing smaller every day. A stable interface to creating custom ABIs will be essential if we want to interop with the same speed and reliability we promise with statically linked Rust. A 4-year timeline restricted by editions will give the working group lots of time to experiment with the project space and create a solid foundation.

[1]: Widening lanes on a congested highway doesn't decrease traffic but it does increase throughput.

2 Likes

@matthieum I'm linking to what you posted over in the users forum here so that people who don't know about the first thread can hear your thoughts and can respond to them:

1 Like

I'm not commenting on the merits of this discussion, it's certainly an interesting one. However, I'd like to point out the claims about Rust on Fuchsia are inaccurate. The Fuchsia project has made no decisions about Rust due to ABI stability, this is actually mostly a non-issue on the platform due to our heavy usage of IPC. Rust is still one of our supported languages :slight_smile:

4 Likes

It's great that there is a renewed discussion about the ABIs.

This proposal mainly regards data structure layouts. Have you given any consideration to how naming of the linkage symbols for functions could be part of a stabilized ABI?

1 Like

Nobody has started thinking about the specifics of how symbols or functions would work yet in a stable ABI, but that is definitely somethign that the ABI plugins will need to be able to customize to support different ABIs.

We're leaning towards not creating a new stable ABI, but building the modular ABI system such that it can support an already-existing stable ABI such as Swift. This saves us the work of actually designing the ABI and leaves us only with the task of designing a modular ABI system that allows plugins of some sort to add new ABIs.

3 Likes

To me it seems like building a modular, semver-aware ABI system would be a lot more work than just implementing another ABI in the compiler directly. Furthermore, the modular ABI system is useless until someone also implements the ABI crate(s) we need.

The modular system sounds nice, but we should really ask ourself if the benefits outweigh the additional complexity.

1 Like

The motivation for the modular aspect is the fact that no attempt to get Rust a stable ABI has been successful so far because of the language team's reluctance ( understandably ) to commit to stabilizing an ABI. Maybe it would be easier just to add Swift ABI support to rustc? I don't know.

1 Like

While I fully understand @Aloso's concerns about how difficult creating a modular, semver-aware ABI system would be, I strongly prefer this solution for the following reasons

  • It allows new ABIs to be added reasonably easily in the future.
  • It permits @Tom-Phinney's idea of inter-ABI glue code. If there was an 'ABI to rule them all', then you could have a star-shaped topology that isolated ABI authors from one another, allowing an old ABI crate to magically translate to a new one that it wasn't aware of.
  • It ensures that if the ABI needs to be changed in the future, it can be done in a principled way.
  • It forces good design.

The last point is what I think is one of the most important ones. The rust developers have expended an enormous amount of energy into designing rust really well, so that we (as end users) don't really need to worry about breakage in some future edition. Stabilizing on a single ABI without a mechanism in place to upgrade if needed will likely bite rust at some point in the future, and probably long after it's too late to fix it. By designing with modularity in mind right from the start, it forces us to think of ways of future proofing an ABI that we might not have thought of otherwise. For example, how do we detect which ABI is in use? What about the particular version of the ABI? What do we do when we run into it? If there is a star topology as above, can an automatic negotiation system be built to dynamically create glue code between the ABIs be developed?

Settling on a single ABI is the quick and easy path, but it probably isn't the best choice in the end.

3 Likes

I think the saying "perfect is the enemy of good" applies here. If we commit to creating this modular ABI system, it can easily take many years until we have a MVP. Instead, we could add a Swift ABI as a nightly feature, so people could start using it right away.

Since nightly features can be removed at any time, it would still be possible to introduce a modular ABI system later. By then, we would already have some insights, what the modular system has to support, to support an ABI as complex as Swift's.

3 Likes

I would love to see this. Something like WebAssembly's interface-types specifications would be a good basis for this.

3 Likes

This is something I see often enough when language interop is discussed on rust-internals. A lot of people often suggest that WebAssembly could be used as a building block somehow, or as a source of inspiration. I want to be as clear as possible:

WebAssembly is not designed to facilitate ABI compatibility between languages, and won't be in the foreseeable future.

In fact, some of the standard's lead designers are strongly opposed to any additions to the standard that aim to facilitate using wasm as a rosetta stone (though "strongly opposed" is a little unfair; more charitably, they are highly skeptical that such features would actually facilitate language interop).

Shared-nothing linking is pretty much dead in the water. Some good work is being done on interface types, but it's focused on standardizing and facilitating things wasm-bindgen can already do today (eg passing strings to and from web APIs).

None of the wasm working groups have made a serious effort to tackle the hard problems that an inter-language type system would face; notably reference graphs, lifetimes, ownership, tagged unions, GCE, vtable format, cross-language garbage collection, type invariants (eg str having to be valid unicode), etc. If Rust wanted to implement such a system, a lot of that design work would have to be done from scratch.

(or using another system; I don't really know what the state-of-the-art is for language-agnostic types; I suspect it's not too impressive, seeing as everyone uses serialization and IPCs these days)

4 Likes

Does that mean FFI is limited to POD types? Or does the above rule also apply to Box, Vec, Arc, etc?

Because the first option is pretty limiting, but the second option has some fairly non-trivial implications.

I'm not that familiar with Swift's ABI, but from the gankra article describing its behavior, I think you're making sound more complicated than it is.

Fundamentally, it's probably just a whole lot of fancier vtables; same thing as when Rust uses dyn, except everywhere.

Both these options make sense; generally speaking, I think Rust is leaving a lot of potential on the table with its current dyn-safe rules. Finding a way to make more traits dyn-safe (for instance, by adding workarounds to the "no associated constants" rule) would help with both of the above options.

Also, implementing the second one would probably help a lot with compile times in debug mode, since it removes the need for monomorphizing most template functions.

2 Likes

To be clear, when I said:

That was in the context of how to implement the ABI crate compiler plugin interface, not how design the inter-language ABI itself.

  1. This is not idle speculation, this is an active area of development. Many of the areas you're talking about are, in fact, being actively worked on.
  2. I'm not talking about an interoperable type system that would start out as capable as the full power of Rust's type system (e.g. lifetimes and full generics); I'm talking about an ABI substantially more capable than the least-common-denominator C ABI. That includes strings, UTF-8, slices, Option, tagged unions, and other types, without having to manually translate them to and from an unsafe C representation.
5 Likes

@Aloso, I agree that the 'perfect is the enemy of the good', and actually support the idea of implementing this as an unstable, nightly-only feature that might be removed at any time unless it interferes with the work that @josh is talking about.

I like @matklad's idea of a universal ABI, but I also know that since it would be a standard that extends outside of rust that it would live more or less forever, ugly warts and all. I'd like to see the group get experience with several different ABIs, then decide what the universal ABI needs to have based on that experience. So, nightly is a good place for stuff right now.

3 Likes

I've been tied up with work lately, so I haven't gotten around to this thread for a while and I apologize for that. Anyway, I'll address some of the questions / comments directed at me since my last post. Some of the answers may have been already been discussed, so if that's the case, please ignore them.

In my opinion, the end goal for a 'stable ABI' would be to allow for Rust to act as 'glue' between languages through dynamic linking and the like. To reach this goal, I propose the following development path:

  1. Create a working wiki/specification/knowledgebase for organization of progress.
  2. Implement demo crate showing interop with Swift's ABI. This is a proof-of-concept to show that such a project is feasible.
  3. Using lessons learned from the Swift demo, design a generalized ABI API.
  4. Implement a loose-yet-functional version of the generalized ABI API.
  5. Once definitions are agreed upon, open an RFC.
  6. Polish the existing ABI API implementation and integrate it with the Rust compiler.
  7. Document the above implementation and develop interfaces for common ABIs (including C, Swift, and Rust, for instance).
  8. Follow RFC processes to merge into the compiler.

Note that the above proposal is technically ambiguous as to how the 'stable ABI' is implemented - I am no expert on the implementation of application binary interfaces, I've merely had to use them a few times.

This is a good point. I agree with that, as @zicklag mentioned, the way WASM compilation is done now is a good model as to how to develop a flexible compiler architecture.

From experience, I've found that many NxM problems can be reduced to Nx1 problems through clever design. Though I do not know entirely what this would look like in the case of a Rust ABI, I think that compile-time procedural-macro plugins that generate layout, calling-convention, etc. information might help solve this. Instead of the Rust core compiler having to implement every ABI for every system, it could define a general interface allowing for individual 'ABI crates' to create that information for the compiler.

As you mentioned earlier, an 'ABI' is a very loose term with far-reaching implications. I agree with you that at the start, such an ABI for Rust should be limited in scope. Additionally, establishing exact terms through a wiki/knowledgebase will allow for more specific issues to be discussed.

I believe for any generalized ABI to be successful, calling conventions should be definable. At first it might just be a good idea to focus on memory layout and assume certain calling conventions, but as a more flexible ABI definition interface is introduced, more of the specification can be moved from the compiler to the individual ABI crates.

I believe that at first, something like #[repr(C)] makes the most sense. Although it might be a pain to annotate everything, I think that explicit annotation is a good thing, for the following reasons: first, as this will be a new feature, making it explicitly opt-in will prevent confusion over hidden behaviour. secondly, something similar was suggested for auto-deriving #[derive(Debug)] for debug builds. The consensus seemed to be that this was not a good idea because it's hidden behaviour, it might break things when compiled in release, and because Debug isn't special enough to be its own case.

This is something that I think that explicit annotation will resolve - in the case of global application, applying it solely to public-facing types makes sense. Re-exported types in this context are more nuanced, so I'm not sure what the optimal solution might be.

I think that'd (one ABI module per target) be the case. 'ABI Crates' would define a specific ABI. when annotated with a proc-macro, they'd expand and provide information to the compiler. Each ABI crates would implement a singular ABI. For example, as discussed earlier, a abi_swift ABI Crate would allow for interop with Swift's ABI on a datastructure-by-datastructure basis (think #[repr(Swift)]).

I think your answers are pretty good; mine are above.

These are good points; an ABI is bi-directional, so there needs to be flexibility with how which ABI are used.

There are no such things as stupid questions, only stupid answers :slight_smile:. To answer your great questions:

Yes, it should, but note that there is still some discussion as to how to allow the same datastructure to be implemented for different ABIs.

I personally believe that crates would be the way to go, but there might be some reasons that I'm not aware of as to why this might not be the best case.

There are a few potential solutions to this problem that I can foresee. The first is to define all ABIs for all required types, and convert between representations at runtime when needed. This is far from the ideal solution, as it would be fairly slow. The next solution would be to try to find common layouts for different ABIs, but this comes with its own set of challenges. I think that limiting it to one ABI per datastructure is a viable for an MVP.

This would work. It might be possible for the compiler to do this automatically if the definitions are simple enough.

Something like this is likely to be (a) a bit undertaking and (b) involve breaking changes to the current compiler pipeline. Rolling this into an edition makes complete and total sense.

I apologize for propogating this false claim :sweat_smile:. I jumped to that conclusion due to the resulting discussion surrounding the original document, but the original document itself does not mention anything of the like. As a side node, I've actually started playing around with Fuschia and am quite impressed with the current state of the project, especially the flexibility and security that the microkernel architecutre and namespaced filesystems introduce.

That's why I suggest we start by writing at least one such crate, integrated with the current compiler pipeline. See abi_stable.

I think that in the short term, creating a simple ABI that interops with Swift or C perce might be the best option. Over the long term, I think extending this framework to be fully modularizable will be work the extra effort.

What if the MVP was the Swift ABI?

It is really just 'fancy v-tables', but the 'fancy' part is what makes it work. We discussed the use of dyn traits earlier, but the problem with dyn is the performace overhead it introduces. Instead of dyn everywhere, I think that dyn for only explicitly-annotated non-monomorphizable functions might be a good idea.

Box is a special case within the Rust compiler right now, so it would be interesting to see how such a problem could be resolved. I've thought through a few of the implications of moving data across FFI boundaries, but I'm not sure I thought of them all. Would you mind listing the implications you thought of?

That's great news! Would you mind linking to some of the places/github issues/PRs/etc. where we can read about the current state of said development?

As I've mentioned earlier, any MVP of such an ABI should include a minimum viable subset but does not need to support the entire type system (and all that it entails) right off the bat. Going forward, I think that it's completely impossible to perfectly specify any non-trivial system. We're best off making an MVP to show the concepts are functional and applicable, then going from there.

What next steps do you think we should take? Should work on an RFC be started, or should an MVP be developed first. What features should the MVP include, and how can exact terminology be established for future work going forward?

1 Like

I don't think you should start with an RFC since the design space is so large. I think it would be a good idea to create a working group of interested people, who can discuss and iterate on the concept until it is ready for an RFC.

If somebody were to start trying to implement some of these ideas in a rustc branch, that would also be very helpful since it could inform the discussion of what is possible and necessary to consider.

2 Likes
2 Likes