A vision for portability in Rust

1

TL;DR: This post proposes to deprecate the std facade, instead having a unified std that uses target- and capability-based cfgs to control API availability.

Portability is extremely important for Rust, in two distinct (and sometimes competing!) ways:

  • Rust should be usable in almost any environment, and ideally much of the ecosystem would be as well.

  • Rust should be low-friction when writing for “mainstream” platforms (32- and 64-bit machines running Windows, Linux, or macOS).

An example of the tension between these two goals is handling allocation:

  • Some targets for Rust do not support allocation natively, so Rust must at least have a “mode” in which no allocation is assumed.

  • For “mainstream” applications and platforms, we want to assume not only that allocation is available, but that running out of memory is a catastrophic failure. Those assumptions are reasonable for a huge amount of software, and making them greatly reduces the friction to writing Rust code.

We’ve been slowly evolving a set of answers to this kind of question, and part of the point of this blog post is to step back and try to give a unifying vision for how to approach portability issues in Rust.

Read the rest of the post and leave comments here if you want to be involved!

49 Likes
2

This sounds fantastic, and would address my longstanding woes with using alloc in “a heap and not a whole lot else” no_std environments.

One important consideration, I think, would be allowing environments to swap in their own implementations of certain std “capabilities”. This has been a major pain point for Rust SGX, where e.g. the Baidu SDK has much of std working but using a mechanism that has to trampoline through SGX OCALLs to do things like reach the network or interact with the filesystem. They have produced their own sgx_tstd crate to address this:

To my knowledge though they are now using xargo, they have had trouble swapping this implementation in for std, and I’m sure it contains a lot of redundancies and copypasta from the real std. I think it would be very nice if they could use the parts of real std that work within SGX, then only swap in their own implementations of e.g. std::io

11 Likes
3

I like this plan a lot, especially undoing the facade and unifying std into a single crate.

And to fully gain from abandoning the facade (i.e., to remove the special magic used in std today), we would need to use an epoch boundary to fully remove libcore.

I don’t think we can remove libcore since we need to keep supported previous epochs. But I think we could:

  • Replace its contents with pub use std::*;
  • Make !#[no_std] imply the appropriate portability lint
  • Later, (perhaps in new epochs only,) make extern crate core; or #![no_std] emit a deprecation warning.
14 Likes
4

cc @jethrogb @arcnmx @cuviper @josh @retep998 @zoxc

1 Like
5

Aha, of course! This sounds great.

1 Like
6

cc @panicbit

7

Thanks @aturon ! This is sorely needed.

I would particularly like

  • if std was completely oblivious to mainstream platforms. i.e. there is no cfg(windows) or cfg(unix). Rather, we have a huge collection of feature/capability flags. We can then define subsets of the flags in cargo (not rustc or std) to turn on/off the appropriate features.
    • In embedded/bare metal systems, where we really want minimal binaries, this provides a way to opt-in to the minimal subset of std needed.
    • It is a lot cleaner to not have to maintain cfg(platform1), cfg(platform2), … cfg(platformN) in-band
    • This would improve documentation. Rather than having a different libstd rustdoc set for each platform, we can simply have one set of docs and tag each item with the required capabilities. Then, you can just check if the needed capability exists on your target (and maybe implement it yourself).
  • if each feature/capability flag can be replaced/implemented in a completely stable external crate.
    • This would allow support for windows, unix, redox to be completely factored out.
    • This would make it possible to port the entire libstd to a new platform more easily. No need to fork rust-lang/rust. No need to add PRs to normal rust for specific platforms. No need to hunt through libstd for the right impls to add for your new platform. Instead, you just implement the feature flags you want for your platform and anything that depends on them magically appears.
9 Likes
8

if each feature/capability flag can be replaced/implemented in a completely stable external crate.

This would be fantastic.

I have always wondered if a trait-based approach to customization could be used. That is, if it'd be possible to cleanly separate the various functionalities provided by the platform under std into a DAG of traits.

First of all, having traits would document the actual interface that a platform needs to provide, which helps with portability.

Secondly, having a DAG would help a platform port little by little, start at the root, and implement traits one at a time, unlocking new functionalities each time... and stop implementing when you reach the limits of the platform (maybe there's no filesystem or network access?).

Finally, composing the "final" std would be a simple matter of pulling in one implementation for each trait into the std library, based on the target platform, for a compile-time resolution of the trait implementations allowing appropriate inlining.

2 Likes
9

I think is this a good idea, though I suggest we use cargo features for libstd capabilities. This makes it possible to tell which libstd capabilities crates require. So we should have alloc, threads and windows features. The windows feature which adds Windows specific extensions should not be default and crates must explictly opt-in in Cargo.toml to use Windows-only features. The same should apply to other platform specific features.

This does not address the problem of porting libstd to new platforms without forking it however. I think that is also quite important. It seems like removing libcore would make it hard to split platform specific code out of the libstd crate. Perhaps just leaving libcore as it is as an internal implementation detail of libstd would work?

4 Likes
10

That seems to require Cargo to be aware of std, which has been desired for a long time but also, well, hasn't moved forward in a long time. It would be quite unfortunate to block progress on this front on an unrelated feature that will take who-knows-how-long to land.

Besides that, I'm not sure whether this is also supposed to replace the cfg-based portability lint? If so, I'm not sure whether that can work. Cargo takes the union of all features requested by all dependent crates. So if any crate in the entire dependency graph enables (for example) the windows feature, all other crates get access to windows-specific features too, whether they requested the feature in their Cargo.toml or not.

11

I am not sure that features are the right thing to do here.

The problem is not with threads and windows, it’s with windows and linux. What does it mean for a binary to have both windows and linux features enabled?

It seems to me that if cargo were to drive this, it would need a new mechanism for exclusivity. std cannot be built for two (or more) OSes at the same time, cannot have two memory allocators, cannot have two threading implementations, etc…

6 Likes
12

Hasn't it had some movement in [Request for Experiment] Sysroot building functionality · Issue #4959 · rust-lang/cargo · GitHub?

13

I wrote a little bit about what I’m planning to implement here.

The current state of the implementation can be found here (abstract_platform) and here (platform_unix).

Currently I’m blocked by the lack of GATs, which are needed to abstract some borrowing types.

I didn’t plan for capabilities at compile-time, but they are easy enough to add via cfg flags to the items of abstract_platform::traits::Std.

1 Like
14

Oh, right, how did I forget about this?!

Nevertheless, features seem unsuitable for the other reasons given.

15

I haven’t given the following much though, butiIn the Rust ecosystem these issues are fixed by slicing crates into smaller crates, and if you encounter a problematic crate (bug, platform, …) you just swap it away using cargo. I would like a similar design to be explored for std where each component can be swapped away and, for example, cargo’s target.specific dependencies are used to swap components transparently to the users.

This might require some new language features, like requiring that some crate must exist implementing some API (traits for crates? :D) in the final binary (e.g. for an allocator). I think that this is a problem that we might want to solve anyways because there aren’t only “global allocators”, but also “global executors” and possibly other types of similar resources. Currently we approach this by special casing each type of global resource and adding new features to allow swapping them, but maybe we can do better ?

I don’t know. This is a completely different take on this problem, and even following this route, at some point one is going to need “capabilities” to know which crates to swap where. Also I agree that this route would have many problems but also many of them would be solved if we could move std into the nursery, slice it there, and somehow expose that from rustc.

1 Like
16

It's not only resources, I think, but also "behavior". For example, I'd bundle the "panic behavior" in there.

And going back to how panic works, it seems that letting the final binary decides which trait implementation goes has some benefits, such as ensuring that a single trait implementation goes in.

It's a bit like the binary would instantiate:

struct Runtime<
    A: Allocator,
    F: FileSystem,
    O: OperatingSystem<FileSystem = F>,
    P: Panic,
    T: Thread<OperatingSystem = O>,
>();

impl<...> Runtime<...> {
    fn start() { ... }
}

plugging in the implementation of each trait as per the user selection.

3 Likes
17

I though about this before a bit. My main gripe against this approach is that you can't really implement part of a trait. In other words, we either end up with tons of really small traits (which is annoying) or we end up with a small number of huge traits and if your platform doesn't support something, you simply unimplemented!() (which is poor UX). Neither option seems ideal.

:+1: I agree. It also makes it unclear how you build a platform-independent crate...

18

I’m glad this topic is getting it’s due, but I really don’t want to get rid of the facade, even if its just an implementation detail.

  • Facade enforces acyclicity, which is good for understanding, quite frankly, what the hell is going on.
  • Acyclicity allows us to move stable libraries out of rust-lang/rust into rust-lang-nursury, which is really good for
  • on-boarding more contributors
  • Helping out alternative implementations

Yes, it’s true that dealing with different capability of common high level operating systems (how unix-like is darwin…) doesn’t break down nicely, but that’s not what I’d envision the facade being good for anyways. Clean-slate embedded designs are better viewed as a set of environment capabilities, and that’s where the facade approach really shines.

Put another way, the portability lint, (or ML-like module systems), with their support for cycles and full SAT solving, are very flexible, and support all sorts of things which really aren’t a good idea—good or friendly engineering. Its important that we have those tools in our toolbox for really difficult cases, but we should try to rely on them as little as possible!

1 Like
19

I would like to help! How can I help?

2 Likes
20

It sounds like we’re considering a future std that will be covered in #[cfg]s to provide a reasonable API to almost every major target platform (which sounds pretty nice to me). For the minor targets is an aversion to forking std still going to be reasonable after cargo eats and gains the powers of xargo?

Also, is there anything stopping us re-using the existing #![no_std] mechanism for optionally ‘delegating’ lang items? As in:

// in `std`:
#[cfg(feature = "no_default_panic_fmt")]
extern "C" fn panic_fmt(...);

Although, that has the problem (as do a bunch of solutions) that std needs to know ahead of time what’s likely to need to be overridden. Ideas like @matthieum above make me want Rust to have a ‘real’ module system with the ability to reach in and swap out parts of another crate, which would allow something weird and dangerous like this:

extern crate mod std::io::thread = mod my_custom_thread_mod;

But that’s a whole pile of new problems to deal with.

On that note, like @Alfriadox I would like to help! I’ve tried in the past to do a bare-bones IO-and-alloc port of std to a hypervisor environment which I think hits the ‘fourth-tier target’ category, if that’s relevant.

1 Like