A vision for portability in Rust

[aturon]

Thanks all for the discussion so far!

It’s clear that there’s a bunch to talk through here, to more fully flesh out a vision. What’d be really helpful is to hear from folks whether are interested in committing some time this year to pushing on this story, and becoming part of a Portability Working Group. (@Alfriadox I saw your post, thanks!)

I mention this both because it’s important to gauge what kind of bandwidth is available (the existing Rust teams can’t take this on) and because I think some voice calls would probably be helpful in brainstorming some of the overall picture.

[eternaleye]

The problem is not with threads and windows, it’s with windows and linux. What does it mean for a binary to have both windows and linux features enabled?

It means you’re on Cygwin, or perhaps midipix, or maybe even winelib.

EDIT: I’ll note that I’m about as fond of these umbrella feature flags as I am of user-agent sniffing. Midipix and Cygwin support Unix paths, but have to deal with Windows’ idea of reserved characters; meanwhile Winelib allows anything the underlying FS supports. On the other hand, Midipix and Winelib can use Windows GUI constructs, while Cygwin cannot.

It seems to me that if cargo were to drive this, it would need a new mechanism for exclusivity. std cannot be built for two (or more) OSes at the same time, cannot have two memory allocators, cannot have two threading implementations, etc…

Yeah, I’ve mentioned that being desirable before.

[jmst]

Seems dubious, since splitting in multiple crates makes the code easier to understand and maintain.

How about instead doing essentially the opposite:

1. Make std/core/alloc normal crates importable just like any other dependency
2. In a new epoch, remove all re-exports from libstd
3. Where possible, move anything gated by cfg to a crate specific to that cfg, e.g. Windows-only things to std_windows, Unix-only to std_unix, etc.
4. Split std further where appropriate: for example, filesystem access and networking should each go in its own crate, etc.
5. i128 can and should be implemented for all platforms
6. SIMD/atomics can potentially be implemented for all platforms with emulation, and anyway they can be in their own crates

This way, everything will be forced to use core for libcore exports, alloc for liballoc exports, the Windows crate for Windows-only functionality and so on and it will be obvious if something is #[no_std] compatible, whether it only works on Windows, etc. with no lints required.

[Alfriadox]

@jmst I believe the purpose of this is to make rust (and rust projects) more portable. This sort of crate splitting would make that very difficult, as now anyone who wants to write something cross-platform in rust would have to write and maintain three separate versions, instead of one using a unified std.

Using a single std with cfgs (and perhaps better documentation of those cfgs) is probably better.

Of course, it would be optimal to follow a sort of WORA (write-once-run-anywhere) principle, and we should do that wherever possible.

[scottmcm]

One thing I do like about core is that you can easily say “The language itself only depends on core, which is easy to support on all platforms”. I think that message gets fuzzier if it’s “well, on limited platforms a bunch of stuff isn’t available, so it’s small”. Especially when I think of how a bunch of places use C++ — the whole standard library is available for their platform, and they could use it, but they decide they want to use their own abstractions over everything.

I’m definitely a fan of unifying outside that level, though (os, alloc, …). And look forward to being able to use .into() to turn a u32 into a usize

[Ericson2314]

I would be interested in being on such a committee. My nixpkgs cross compilation work has been winding down, so hopefully I’ll have more time.

[jethrogb]

I’d be happy to contribute time to the portability WG.

I think for this proposal to work, the compiler needs to be able to compile different configurations at the same time, making decisions about what to include only at link time. This is quite a departure from the current situation.

In addition, I think the following features need to be added to cargo:

• Make choices at the top level for all dependencies (panic handling, allocation, etc.)
• Allow (but don’t require) dependencies to pick up on these choices during compilation (crates might want to behave differently based on e.g. whether a filesystem is available).

[est31]

I really like this idea. It makes adding support for limited environments much easier. One example where this would help greatly is with std features for crates.io crates. Right now, if you want to give compat for non-std environments, you need to add such a std feature and add a lot of (useless) boilerplate like use core as std or something, even if your entire codebase doesn’t use a single thing from std. Due to limitations with how cargo features work, if you are depending on a crate that has such a std feature, and you are adding additional functionality to that crate, you’d have to add an std feature of your own and make your std feature depend on it.

A good example for this would be the RustCrypto hashes crates that all depend on the digest crate. The digest crate itself has an optional std feature. But the hashes crates don’t expose their own std feature so you are forced to live without std support. This means missing impls for Read for example. Read more on this issue here. You can probably avoid it by depending on the digest crate yourself and turning on the std feature but that trick is more of a hack. With the new system this would be so much nicer.

Part of this could be alleviated by having a system like the one envisioned by @withoutboats, with opportunistic features. This would be as simple as adding a std and alloc cfg flags to rustc or cargo for targets where there is no std available. This would allow us to avoid the flawed cargo feature system. While the automatic features suggestion is great in itself, one can go further and unify everything into one std crate as then you won’t have to remember to do extern crate core as std; or something all the time. So the proposal put forth by @aturon is I think the best solution to this!

There are a few points I’d make though:

• Like @jmst said, i128 can be implemented on every platform. The fact that it is not has nothing to do with missing support by the target hardware. Most hardware doesn’t support i128 natively. Instead, the people who wrote the backends didn’t focus their attention onto i128. So IMO instead of forcing users to use custom i128 polyfills, i128 should be implemented for every platform, either by convincing the backend developers that i128 is important, or by doing lowering and other operations ourselves. The second option is easier on the backend developers and allows us to target things like nvptx earlier in time so probably prefferable. There should be no target where users are required to use custom-made i128 libraries! I do not think that we ever should have a cfg on i128. Instead if someone adds a target to Rust they should either implement custom lowering or ensure the backend works. We shouldn’t make users suffer from LLVM bugs!
• This post is very vague about the portability lint itself, just like the actual RFC that introduced it was. The RFC that introduced it spent more time talking about irrelevant stuff like removing std::os and moving its elements elsewhere. I’d love to talk about the design of the portability lint, especially how to enable it. IMO, this should be a Cargo.toml feature. You should be able to say “this crate supports no_std environments” inside Cargo.toml. I don’t think that categories are a good solution for this, instead I’d love if you say “this crate supports no_std, no_alloc environments”, the portability lint gets turned on automatically and can’t be turned off, even for targets that do have std and allocator support.
• @scottmcm raises a good point I think that right now it is very easy to talk about no_std and core and similar. We should develop new terminology that is just as convenient to use.

[thejpster]

I don’t have the greatest bandwidth, but most of the Rust I write is embedded Rust, which means I regularly get papercuts with core vs std. I’d therefore be very interested in helping where I can. I’m going to poke @japaric too.

PS: I’m also excited about what we can do for Redox, TockOs and Fuschia, as well as the usual suspects.

[gnzlbg]

and becoming part of a Portability Working Group.

I’d be interested on this as well.

[steveklabnik]

even if your entire codebase doesn’t use a single thing from std.

If your project uses nothing from std, you can just unconditionally no_std.

[matthieum]

I though about this before a bit. My main gripe against this approach is that you can’t really implement part of a trait. In other words, we either end up with tons of really small traits (which is annoying) or we end up with a small number of huge traits and if your platform doesn’t support something, you simply unimplemented!() (which is poor UX). Neither option seems ideal.

There is a middle-ground approach: use traits with feature-gated methods.

That is, the trait itself gives the complete interface that can be implemented, while features are used to control exactly which methods are available.

If a trait implementation is incomplete, and the feature is turned on, then the compiler will complain about the incompleteness.

[mark-i-m]

There is a middle-ground approach: use traits with feature-gated methods.

That’s a bit too magic-y for my taste, though. It would be this weird thing that only std traits can do.

[Alfriadox]

Why only std traits? This could be possible for all traits, couldn’t it?

[matthieum]

That’s a bit too magic-y for my taste, though. It would be this weird thing that only std traits can do.

Ah! Ah!

feature is way overused. I didn’t mean feature-gated as in feature-gates, but as in gated by a cfg feature…

That is:

trait Threading {
    #[cfg(feature = "process")]
    unsafe fn execve(...);
}

And then, you only need to implement execve if the process feature is activated by the user. And if activated but not supported it leads to a compile-time error.

[bossmc]

I’m willing to help here, the idea of a treasure map of traits to implement for ever increasing std support on a new platform appeals to me, an in general anything that makes the standard library more grokkable/maintainable is a +1 from me .

Also wrapping std in traits might make it easier to mock out OS-level calls which are always a pain to mock in UTs (I know I’ve written my share of MockCondVarAndMutex in C++ to test threading logic…)

[matthieum]

Also wrapping std in traits might make it easier to mock out OS-level calls which are always a pain to mock in UTs (I know I’ve written my share of MockCondVarAndMutex in C++ to test threading logic…)

I was actually wondering about that.

The way I see it, a user would specify #[test(thread = MockThread)] and the test would be instantiated with the MockThread implementation of the Thread trait, referring to a local value.

Among immediate benefits:

• possible to test I/O code without actually doing I/O, …
• possible to isolate each test, since each gets a different instance of the trait, so you can still parallelize tests which use a mock filesystem/threading/… trait without risk of conflicts.

The possibilities are quite exciting.

[retep998]

Runtime trait dispatch brings back traumatic memories of the old IO where everything was done through dynamic dispatch to support both libgreen and libnative. Please make sure the solution is done using static dispatch.

[rpjohnst]

Really traits feel like the wrong solution for cross-platform dispatch- they not only suggest that dynamic dispatch might be available, but they also suggest that multiple implementations may be available in a single compilation and force you to program as if they are. They also make more work for the compiler, which in this case is completely redundant because there will only ever be one implementation at a time.

Ideally this would not use traits at all- a clearly defined stable interface for porting std is great, but defining it in terms of #[cfg] is probably more straightforward, easier to work on top of, and faster to compile.

[comex]

They also make more work for the compiler, which in this case is completely redundant because there will only ever be one implementation at a time.

But in theory, that work could be guaranteeing (ish) that your code will successfully compile on any platform satisfying your requirements, not only the current one, without actually having to test it on every Rust-supported platform. In other words, the difference between Rust’s type-safe generics and C++ templates, but applied to global parameters, such as the current platform.

This would require a bit of new language functionality, and it wouldn’t be perfect, but IMO it would be a real ‘holy grail’ of a feature.