I am not sure that features are the right thing to do here.
The problem is not with threads and windows, it’s with windows and linux. What does it mean for a binary to have both windows and linux features enabled?
It seems to me that if cargo were to drive this, it would need a new mechanism for exclusivity. std cannot be built for two (or more) OSes at the same time, cannot have two memory allocators, cannot have two threading implementations, etc…
I wrote a little bit about what I’m planning to implement here.
The current state of the implementation can be found here (abstract_platform) and here (platform_unix).
Currently I’m blocked by the lack of GATs, which are needed to abstract some borrowing types.
I didn’t plan for capabilities at compile-time, but they are easy enough to add via cfg flags to the items of abstract_platform::traits::Std.
Oh, right, how did I forget about this?!
Nevertheless, features seem unsuitable for the other reasons given.
I haven’t given the following much though, butiIn the Rust ecosystem these issues are fixed by slicing crates into smaller crates, and if you encounter a problematic crate (bug, platform, …) you just swap it away using cargo. I would like a similar design to be explored for std where each component can be swapped away and, for example, cargo’s target.specific dependencies are used to swap components transparently to the users.
This might require some new language features, like requiring that some crate must exist implementing some API (traits for crates? :D) in the final binary (e.g. for an allocator). I think that this is a problem that we might want to solve anyways because there aren’t only “global allocators”, but also “global executors” and possibly other types of similar resources. Currently we approach this by special casing each type of global resource and adding new features to allow swapping them, but maybe we can do better ?
I don’t know. This is a completely different take on this problem, and even following this route, at some point one is going to need “capabilities” to know which crates to swap where. Also I agree that this route would have many problems but also many of them would be solved if we could move std into the nursery, slice it there, and somehow expose that from rustc.
It’s not only resources, I think, but also “behavior”. For example, I’d bundle the “panic behavior” in there.
And going back to how panic works, it seems that letting the final binary decides which trait implementation goes has some benefits, such as ensuring that a single trait implementation goes in.
It’s a bit like the binary would instantiate:
struct Runtime<
A: Allocator,
F: FileSystem,
O: OperatingSystem<FileSystem = F>,
P: Panic,
T: Thread<OperatingSystem = O>,
>();
impl<...> Runtime<...> {
fn start() { ... }
}
plugging in the implementation of each trait as per the user selection.
I though about this before a bit. My main gripe against this approach is that you can’t really implement part of a trait. In other words, we either end up with tons of really small traits (which is annoying) or we end up with a small number of huge traits and if your platform doesn’t support something, you simply unimplemented!() (which is poor UX). Neither option seems ideal.
I agree. It also makes it unclear how you build a platform-independent crate…
I’m glad this topic is getting it’s due, but I really don’t want to get rid of the facade, even if its just an implementation detail.
rust-lang/rust into rust-lang-nursury, which is really good forYes, it’s true that dealing with different capability of common high level operating systems (how unix-like is darwin…) doesn’t break down nicely, but that’s not what I’d envision the facade being good for anyways. Clean-slate embedded designs are better viewed as a set of environment capabilities, and that’s where the facade approach really shines.
Put another way, the portability lint, (or ML-like module systems), with their support for cycles and full SAT solving, are very flexible, and support all sorts of things which really aren’t a good idea—good or friendly engineering. Its important that we have those tools in our toolbox for really difficult cases, but we should try to rely on them as little as possible!
It sounds like we’re considering a future std that will be covered in #[cfg]s to provide a reasonable API to almost every major target platform (which sounds pretty nice to me). For the minor targets is an aversion to forking std still going to be reasonable after cargo eats and gains the powers of xargo?
Also, is there anything stopping us re-using the existing #![no_std] mechanism for optionally ‘delegating’ lang items? As in:
// in `std`:
#[cfg(feature = "no_default_panic_fmt")]
extern "C" fn panic_fmt(...);
Although, that has the problem (as do a bunch of solutions) that std needs to know ahead of time what’s likely to need to be overridden.
Ideas like @matthieum above make me want Rust to have a ‘real’ module system with the ability to reach in and swap out parts of another crate, which would allow something weird and dangerous like this:
extern crate mod std::io::thread = mod my_custom_thread_mod;
But that’s a whole pile of new problems to deal with.
On that note, like @Alfriadox I would like to help! I’ve tried in the past to do a bare-bones IO-and-alloc port of std to a hypervisor environment which I think hits the ‘fourth-tier target’ category, if that’s relevant.
Thanks all for the discussion so far!
It’s clear that there’s a bunch to talk through here, to more fully flesh out a vision. What’d be really helpful is to hear from folks whether are interested in committing some time this year to pushing on this story, and becoming part of a Portability Working Group. (@Alfriadox I saw your post, thanks!)
I mention this both because it’s important to gauge what kind of bandwidth is available (the existing Rust teams can’t take this on) and because I think some voice calls would probably be helpful in brainstorming some of the overall picture.
It means you’re on Cygwin, or perhaps midipix, or maybe even winelib.
EDIT: I’ll note that I’m about as fond of these umbrella feature flags as I am of user-agent sniffing. Midipix and Cygwin support Unix paths, but have to deal with Windows’ idea of reserved characters; meanwhile Winelib allows anything the underlying FS supports. On the other hand, Midipix and Winelib can use Windows GUI constructs, while Cygwin cannot.
Seems dubious, since splitting in multiple crates makes the code easier to understand and maintain.
How about instead doing essentially the opposite:
This way, everything will be forced to use core for libcore exports, alloc for liballoc exports, the Windows crate for Windows-only functionality and so on and it will be obvious if something is #[no_std] compatible, whether it only works on Windows, etc. with no lints required.
@jmst I believe the purpose of this is to make rust (and rust projects) more portable. This sort of crate splitting would make that very difficult, as now anyone who wants to write something cross-platform in rust would have to write and maintain three separate versions, instead of one using a unified std.
Using a single std with cfgs (and perhaps better documentation of those cfgs) is probably better.
Of course, it would be optimal to follow a sort of WORA (write-once-run-anywhere) principle, and we should do that wherever possible.
One thing I do like about core is that you can easily say “The language itself only depends on core, which is easy to support on all platforms”. I think that message gets fuzzier if it’s “well, on limited platforms a bunch of stuff isn’t available, so it’s small”. Especially when I think of how a bunch of places use C++ — the whole standard library is available for their platform, and they could use it, but they decide they want to use their own abstractions over everything.
I’m definitely a fan of unifying outside that level, though (os, alloc, …). And look forward to being able to use .into() to turn a u32 into a usize 
I would be interested in being on such a committee. My nixpkgs cross compilation work has been winding down, so hopefully I’ll have more time.
I’d be happy to contribute time to the portability WG.
I think for this proposal to work, the compiler needs to be able to compile different configurations at the same time, making decisions about what to include only at link time. This is quite a departure from the current situation.
In addition, I think the following features need to be added to cargo:
I really like this idea. It makes adding support for limited environments much easier. One example where this would help greatly is with std features for crates.io crates. Right now, if you want to give compat for non-std environments, you need to add such a std feature and add a lot of (useless) boilerplate like use core as std or something, even if your entire codebase doesn’t use a single thing from std. Due to limitations with how cargo features work, if you are depending on a crate that has such a std feature, and you are adding additional functionality to that crate, you’d have to add an std feature of your own and make your std feature depend on it.
A good example for this would be the RustCrypto hashes crates that all depend on the digest crate. The digest crate itself has an optional std feature. But the hashes crates don’t expose their own std feature so you are forced to live without std support. This means missing impls for Read for example. Read more on this issue here. You can probably avoid it by depending on the digest crate yourself and turning on the std feature but that trick is more of a hack. With the new system this would be so much nicer.
Part of this could be alleviated by having a system like the one envisioned by @withoutboats, with opportunistic features. This would be as simple as adding a std and alloc cfg flags to rustc or cargo for targets where there is no std available. This would allow us to avoid the flawed cargo feature system. While the automatic features suggestion is great in itself, one can go further and unify everything into one std crate as then you won’t have to remember to do extern crate core as std; or something all the time. So the proposal put forth by @aturon is I think the best solution to this!
There are a few points I’d make though:
i128 can be implemented on every platform. The fact that it is not has nothing to do with missing support by the target hardware. Most hardware doesn’t support i128 natively. Instead, the people who wrote the backends didn’t focus their attention onto i128. So IMO instead of forcing users to use custom i128 polyfills, i128 should be implemented for every platform, either by convincing the backend developers that i128 is important, or by doing lowering and other operations ourselves. The second option is easier on the backend developers and allows us to target things like nvptx earlier in time so probably prefferable. There should be no target where users are required to use custom-made i128 libraries! I do not think that we ever should have a cfg on i128. Instead if someone adds a target to Rust they should either implement custom lowering or ensure the backend works. We shouldn’t make users suffer from LLVM bugs!std::os and moving its elements elsewhere. I’d love to talk about the design of the portability lint, especially how to enable it. IMO, this should be a Cargo.toml feature. You should be able to say “this crate supports no_std environments” inside Cargo.toml. I don’t think that categories are a good solution for this, instead I’d love if you say “this crate supports no_std, no_alloc environments”, the portability lint gets turned on automatically and can’t be turned off, even for targets that do have std and allocator support.no_std and core and similar. We should develop new terminology that is just as convenient to use.I don’t have the greatest bandwidth, but most of the Rust I write is embedded Rust, which means I regularly get papercuts with core vs std. I’d therefore be very interested in helping where I can. I’m going to poke @japaric too.
PS: I’m also excited about what we can do for Redox, TockOs and Fuschia, as well as the usual suspects.