Hi everyone. So -Z build-std has progressed swimmingly, but there still some rough edges. My worry is that because -Z build-std is mainly useful for "weirder" use-cases of Rust, far from the daily lives of most people working on the tooling, it is unlikely that these issues will go away completely. (Though, never discount more heroics from @ehuss :).)
I think the solution here is dogfooding by a larger cohort of Rust's core contributors. And the most natural place to dogfood would be rustbuild, i.e. how the standard library is officially built.
A seeming downside/risk of any plan that moves logic from the rustbuild to Cargo is that Cargo will need to be changed more often to accommodate the latest needs of bootstrapping. However, after talking with @eddyb I realized we can do even better than the two original plans listed below. Thanks to the release train system, and beta, this is much less bad than in a stable -> master system where, as soon as the previous release is cut, one is largely unable to change the bootstrap. With beta, yes, we still cannot change how it is bootstrapped because it boots from the (now fixed) stable release, but we can change the version of Cargo it contains, to have any new (unstable of course) features than the nightly bootstrap needs. In practice, we shouldn't be changing the beta bootstrap anyways, but should be changing nightly's so this is a well-suited escape hatch. (Of course, the best situation is simply planning ahead and adding the features needed to Cargo the previous nightly, no beta PR needed.)
So putting all that together, the migration would look something like this:
-
Open PRs:
-
On the cargo side, Implement new (unstable!) features to Cargo needed for this:
- Explicit std lib deps for the likes of
hashbrownandcfg_ifthat depend oncorebut are depended-upon bystd. - Ability to specify the stdlib sources with more freedom, since the
stage0stdlib is using the new stdlib with the old compiler, and there is little point copying the contents repo just to fake the layoutrust-srccomponent. - ...whatever else we find is needed
- Explicit std lib deps for the likes of
-
On the rust side:
- Create a PR to Beta just bumping the Cargo submodule to contain the new Cargo PRs.
- Create a PR to master also bumping the Cargo submodule, but also doing the actual work of getting rustbuild to use
-Z build-std. In this second PR, arrange CI to bootstrap from the Beta PR instead of actual Beta so CI will pass.
-
-
Once the Cargo PRs are merged, and beta branch (through the normal process!) has pins a Cargo version with them merged:
- Close the Rust PR to beta, it is no longer needed
- Remove the Hacks from the Rust PR to master, as regular beta will do.
Because the Cargo versions are similar/the same, before and after the hacks are removed, the PR should continue to pass CI, and it can be merged.
-
Once the new
-Z build-std-based bootstrap hits beta/stable, it will (by definition) have buy-in, and on the off-chance that there are more mission-critical than refactoring the build system, one can hotfix-bump Cargo submodule in Beta rather than wait a cycle.
So what do you all think? I opened Use in rustbuild · Issue #19 · rust-lang/wg-cargo-std-aware · GitHub on this topic long ago, but that didn't see much activity. I gather this question needs to be asked a broader venue, as the working group cannot unilaterally decide what to do with the official build system, anyways, so the planning must be in conjunction with others.
P.S. I'm also incentivized to work on this now because of my slight involvement with the Rust-for-Linux efforts (now with a new mailing list if I may plug. I rather forcibly argued previously that -Z build-std was the way to go since projects like Linux need to be in deep control of the compilation target, even at the cost of stability. (C.f. Linux actually ships it's own compiler-builtins/libgcc/compiler-rt equivalent --- something I only learned during that prior conversation.)
But for my argument to pan out, -Z build-std needs to be a bit more distribution friendly, e.g. not requiring rustup (for compiler sources). These issues are exactly the same ones that using -Z build-std for rustbuild would face, so the proposed dogfooding aligns especially well with this Linux kernel use-case.
Old plan variations
At first, rustbuild would need to support the existing and a new -Z build-std method, since the latter will probably require new work in Cargo etc. to get going. And yes, supporting two methods would be a maintenance burden. However, after the release cycle rolls over and Rust is booting from a sufficiently new version of Cargo, the old way can go away. Not only is there now only one code path in rustbuild again, but also one across the entire community, as everyone using -Z build-std for weird platforms is no longer doing something "off the beaten path". I think this is a huge benefit worth that initial pain.
Edit A better plan per the discussion below is to change the bootstrapping so that we first build the latest Cargo, and then do everything else with that. This will allow us go to one unified code path immediately, and never be stuck waiting for change to reach Beta. This is good since, as @bjorn3 points out, even once things do work bootstrapping with -Z build-std, it's likely further changes to Cargo will be desired. Fundamentally, the more work is moved from the bootstrap build system to Cargo proper, the more the bootstrap will need to leverage new features in Cargo rather than itself, so this change seems in order for any plan to reduce the "bespokeness" of the bootstrapping process, not just this specific plan with -Z build-std.
