Dogfooding -Z build-std in rustbuild

Hi everyone. So -Z build-std has progressed swimmingly, but there still some rough edges. My worry is that because -Z build-std is mainly useful for "weirder" use-cases of Rust, far from the daily lives of most people working on the tooling, it is unlikely that these issues will go away completely. (Though, never discount more heroics from @ehuss :).)

I think the solution here is dogfooding by a larger cohort of Rust's core contributors. And the most natural place to dogfood would be rustbuild, i.e. how the standard library is officially built.

A seeming downside/risk of any plan that moves logic from the rustbuild to Cargo is that Cargo will need to be changed more often to accommodate the latest needs of bootstrapping. However, after talking with @eddyb I realized we can do even better than the two original plans listed below. Thanks to the release train system, and beta, this is much less bad than in a stable -> master system where, as soon as the previous release is cut, one is largely unable to change the bootstrap. With beta, yes, we still cannot change how it is bootstrapped because it boots from the (now fixed) stable release, but we can change the version of Cargo it contains, to have any new (unstable of course) features than the nightly bootstrap needs. In practice, we shouldn't be changing the beta bootstrap anyways, but should be changing nightly's so this is a well-suited escape hatch. (Of course, the best situation is simply planning ahead and adding the features needed to Cargo the previous nightly, no beta PR needed.)

So putting all that together, the migration would look something like this:

  1. Open PRs:

    • On the cargo side, Implement new (unstable!) features to Cargo needed for this:

      • Explicit std lib deps for the likes of hashbrown and cfg_if that depend on core but are depended-upon by std.
      • Ability to specify the stdlib sources with more freedom, since the stage0 stdlib is using the new stdlib with the old compiler, and there is little point copying the contents repo just to fake the layout rust-src component.
      • ...whatever else we find is needed
    • On the rust side:

      • Create a PR to Beta just bumping the Cargo submodule to contain the new Cargo PRs.
      • Create a PR to master also bumping the Cargo submodule, but also doing the actual work of getting rustbuild to use -Z build-std. In this second PR, arrange CI to bootstrap from the Beta PR instead of actual Beta so CI will pass.
  2. Once the Cargo PRs are merged, and beta branch (through the normal process!) has pins a Cargo version with them merged:

    • Close the Rust PR to beta, it is no longer needed
    • Remove the Hacks from the Rust PR to master, as regular beta will do.

    Because the Cargo versions are similar/the same, before and after the hacks are removed, the PR should continue to pass CI, and it can be merged.

  3. Once the new -Z build-std-based bootstrap hits beta/stable, it will (by definition) have buy-in, and on the off-chance that there are more mission-critical than refactoring the build system, one can hotfix-bump Cargo submodule in Beta rather than wait a cycle.

So what do you all think? I opened Use in rustbuild · Issue #19 · rust-lang/wg-cargo-std-aware · GitHub on this topic long ago, but that didn't see much activity. I gather this question needs to be asked a broader venue, as the working group cannot unilaterally decide what to do with the official build system, anyways, so the planning must be in conjunction with others.


P.S. I'm also incentivized to work on this now because of my slight involvement with the Rust-for-Linux efforts (now with a new mailing list if I may plug. I rather forcibly argued previously that -Z build-std was the way to go since projects like Linux need to be in deep control of the compilation target, even at the cost of stability. (C.f. Linux actually ships it's own compiler-builtins/libgcc/compiler-rt equivalent --- something I only learned during that prior conversation.)

But for my argument to pan out, -Z build-std needs to be a bit more distribution friendly, e.g. not requiring rustup (for compiler sources). These issues are exactly the same ones that using -Z build-std for rustbuild would face, so the proposed dogfooding aligns especially well with this Linux kernel use-case.


Old plan variations

At first, rustbuild would need to support the existing and a new -Z build-std method, since the latter will probably require new work in Cargo etc. to get going. And yes, supporting two methods would be a maintenance burden. However, after the release cycle rolls over and Rust is booting from a sufficiently new version of Cargo, the old way can go away. Not only is there now only one code path in rustbuild again, but also one across the entire community, as everyone using -Z build-std for weird platforms is no longer doing something "off the beaten path". I think this is a huge benefit worth that initial pain.

Edit A better plan per the discussion below is to change the bootstrapping so that we first build the latest Cargo, and then do everything else with that. This will allow us go to one unified code path immediately, and never be stuck waiting for change to reach Beta. This is good since, as @bjorn3 points out, even once things do work bootstrapping with -Z build-std, it's likely further changes to Cargo will be desired. Fundamentally, the more work is moved from the bootstrap build system to Cargo proper, the more the bootstrap will need to leverage new features in Cargo rather than itself, so this change seems in order for any plan to reduce the "bespokeness" of the bootstrapping process, not just this specific plan with -Z build-std.

8 Likes

cc @pietroalbini

cc @Mark_Simulacrum on this, since he's the one doing most of the maintenance of rustbuild.

All "rustup" components can be installed without rustup. Many are just not installed by default. On Debian for example the following rust packages exist: cargo, cargo-doc, libstd-rust-1.41, rust-doc, rust-gdb, rust-lldb, rust-src, rustc.

I think dogfooding -Zbuild-std would require rustc to be built against the libstd sources of the bootstrap compiler. IMO this is a nice thing as it will avoid having to keep a single libstd buildable by two rustc versions, but it is something to be aware of.

This is already enough to sink the idea, it got rejected last time I suggested it: https://zulip-archive.rust-lang.org/131828tcompiler/38874Buildingrustcwithbetalibstd.html

@bjorn3

All "rustup" components can be installed without rustup.

My perhaps out-of-date understanding was then even if one can install the sources, Cargo is very picky about where it wants to find them. Hopefully I am wrong!

I think dogfooding -Zbuild-std would require rustc to be built against the libstd sources of the bootstrap compiler.

How come? I am sympathetic to doing that for other reasons, but I don't see why it's necessary here. If rustbuild already uses Cargo to build the new standard library with the old compiler, what about -Z build-std would prevent that?

Yes, it want them at a very specific place in the rustc sysroot.

-Zbuild-std uses the source bundled with the compiler that is used to compile libstd. It doesn't have a way to override it to the source in library/ of the rust-lang/rust checkout.

1 Like

Right, to me there are fairly superficial restrictions that will absolutely need to lift for -Z build-std to make this go.

I think my concern is that these issues don't seem like they generalize to anything other than rust-lang/rust. Who else will want to point to a custom sysroot? Who else will want an output directory for the standard library that's different from other dependencies?

And then the other issue is that even if you make build-std flexible enough for bootstrap, it's still not achieving your goal because x.py is a build system around cargo. Just because build-std happens to work on SPARC Solaris with RUSTC_SNAPSHOT set to a rustc shim doesn't mean it will work with normal -Z build-std.

I think using build-std in x.py is useful in the sense that it makes the build system for rust-lang/rust less special, but I don't think it will have major advantages for stabilizing build-std itself.

I think my concern is that these issues don't seem like they generalize to anything other than rust-lang/rust.

Oh but they do!

Who else will want to point to a custom sysroot?

  • I want to be able to patch the standard library without rebuilding rustc itself, e.g. for NixOS's support for Redox OS. (I contribute to NixOS so these things are close to mind.)

  • the very concept of a "sysroot" is unnecessary with -Z build-std, so I would like to try to avoid it wherever possible. (Doing that makes NixOS things easier too.)

Who else will want an output directory for the standard library that's different from other dependencies?

I don't understand this? You mean because bootstrap has a certain directory structure?

because x.py is a build system around cargo

I would argue a significant part of what x.py does is recreating -Z build-std. I think x.py will slowly wither away once we do this change.


I think using build-std in x.py is useful in the sense that it makes the build system for rust-lang/rust less special, but I don't think it will have major advantages for stabilizing build-std itself.

If we agree it's "useful in the sense that it makes the build system for rust-lang/rust less special", that's good enough for me!

To be clear, I think this dog-fooding is a beginning of a journey, not an end. :slight_smile:

1 Like

Please read through Bootstrapping - Guide to Rustc Development before saying that 'x.py is just calling cargo'. Everyone who learns about this for the first time thinks it can be done with cargo and it really can't. See also https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/why.20is.20x.2Epy.20necessary.3F, https://zulip-archive.rust-lang.org/131828tcompiler/38874Buildingrustcwithbetalibstd.html, https://rust-lang.zulipchat.com/#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/MCP.3A.20move.20compiler.2F.20crates.20to.20stable.20Rust.20compiler-team.23358, this has been discussed a lot.

the very concept of a "sysroot" is unnecessary with -Z build-std, so I would like to try to avoid it wherever possible. (Doing that makes NixOS things easier too.)

Please read through the section on sysroots before dismissing it as unnecessary. Chesterton’s Fence: A Lesson in Second Order Thinking

I don't understand this? You mean because bootstrap has a certain directory structure?

Bootstrap does not want the dependencies from libstd to mix with the dependencies for rustc, because crates in the libstd sysroot are distributed with libstd and have to be marked with -Z force-unstable-if-unmarked; crates for rustc are only distributed with rustc-dev so it's ok if they get missed. This is explained more in the sysroot section of the guide.

To be clear, I think this dog-fooding is a beginning of a journey, not an end. :slight_smile:

Fine by me, as long as you don't expect build-std to replace x.py.

Let me rewind a bit. So the sysroot isn't going away in that creating sysroots is largely the point of x.py. But sysroots don't need to be the method of x.py, and -Z build-std allows them not to be.

Yes, I'm very aware of the staging inherent to bootstrapping. Until Cargo learns more about bootstrapping (though it does know a little something build-dependencies and proc-macros :slight_smile:), x.py will certainly, at a minimum, be invoking Cargo multiple times with different inputs.

Right, so the idea might be at first to not use -Z build-std to build rustc, but do use -Z build-std to build the standard library and its dependencies in rustlib. However x.py builds rustc using the in-progress sysroot (contains std lib but not (this stage's) rustc) is fine for now, but building the standard library itself and it's crates.io dependencies, and getting the interleaving right (std -> cfg_if -> core, for example?), is exactly what -Z build-std is all about.

1 Like

-Zbuild-std doesn't have a way to only build a sysroot. It only supports building a sysroot somewhere at an unspecified location and then building a crate with all of it's dependencies against that sysroot. Using -Zbuild-std in rustbuild would require to either build rustc directly using -Zbuild-std or teach -Zbuild-std to only build a sysroot without building anything against said sysroot.

I just though of another disadvantage of using -Zbuild-std in rustbuild. It will make it impossible to change how exactly the sysroot is built without first landing the change in the beta cargo. This is likely going to cause problems when porting to new systems or working around bugs/curious behaviour in the native C toolchain.

1 Like

It only supports building a sysroot somewhere at an unspecified location and then building a crate with all of it's dependencies against that sysroot.

Since Switch build-std to use --extern by ehuss · Pull Request #7699 · rust-lang/cargo · GitHub I do not believe there are any sysroots involved. All the crates are built the "normal" way with some flavor of --extern.

Using -Zbuild-std in rustbuild would require to either build rustc directly using -Zbuild-std or teach -Zbuild-std to only build a sysroot without building anything against said sysroot.

I still don't get what's wrong with what I said in the previous comment:

  1. Build the standard library and its dependencies with -Z build-std,

  2. Create sysroot with the results of the above.

  3. Build rustc the normal way with newly created sysroot.

It will make it impossible to change how exactly the sysroot is built without first landing the change in the beta cargo.

Well, since it already uses Cargo we already have a version of the problem, albiet less of one?

What we could do is first build the new Cargo with the old stage 0 tools, and then use that for the rest of the bootstrap. Yes, this means a longer critical path, but it is easy to cache (so CI is no slower), and also means that -Z builds-std can replace the old way of doing things immediately, so there's no awkward period of needing two code paths or being stuck waiting for stuff to hit Beta. Sounds great to me!

1 Like

Is this actually possible? Somewhere else in this thread, it's mentioned that you can build a crate foo with -Z build-std which will build std and then foo (with that std). But how would you just build std? Would that really be “dogfooding -Z build-std”? Cargo already builds std on its own just fine in the current rustbuild setup.

See the use of the explicit core dep in cfg-if/Cargo.toml at 349913689695130c2c63d1075e57b0917c98ee6e · alexcrichton/cfg-if · GitHub. this is very much replicating functionality that should be standard and work with -Z build-std and the official bootstrap alike. So even if std itself is sort of a boring case, we definitely want crates.io libraries that sit between std and core to build more idiomatically.

Ah, yes, getting rid of that hack from some of the crates I maintain would be marvellous. I guess that's conceptually more like -Z build-core but probably technically the same.

1 Like

Yes it's confusing, but per Unstable Features - The Cargo Book one does e.g. -Z build-std=core,alloc today to build less than std. Once we have proper ways to right own the dependencies, we shouldn't need that as it can be purely demand-driven.

In the Linux kernel (assuming we are successful) we will be building core etc. without Cargo. At the moment, we are building it pretty much as any other Rust source. It works quite well, it is reasonably fast to compile and it is a simple build procedure. Overall, it has been a pleasure to setup.

So while build-std seems to me like an important feature, please keep in mind that some projects will want to have a simple procedure to build the standard library (in particular core and alloc).

A couple notes I got so far:

  • For us to use core etc. "as-is", we will eventually need to be able to compile them with a stable off-the-shelf compiler.
  • It would be nice to have some cfg/feature flags to disable certain subsets of core functionality that some environments may not need at all (e.g. floating-point, Unicode...).