The rustbuild feature thread

[Ericson2314]

Thanks I want (to do):

1. Add features to the stdlib! With scenarios there are some big opportunities here.
2. Replace the hoedown submodule with https://crates.io/crates/hoedown . One less submodule!

[alexcrichton]

Thanks for starting a thread @arielb1! I’ve meant to do this for some time now so this is a good motivation to post as well

So for those who may be new to rustbuild, this is the “code name” (still looking for a better one) for our new build system in the Rust compiler itself. Our old makefiles were seriously showing signs of aging and they were ripe for replacement. The heart of rustbuild is building the compiler via Cargo, which is why you’ll find a number of Cargo.toml files lying around in the repo as well.

A rewrite of the build system is no easy undertaking, but there were some key goals of rustbuild which are the underlying motivation:

• The makefiles were completely impenetrable to all but a few, deterring many prospective contributors. A primary goal of rustbuild is to be a readable and easily modifiable build system. Basically anything crazy we want to do with our build should not be hard to implement.
• Unfortunately make isn’t portable to one of our Tier 1 platforms, MSVC, where we want the developer experience to be similar as that of Unix.
• There’s already a standard build system for Rust, and that’s Cargo, so we should use it as well! This ends up opening quite a few options down the road such as compiling libstd on demand if required.

Basically the tl;dr; is everyone should be able to contribute to the build system, no matter if this is your first patch, what platform you’re on, or what you’re goal is.

---

Currently the build system lives in src/bootstrap with a README to boot explaining how to get started with it as well as some of the high level architecture. The source should also be pretty well documented for prospective contributors or just the curious. If you’d like to know any more information about jumping in or the build system in general, feel free to let me know! I’d love to write up more docs to help out.

If you’ve got a feature request feel free to mention it here or open an issue on rust-lang/rust tagged with A-rustbuild (or just cc me, alexcrichton, and I’ll tag it for you). I can help spell out the steps needed to implement anything and help anyone get started as well! You can also reach out to me as acrichto on IRC if you’re interested in working on rustbuild.

Unfortunately a caveat for now is that any fundamental modifications to the build (e.g. changes in dependencies) need to be reflected in the makefiles as well as rustbuild for now. This means we can’t quite have all our hopes and dreams just yet, but I hope to aggressively message out the deprecation of the makefiles this next cycle and if all goes well delete them soon.

[cuviper]

Are the build bots using the new system yet? I would make sure it’s comfortable in the driver’s seat long before you start talking about deprecation and deletion.

From a distro perspective, I’m worried that this will make initial bootstrapping even harder to have cargo in the mix, even more so if it starts depending on a bunch of external crates. It becomes a much bigger footprint before the distro can build in a self-contained way, offline and without external binaries.

I’m hoping the makefiles stick around for a while yet, even if they’re demoted to the secondary build system.

[PeteVine]

I’m hoping the makefiles stick around for a while yet, even if they’re demoted to the secondary build system.

I concur; without a lowest common denominator type build system, it’s not even a proper bootstrap any more. Just using rust tools on the available platforms.

[steveklabnik]

I don’t understand this criticism. Even with the Makefiles, you’re still downloading a binary stage0 in order to perform the bootstrap.

[sorear]

Possibly naive question: Once workspaces and on-demand libstd/libcore compilation land, how far will we be from using cargo to drive the process?

[PeteVine]

We’re talking about cargo here - a real bootstrap procedure would involve building rustc first using just basic tools and a stage0 binary, followed by building cargo with the newly bootstrapped rustc (without needing cargo again, but that’s not possible either).

As rustbuild is a done deal I’m not even trying to be critical.

[eddyb]

I think it’s simpler to think of bootstrapping rustc+cargo from rustc+cargo than having cargo be built before rustc during the bootstrap.
What’s one binary versus two? The real problem @cuviper mentioned, IIUC, is that we could too easily depend on crates.io which for packaging would create much larger complications.

[PeteVine]

It’s not just any binary, but a networked one. If you can’t bootstrap from a local source snapshot, that’s something else entirely. So yeah, you’ve just reworded the real problem.

[cuviper]

I don’t understand this criticism. Even with the Makefiles, you’re still downloading a binary stage0 in order to perform the bootstrap.

At first, yes, this is necessary. Or at least we need to have those binary stage0 tarballs in the srpm, then we can do without network for the actual build. Once we have that rpm completed though, then we need to do it again with an srpm that does not have any binaries, instead using that new distro rustc with --enable-local-rust and/or --enable-local-rebuild. For every new build from there, we’ll need to not have any binaries in the srpm, else we have to do this bootstrap again.

It’s a bit messy, yes, but I’m hoping to keep it from getting any messier. This scheme is in accordance with Fedora guidelines, and a big reason for #29555, #29556, and #29557 from The Plan.

For Cargo it will be similar – bootstrap once with a binary snapshot, then build again using a source-only srpm and --local-cargo. But we do have to decide how to package all of its crates.io dependencies too, if we want a successful offline build.

FWIW, I know I keep speaking sort of hypothetically, and Fedora packages have been a long time coming. But I am finally working on it for 1.10 (spec, copr). Even if it still takes a while for that to get into Fedora proper, I intend to actively track that goal there.

[cuviper]

Would it be useful to have a face-to-face meeting about packaging at or around RustConf? I am attending, and I also live in the area so I could make time while you folks are in town. Hopefully we could get some other distros represented too.

[Ericson2314]

For Cargo it will be similar – bootstrap once with a binary snapshot, then build again using a source-only srpm and --local-cargo.

At least earlier versions of cargo were built with make. There is no read to depend on another pre-built binary.

But we do have to decide how to package all of its crates.io dependencies too, if we want a successful offline build.

I would like to teach Cargo to output a “build plan” instead of executing the build itself. This could be converted into a package in another language.

[alexcrichton]

Are the build bots using the new system yet?

They are indeed! We’ve got builders for Linux, OSX, MSVC, and MinGW all testing rustbuild. Additionally, all cross-compiled nightlies (e.g. ARM Linux, FreeBSD, etc) are all produced by rustbuild.

And yes we’ll of course not just remove the makefiles one day, I should have been more clear on this! My current plan is along the lines of:

• For one cycle, discourage use of the makefiles, encourage developers to use rustbuild and add features to it
• For the next cycle, message out deprecation of the makefiles. Switch bots to using rustbuild by default and leave a bot or two testing the makefiles
• For the next cycle, delete the makefiles.

I’m hoping that gives everyone enough of a warning period to switch off the makefiles, and it primarily should affect developers as the external interface of rustbuild is the same (./configure + make still works)

From a distro perspective

Yes this is something we’ll have to solve, but it’s always been a problem trying to deal with Cargo, I’m sure we’ll come to a reasonable solution.

Possibly naive question: Once workspaces and on-demand libstd/libcore compilation land, how far will we be from using cargo to drive the process?

If you mean from the perspective of the bootstrap happening as a result of a literal cargo build, we’ll likely never enable that. If you mean in terms of cargo actually being used to lazily compile libstd, that’s farther out and out of the scope of this thread unfortunately.

Would it be useful to have a face-to-face meeting about packaging at or around RustConf?

I’m always willing to chat about Cargo and Rust! Can at least say hi and see what’s up

We’ve got some fun vendoring-style features in the Cargo pipeline which I suspect will solve this issue as well. Note that I don’t want to derail this thread too much though as it’s centered around features in rustbuild. Would you mind opening up an issue to track the progress of this so we can continue the discussion there?

[cuviper]

My current plan is along the lines of: […]

As long as we keep some flexibility for us to say “We’re not ready yet!” then this sounds fine. I’m more than willing to pitch in for work needed to keep the makefiles alive too.

We’ve got some fun vendoring-style features in the Cargo pipeline which I suspect will solve this issue as well.

Generally distros (or at least Fedora) are not too keen on vendoring/bundling sources either. We’d prefer to have sources for a given project exist in just one place, for easier tracking. But tools that help enumerate dependencies would probably help the packaging process.

Note that I don’t want to derail this thread too much though as it’s centered around features in rustbuild. Would you mind opening up an issue to track the progress of this so we can continue the discussion there?

Sorry for that – I wanted to raise the concern about makefile deprecation, but I didn’t mean to take over the thread. Now that this is stated, I’ll try to review what issues are already open and see if we need more…

[Ericson2314]

If you mean from the perspective of the bootstrap happening as a result of a literal cargo build, we’ll likely never enable that.

I don’t want to get rid of rustbuild, but I think it’s very feasible that sans testing what rustbuild does essentially boils down to that.

Generally distros (or at least Fedora) are not too keen on vendoring/bundling sources either.

Yeah Vendoring is a huge anti-pattern. What I want for Nix is just dumping out some hashes for source. We can do everything automatically from that (and the original URL), but other package managers can package the source manually and use the hash to verify things were done correctly (at package build time).

---

If all the main distro’s get cleared up, can we accelerate the chucking of the Makefiles (e.g. 2 cycles: deprecate then delete)? IIUC @cuviper and others aren’t worried about cargo per say, but rather the presumed fallout like checking in more binary blobs and needing internet access at build time. I’m very interested in things which are only feasible to do once the makefiles are gone, and am thus willing to put into the time downstream to make that happen.

[petrochenkov]

What’s the equivalent of TESTNAME=my_test in rustbuild? (I’m trying to migrate to rustbuild now, but can’t find this part anywhere so far.)

[sanxiyn]

You can pass test filter as an argument. python ../src/boostrap/boostrap.py --step check-cfail --stage 1 where is equivalent to make check-stage1-cfail TESTNAME=where.

[petrochenkov]

Ok, question number two

How to mark a target as “done”? Suppose a run-pass target fails on my machine due to a rustc/test/whatever bug (e.g. this). With the old build system I can touch tmp/x86-64-blah-blah-blah-rpass.ok and then build remaining targets, what is the rustbuild equivalent? Spurious (a la “can’t write directory”) or persistent failures almost always happen on Windows, so this is a blocker.

Also, why are targets named “steps”? And how to know which targets depend on which? E.g. how do I know if check runs tidy or not? With makefiles it was obvious, with rustbuild the dependencies must be somewhere in the source code, but I haven’t read the bootstrap code carefully yet. (Also, what is the equivalent of make check-notidy?)

[sanxiyn]

How to mark a target as “done”?

This is a bug (or an unimplemented feature).

Issue: rustbuild: Don't re-run tests suites that already passed

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.