Not sure if this is the right place to ask this but, is there a guide somewhere for how to setup making changes to the standard library locally? The rustc guide seems to be taking the perspective that you want to make changes to the compiler while avoiding rebuilding the standard library but that's actually the opposite of what I want. If possible I would prefer to rebuild the standard library without touching the compiler (I want to experiment with adding some new trait implementations). I know for regular crates I can put a local path in my Cargo.toml, is it possible to do that for std?
2 Likes
All the tools described in that guide still work.
To build your changes to std: ./x.py --keep-stage=0 --stage=1 library/std
You can use your modified std with rustup toolchain link.
Alternatively, you can use something like xargo, then you don't have to build the stage0 compiler. But configuring it correctly can be painful.
2 Likes
Yep, everything @jethrogb said, and this section may be helpful: https://rustc-dev-guide.rust-lang.org/getting-started.html#building-and-testing-stdcorealloctestproc_macroetc
1 Like
Would it be worth having a separate page specifically for contributing to the std (or even separate docs)? I understand that technically it's very similar to contributing to rustc (in the same tree, uses the same tooling, etc) but the workflow feels very different to me. Or at least it feels like a substantially different task.
Contributing to the standard library is, I think, almost (but not quite) like contributing to any crate, except you have to checkout rustc as well and run a funny looking command instead of using cargo.
Contributing to the compiler can be much more involved, hence the complexity of the docs and the focus on rustc.
I feel there should at least be a top level page for std contributing that shortcuts all the irrelevant information and makes it easier to get started.
2 Likes
Seems obvious to me that there should be a minimal, straightforward description of how to contribute to std while ignoring the compiler as much as possible. Surely people potentially inclined to contribute to std greatly outnumber those prepared to dive into rustc?
The Rust team's stated philosophy is to minimize what's in std, because anything included in std is held forever at SemVer 1.0 and thus becomes largely uncorrectable and unimprovable (viz. hashmap vs hashbrown). Thus any proposed contribution to std
-
needs to start with a lengthy justification of why the proposed contribution needs to be included in
stdrather than be offered as a separate crate where future improvements via SemVer would be possible; and -
needs to reference an existing crate that proves the contribution is feasible and demonstrates that the proposed API has been well designed and is already well tested.
If the proposed contribution needs to be in std because it requires special compiler support, that fact needs to be stated very clearly, along with a sketch of the needed compiler changes and an assessment of the effort required to make and maintain those changes.
.
Which is just the sort of thing that could go into a "std guide".
However I suspect that most potential contributions are about fixing or improving things already in the standard library rather than adding anything particularly new. Personally I do think the std is too large but that ship has sailed. Fixing what's there is important (e.g. there are a lot of open issues) so it's helpful to make contributing fixes as easy as possible.
4 Likes
That was my intent in summarizing those criteria that I see as most informative to someone new to Rust who proposes expanding Rust's "standard" library. Such a guide should also discuss how Rust, with its crates.io ecosystem, differs by design from languages with large standard libraries.
That won't work for the initial build. And anyway since it's only about the standard library, why not ./x.py --stage=0 library/std? That's what I usually use.
2 Likes
Does that give you a toolchain that will link the custom std in artifacts? I feel like I've tried this in the past, but it didn't seem to work.
Not on its own, no... though that should definitely be possible.
However, I have also never needed such a toolchain. You can run doctests and the library test suite (./x.py test --stage 0 library/std) to make sure the code makes sense. Add --test-args FILTER to not run all of it, just the tests with FILTER in their name.
2 Likes
I'm adding a new subcommand for x.py which sets up the defaults so that hopefully all the commands in the dev guide will 'just work': https://github.com/rust-lang/rust/pull/76631
It would also be good to have a guide for working on the standard library, but I think not rebuilding rustc by default should get 80% of the way there.
3 Likes
I basically exclusively make PRs to the Rust repo to contribute to std, and I don't do so frequently. In march I wrote this post about my experience: Experience report contributing to rust-lang/rust Having made another small patch this week, my experience largely remains the same.
I wish it was much easier to contribute to std than it is. I do not contribute frequently enough that it is ever worthwhile learn the incantations the build system needs to build only std. As a result, I just don't build locally before making a PR (if my change is complicated, I prototype it in a separate crate and make sure that builds). Then I wackamoll build failures resulting from silly things like having too many newlines in my diff because the project blocks CI on a very stringent tidy check. This wastes time and our donated CI resources, but there's no process that doesn't waste substantially more of my time, which I am donating as well.
6 Likes
I thought formatting was automated now?
Formatting is automatically checked in CI, and you can also run the formatter on your machine. But there's nothing that will go in and modify your commits, so it's still not "automated" enough to ignore.
Figuring out the necessary invocation and writing it somewhere for later takes less time than running CI once. Someone can figure out out the invocation for you, if you think it's unworthy to do it by yourself.
After that the time is only spent on actually rebuilding the std and running tests (the useful payload) + short-running tidy.
I had similar problems with tidy. I wanted to literally change a couple of words in an error message, but it happened to push a line length over an arbitrary limit. I thought it'd be a 5-minute change, but it has turned into whole day wasted on fighting with CI and linter directives.
When I complained about how unfriendly tidy in CI is, I only got "shrug" like response, as if knowing special directives of custom linting tools was a normal thing I should have known.
The whole experience was very demotivating.
8 Likes
Formatting is automatically checked in CI, and you can also run the formatter on your machine. But there's nothing that will go in and modify your commits, so it's still not "automated" enough to ignore.
There's an issue open for this: auto-bless tests/formatting on rust-lang/rust · Issue #760 · rust-lang/triagebot · GitHub
Someone can figure out out the invocation for you
This is what we should document, I think. Currently, it's x.py test --stage 0 library/std; after Add `x.py setup` by jyn514 · Pull Request #76631 · rust-lang/rust · GitHub it will be x.py test library/std. I've also been considering a default filter for x.py which could be helpful (it would allow turning that into just x.py test).
Then I wackamoll build failures resulting from silly things like having too many newlines in my diff because the project blocks CI on a very stringent tidy check. This wastes time and our donated CI resources, but there's no process that doesn't waste substantially more of my time, which I am donating as well.
x.py test tidy would show only the whack-a-mole issues without having to build the whole compiler. I agree a lot of this needs to be better documented.
I wanted to literally change a couple of words in an error message, but it happened to push a line length over an arbitrary limit. I thought it'd be a 5-minute change, but it has turned into whole day wasted on fighting with CI and linter directives.
Do you remember which PR that was? That sounds really frustrating, but I wouldn't expect linting to take so long to fix.
(from the other thread) One thing that would make my experience a lot better would be an easy way to build and check libstd with my system rustc.
You can do this currently: cargo check -p test. cargo test -p test does not work, I'm not sure why although @Mark_Simulacrum might have ideas.
(from the other thread) Ultimately I managed to make my contribution without doing a full bootstrap, but even git operations took unreasonably long. My initial clone took around 2 minutes (the repo is now nearly 500 MB), instantiating submodules took 10.
Download LLVM from CI to bootstrap (linux-only to start) by Mark-Simulacrum · Pull Request #76349 · rust-lang/rust · GitHub and Don't download/sync llvm-project submodule if download-ci-llvm is set by est31 · Pull Request #76864 · rust-lang/rust · GitHub might help with this (although there's been a lot of trouble getting them to work anywhere but linux: Don't dynamically link LLVM tools unless rustc is too by Mark-Simulacrum · Pull Request #76810 · rust-lang/rust · GitHub).
It makes me really sad that rust is so hard to contribute to even for core language developers 
If there's other improvements that could happen I'm happy to hear them. I think a lot of the current issues could be addressed by either more documentation or some of the PRs in flight.
As an aside, I don't think this is a good attitude to take with people actively trying to contribute and failing. rust-lang/rust being hard to work with is not the fault of the people trying to work with it.
6 Likes
Would it be worth having a separate page specifically for contributing to the std (or even separate docs)? I understand that technically it's very similar to contributing to
rustc(in the same tree, uses the same tooling, etc) but the workflow feels very different to me. Or at least it feels like a substantially different task.
There is existing documentation, but it's a little hidden: Document that you don't need to build the compiler to work on libraries · Issue #873 · rust-lang/rustc-dev-guide · GitHub