How to provide standard library sources for tools?

[matklad]

Hi! This is a sequel of What do you think about publishing `libstd` on crates.io?

Problem: IntelliJ Rust and Racer powered IDEs and editors all need source code of the standard library to provide completion and go to definition.

It’s not so easy to download standard library at the moment. First, you need to figure out the version of the compiler you are using. Then, you need to fetch the sources of standard library together with the sources of the compiler. And then you have to find the crates which constitute the standard library.

I see several ways of making this more robust:

1. RFC 1133 – this is the long term solution. But it will take a while to accept and implement this. Besides, it solves a much bigger problem of building the standard library. Perhaps we can get away with a more modest approach?
2. Download sources of the compiler/stdlib as part of rustup update (issue). This does not solve the problem of discovering stdlib structure and will become obsolete once RFC 1133 or similar is implemented.
3. Provide a build in cargo where_is_my_stdlib command, which will download the correct standard library version and present info about it in the JSON format, like cargo metadata. This solution will also pave the way for RFC 1133, because you’ll need to worry only about actually building stdlib, and not about fetching it. And perhaps we don’t want to download stdlib from crates.io because it has somewhat unusual versioning.
4. Like 3, but as an external command.

What do you think about options 3 and 4? Looks like they are can be implemented with a reasonably low amount of effort and don’t have huge backwards compatibility hazards.

cc @alexcrichton @phildawes @Ericson2314

[Ericson2314]

The challenges listed in What do you think about publishing `libstd` on crates.io? boil down to two things, neither of which are a big impediment for this:

1. Foreign deps. Thankfully, the IDEs don’t need to show the definitions of foreign definitions. So use the sysroot for the native libs—use rustup or whatever to get those—and then plain ole’ Cargo should be able to do the rest. Since rustc already will already happily find compiler-rt in the sysroot, I assume it will find the other native libs too today. Even if that is not true, it shouldn’t be much work to make it find them tomorrow.

2. Stable compilers. This is an artificial restriction, we just need to pick a backdoor to allow stable compilers to build just the stdlib but not other unstable code. Since there is exactly one 1 official stdlib per (stable) compiler this isn’t too difficult. Hardcode some hashes; add a #[trust_me_I_am_stdlib_I_will_be_maintained_forever] attribute—I don’t care. They all work.

I believe they’re fairly easy to the point where I am happy to work on these things to make them happen and prove it.

[Ericson2314]

Also, note for no 2 that stable interfaces can transitively depend on unstable interfaces—and we don’t want stable code directly depending on rustc_bitflags for example. But to my knowledge these creates must be distributed in stable compilers’ sysroots, so there is already a solution for that.

[steveklabnik]

The second one is already kind of happening, with the plan to bootstrap from STABLE - 1 rather than shapshots.

[Ericson2314]

Oh really?! What sort of white-list was chosen?

[matklad]

Are you saying that we almost can build standard library, so there is no need to provide “just download, but do not compile” functionality?

Do you think that standard library should live on crates.io? Will this work out for nightly versions of stdlib?

[Ericson2314]

In the latest version of my RFC, stdlib deps are desugared to git deps on the rust repo using the revision from rustc --version --verbose. I think this is easier in, and fine for, the short term.

[Ericson2314]

Are you saying that we almost can build standard library, so there is no need to provide “just download, but do not compile” functionality?

Yup! I think rustboot could become something close to 1) download bootstrap compiler 2) build native deps 3) invoke rustc once per phase 4) tests…

[steveklabnik]

It wasn’t a whitelist; the “secret” key that lets the stable compiler compile unstable code is now predictable, so that it can bootstrap itself from a previous stable release.

[Ericson2314]

Oh so such a key already exists?

[matklad]

Wait, does this mean that I can just put

[dependencies]
std = { git="https://github.com/rust-lang/rust" }

In some random Cargo.toml today, invoke cargo metadata and it will download necessary crates? That would be supper cool. I am actually trying this right now, and looks like cargo is stuck on the Updating git repository https://github.com/rust-lang/rust.git step, presumably because it is huge.

[Ericson2314]

https://github.com/RustOS-Fork-Holding-Ground/RustOS/blob/master/Cargo.toml#L11-L23 this is what I know works.

Yeah Cargo has a really hard time with it for reasons I don’t understand. I manually git fetch being careful to nuke/fake locks and things, and its much faster.

[Ericson2314]

Let me know how std goes. I’d be interested in getting a demo working if it just needs a little tweaking to find the native deps in the sysroot. And alternatively if it already works, this definitely needs to be mentioned in my RFC.

[steveklabnik]

Oh so such a key already exists?

Yeah, it always has. https://github.com/rust-lang/rust/pull/32942 is the change I was referring to.

[Ericson2314]

Oh, well problem solved.

[matklad]

Yeah Cargo has a really hard time with it for reasons I don’t understand. I manually git fetch being careful to nuke/fake locks and things, and its much faster.

Yeah, now Cargo has finished and metadata subcommand worked flawlessly. It is exactly what I need, thanks! But it indeed took ages and plenty of CPU. Will try to find out what’s going on.

As a preliminary result, the first

Updating git repository `https://github.com/rust-lang/rust.git`

step takes a lot of time as well as

INFO:cargo::sources::git::utils: update submodules for: "/home/matklad/.cargo/git/checkouts/rust-ca7715a8f56c0a4f/master/src/liblibc/"

[Ericson2314]

Hmm, still weird because manual git submodules --init is also a lot faster.

I’d hope to replace the hoedown submodule with https://crates.io/crates/hoedown, but it probably isn’t costing compared LLVM anyways.

Did you try building yet?

[matklad]

Did you try building yet?

No, because I don’t need to. All I am interested in is the source code

[matklad]

So looks like the problem indeed is simply that both rust and llvm are huge repositories, and that we clone them both when std = { git="https://github.com/rust-lang/rust" }. Will it be possible to separate compiler sources from standard library sources? Preferably, standard library should be downloadable from static.rust-lang.org. I think it is should be possible to implement the Source trait for one more source in Cargo.

[Ericson2314]

Shallow clone would probably be easier. Last I checked it wasn’t done because libgit did not support shallow clones.