How to provide standard library sources for tools?

[Ericson2314]

Oh really?! What sort of white-list was chosen?

[matklad]

Are you saying that we almost can build standard library, so there is no need to provide “just download, but do not compile” functionality?

Do you think that standard library should live on crates.io? Will this work out for nightly versions of stdlib?

[Ericson2314]

In the latest version of my RFC, stdlib deps are desugared to git deps on the rust repo using the revision from rustc --version --verbose. I think this is easier in, and fine for, the short term.

[Ericson2314]

Are you saying that we almost can build standard library, so there is no need to provide “just download, but do not compile” functionality?

Yup! I think rustboot could become something close to 1) download bootstrap compiler 2) build native deps 3) invoke rustc once per phase 4) tests…

[steveklabnik]

It wasn’t a whitelist; the “secret” key that lets the stable compiler compile unstable code is now predictable, so that it can bootstrap itself from a previous stable release.

[Ericson2314]

Oh so such a key already exists?

[matklad]

Wait, does this mean that I can just put

[dependencies]
std = { git="https://github.com/rust-lang/rust" }

In some random Cargo.toml today, invoke cargo metadata and it will download necessary crates? That would be supper cool. I am actually trying this right now, and looks like cargo is stuck on the Updating git repository https://github.com/rust-lang/rust.git step, presumably because it is huge.

[Ericson2314]

https://github.com/RustOS-Fork-Holding-Ground/RustOS/blob/master/Cargo.toml#L11-L23 this is what I know works.

Yeah Cargo has a really hard time with it for reasons I don’t understand. I manually git fetch being careful to nuke/fake locks and things, and its much faster.

[Ericson2314]

Let me know how std goes. I’d be interested in getting a demo working if it just needs a little tweaking to find the native deps in the sysroot. And alternatively if it already works, this definitely needs to be mentioned in my RFC.

[steveklabnik]

Oh so such a key already exists?

Yeah, it always has. https://github.com/rust-lang/rust/pull/32942 is the change I was referring to.

[Ericson2314]

Oh, well problem solved.

[matklad]

Yeah Cargo has a really hard time with it for reasons I don’t understand. I manually git fetch being careful to nuke/fake locks and things, and its much faster.

Yeah, now Cargo has finished and metadata subcommand worked flawlessly. It is exactly what I need, thanks! But it indeed took ages and plenty of CPU. Will try to find out what’s going on.

As a preliminary result, the first

Updating git repository `https://github.com/rust-lang/rust.git`

step takes a lot of time as well as

INFO:cargo::sources::git::utils: update submodules for: "/home/matklad/.cargo/git/checkouts/rust-ca7715a8f56c0a4f/master/src/liblibc/"

[Ericson2314]

Hmm, still weird because manual git submodules --init is also a lot faster.

I’d hope to replace the hoedown submodule with https://crates.io/crates/hoedown, but it probably isn’t costing compared LLVM anyways.

Did you try building yet?

[matklad]

Did you try building yet?

No, because I don’t need to. All I am interested in is the source code

[matklad]

So looks like the problem indeed is simply that both rust and llvm are huge repositories, and that we clone them both when std = { git="https://github.com/rust-lang/rust" }. Will it be possible to separate compiler sources from standard library sources? Preferably, standard library should be downloadable from static.rust-lang.org. I think it is should be possible to implement the Source trait for one more source in Cargo.

[Ericson2314]

Shallow clone would probably be easier. Last I checked it wasn’t done because libgit did not support shallow clones.

[matklad]

@alexcrichton is there any simple way to make this VVVVV faster?

Wait, does this mean that I can just put

[dependencies] std = { git=“https://github.com/rust-lang/rust” }

In some random Cargo.toml today, invoke cargo metadata and it will download necessary crates? That would be supper cool. I am actually trying this right now, and looks like cargo is stuck on the Updating git repository https://github.com/rust-lang/rust.git step, presumably because it is huge.

[bruno_medeiros]

See also: https://github.com/rust-lang/rust/issues/19535 - Rust library sources are also useful for interactive debugging - but the most significant use case is indeed for code completion / go-to def, etc.

As for solutions, I don’t like option 2 just on its own - I don’t use rustup, so I would rather have the Rust installer (optionally) install the standard library sources. (of course for people that do use rustup, it should have that source download functionality as well). This seems to me to be the best option, short of option 1 itself, making Cargo aware of standard library dependencies.

As for std = { git="https://github.com/rust-lang/rust" }, how is that a proper solution? Won’t that just download the latest standard library sources on Git master? But those might not match the actual Rust installation being used.

[bruno_medeiros]

BTW, Oh, wow:

Seems like Cargo is consuming about 2Gb of memory.

[matklad]

As for std = { git=“https://github.com/rust-lang/rust” }, how is that a proper solution? Won’t that just download the latest standard library sources on Git master? But those might not match the actual Rust installation being used.

That is of course not a proper solution. But it is not that difficult to add necessary commit hash (and we already do this anyway). The real problem with this is the performance (which you have just discovered ) and the general high level of hackery

I would rather have the Rust installer (optionally) install the standard library sources. This seems to me to be the best option

It also has drawbacks. For one thing, if you have downloaded Rust without sources and now your IDE says that you need sources, how would you add them? Seems impossible without rustup. The second problem is that the standard library consists of the number of crates which depend on each other. And this dependency metadata is required to do resolve properly. { git = ... } hack provides this metadata and that’s why I like it.