How to provide standard library sources for tools?

Ericson2314 · 2016-05-20T19:45:33.741Z

Oh so such a key already exists?

matklad · 2016-05-20T19:53:46.016Z

Wait, does this mean that I can just put

[dependencies]
std = { git="https://github.com/rust-lang/rust" }

In some random Cargo.toml today, invoke cargo metadata and it will download necessary crates? That would be supper cool. I am actually trying this right now, and looks like cargo is stuck on the Updating git repository https://github.com/rust-lang/rust.git step, presumably because it is huge.

Ericson2314 · 2016-05-20T19:56:45.128Z

https://github.com/RustOS-Fork-Holding-Ground/RustOS/blob/master/Cargo.toml#L11-L23 this is what I know works.

Yeah Cargo has a really hard time with it for reasons I don’t understand. I manually git fetch being careful to nuke/fake locks and things, and its much faster.

Ericson2314 · 2016-05-20T20:12:01.833Z

Let me know how std goes. I’d be interested in getting a demo working if it just needs a little tweaking to find the native deps in the sysroot. And alternatively if it already works, this definitely needs to be mentioned in my RFC.

steveklabnik · 2016-05-20T20:21:41.020Z

Ericson2314:

Oh so such a key already exists?

Yeah, it always has. https://github.com/rust-lang/rust/pull/32942 is the change I was referring to.

Ericson2314 · 2016-05-20T20:40:09.368Z

Oh, well problem solved.

matklad · 2016-05-20T20:50:35.307Z

Ericson2314:

Yeah Cargo has a really hard time with it for reasons I don’t understand. I manually git fetch being careful to nuke/fake locks and things, and its much faster.

Yeah, now Cargo has finished and metadata subcommand worked flawlessly. It is exactly what I need, thanks! But it indeed took ages and plenty of CPU. Will try to find out what’s going on.

As a preliminary result, the first

Updating git repository `https://github.com/rust-lang/rust.git`

step takes a lot of time as well as

INFO:cargo::sources::git::utils: update submodules for: "/home/matklad/.cargo/git/checkouts/rust-ca7715a8f56c0a4f/master/src/liblibc/"

Ericson2314 · 2016-05-20T21:19:16.319Z

Hmm, still weird because manual git submodules --init is also a lot faster.

I’d hope to replace the hoedown submodule with https://crates.io/crates/hoedown, but it probably isn’t costing compared LLVM anyways.

Did you try building yet?

matklad · 2016-05-20T21:46:59.797Z

Did you try building yet?

No, because I don’t need to. All I am interested in is the source code

matklad · 2016-05-20T22:48:42.344Z

So looks like the problem indeed is simply that both rust and llvm are huge repositories, and that we clone them both when std = { git="https://github.com/rust-lang/rust" }. Will it be possible to separate compiler sources from standard library sources? Preferably, standard library should be downloadable from static.rust-lang.org. I think it is should be possible to implement the Source trait for one more source in Cargo.

Ericson2314 · 2016-05-20T22:57:57.954Z

Shallow clone would probably be easier. Last I checked it wasn’t done because libgit did not support shallow clones.

matklad · 2016-05-23T20:01:42.110Z

@alexcrichton is there any simple way to make this VVVVV faster?

matklad:

Wait, does this mean that I can just put

[dependencies] std = { git=“https://github.com/rust-lang/rust” }

In some random Cargo.toml today, invoke cargo metadata and it will download necessary crates? That would be supper cool. I am actually trying this right now, and looks like cargo is stuck on the Updating git repository https://github.com/rust-lang/rust.git step, presumably because it is huge.

bruno_medeiros · 2016-05-26T11:25:54.723Z

See also: https://github.com/rust-lang/rust/issues/19535 - Rust library sources are also useful for interactive debugging - but the most significant use case is indeed for code completion / go-to def, etc.

As for solutions, I don’t like option 2 just on its own - I don’t use rustup, so I would rather have the Rust installer (optionally) install the standard library sources. (of course for people that do use rustup, it should have that source download functionality as well). This seems to me to be the best option, short of option 1 itself, making Cargo aware of standard library dependencies.

As for std = { git="https://github.com/rust-lang/rust" }, how is that a proper solution? Won’t that just download the latest standard library sources on Git master? But those might not match the actual Rust installation being used.

bruno_medeiros · 2016-05-26T11:52:05.892Z

BTW, Oh, wow:

Untitled.png866×363 21.1 KB

Seems like Cargo is consuming about 2Gb of memory.

matklad · 2016-05-26T11:54:42.918Z

bruno_medeiros:

As for std = { git=“https://github.com/rust-lang/rust” }, how is that a proper solution? Won’t that just download the latest standard library sources on Git master? But those might not match the actual Rust installation being used.

That is of course not a proper solution. But it is not that difficult to add necessary commit hash (and we already do this anyway). The real problem with this is the performance (which you have just discovered ) and the general high level of hackery

I would rather have the Rust installer (optionally) install the standard library sources. This seems to me to be the best option

It also has drawbacks. For one thing, if you have downloaded Rust without sources and now your IDE says that you need sources, how would you add them? Seems impossible without rustup. The second problem is that the standard library consists of the number of crates which depend on each other. And this dependency metadata is required to do resolve properly. { git = ... } hack provides this metadata and that’s why I like it.

bruno_medeiros · 2016-05-26T12:09:05.365Z

matklad:

That is of course not a proper solution. But it is not that difficult to add necessary commit hash (and we already do this anyway). The real problem with this is the performance (which you have just discovered ) and the general high level of hackery

Still, even with a commit hash, how do you figure out which commit hash to put in? Say you have a Rust installation of the latest Stable compiler, and another one with Nightly, how do you find out the correct commit hash for each of those installations?

matklad:

It also has drawbacks. For one thing, if you have downloaded Rust without sources and now your IDE says that you need sources, how would you add them? Seems impossible without rustup.

How do you add them? Erm, you download and extract the source package (https://www.rust-lang.org/downloads.html), then point your IDE to it?

matklad:

The second problem is that the standard library consists of the number of crates which depend on each other. And this dependency metadata is required to do resolve properly. { git = … } hack provides this metadata and that’s why I like it.

What metadata exactly do you get out of that command? I wasn’t able to run it to completion as you see. I imagine it must be something IntelliJ-specific though, since Racer only cares about one single location, the src root of the standard library sources.

matklad · 2016-05-26T12:29:08.194Z

bruno_medeiros:

Still, even with a commit hash, how do you figure out which commit hash to put in? Say you have a Rust installation of the latest Stable compiler, and another one with Nightly, how do you find out the correct commit hash for each of those installations?

You can invoke rustc -vV and the commit hash will be there. Here is the parsing code of InteliJ Rust.

How do you add them? Erm, you download and extract the source package (https://www.rust-lang.org/downloads.html), then point your IDE to it?

Yep, currently IntelliJ just downloads the archive with sources (from github, because zip achieves are a bit easier to handle then tar.gz). The code is here. And this is not really pretty

What metadata exactly do you get out of that command?

We learn which which crates inside src/ dir belong to the standard library (we really don’t want to index the sources of the compiler). We also should learn about dependencies (i.e., you can use liballoc from libstd, but not from libcode), but this not currently implemented. The metadata for the stdlib looks something like this.

matklad · 2016-05-26T12:33:09.568Z

matklad:

The metadata for the stdlib looks something like this.

Hm, I’ve just realized that in order to get that output, I’ve just cd src/libstd && cargo metadata. I guess this solves the metadata problem

matklad · 2016-05-26T12:45:38.291Z

matklad:

Still, even with a commit hash, how do you figure out which commit hash to put in? Say you have a Rust installation of the latest Stable compiler, and another one with Nightly, how do you find out the correct commit hash for each of those installations?

Ah, sorry, I’ve misread the question. Generally the IDE should know which compiler is currently in use and be able to switch standard libraries accordingly. I think that the best way to do so is to provide a way to switch compilers from within IDE. You need this anyway to compile the code with the correct compiler.

bruno_medeiros · 2016-05-26T13:02:49.872Z

matklad:

You can invoke rustc -vV and the commit hash will be there.

Ah, interesting.

matklad:

Yep, currently IntelliJ just downloads the archive with sources (from github, because zip achieves are a bit easier to handle then tar.gz). The code is here. And this is not really pretty

Ok, why did you say it seemed impossible without rustup then?[quote=“matklad, post:26, topic:3490”] We learn which which crates inside src/ dir belong to the standard library (we really don’t want to index the sources of the compiler) [/quote] Ah, good point. Yeah I forgot the sources packages include sources of the compiler too. I can see why this would be a problem. Even for Racer, at some point this will be an issue.