How to provide standard library sources for tools?

[matklad]

@alexcrichton is there any simple way to make this VVVVV faster?

Wait, does this mean that I can just put

[dependencies] std = { git=“https://github.com/rust-lang/rust” }

In some random Cargo.toml today, invoke cargo metadata and it will download necessary crates? That would be supper cool. I am actually trying this right now, and looks like cargo is stuck on the Updating git repository https://github.com/rust-lang/rust.git step, presumably because it is huge.

[bruno_medeiros]

See also: https://github.com/rust-lang/rust/issues/19535 - Rust library sources are also useful for interactive debugging - but the most significant use case is indeed for code completion / go-to def, etc.

As for solutions, I don’t like option 2 just on its own - I don’t use rustup, so I would rather have the Rust installer (optionally) install the standard library sources. (of course for people that do use rustup, it should have that source download functionality as well). This seems to me to be the best option, short of option 1 itself, making Cargo aware of standard library dependencies.

As for std = { git="https://github.com/rust-lang/rust" }, how is that a proper solution? Won’t that just download the latest standard library sources on Git master? But those might not match the actual Rust installation being used.

[bruno_medeiros]

BTW, Oh, wow:

Seems like Cargo is consuming about 2Gb of memory.

[matklad]

As for std = { git=“https://github.com/rust-lang/rust” }, how is that a proper solution? Won’t that just download the latest standard library sources on Git master? But those might not match the actual Rust installation being used.

That is of course not a proper solution. But it is not that difficult to add necessary commit hash (and we already do this anyway). The real problem with this is the performance (which you have just discovered ) and the general high level of hackery

I would rather have the Rust installer (optionally) install the standard library sources. This seems to me to be the best option

It also has drawbacks. For one thing, if you have downloaded Rust without sources and now your IDE says that you need sources, how would you add them? Seems impossible without rustup. The second problem is that the standard library consists of the number of crates which depend on each other. And this dependency metadata is required to do resolve properly. { git = ... } hack provides this metadata and that’s why I like it.

[bruno_medeiros]

That is of course not a proper solution. But it is not that difficult to add necessary commit hash (and we already do this anyway). The real problem with this is the performance (which you have just discovered ) and the general high level of hackery

Still, even with a commit hash, how do you figure out which commit hash to put in? Say you have a Rust installation of the latest Stable compiler, and another one with Nightly, how do you find out the correct commit hash for each of those installations?

It also has drawbacks. For one thing, if you have downloaded Rust without sources and now your IDE says that you need sources, how would you add them? Seems impossible without rustup.

How do you add them? Erm, you download and extract the source package (https://www.rust-lang.org/downloads.html), then point your IDE to it?

The second problem is that the standard library consists of the number of crates which depend on each other. And this dependency metadata is required to do resolve properly. { git = … } hack provides this metadata and that’s why I like it.

What metadata exactly do you get out of that command? I wasn’t able to run it to completion as you see. I imagine it must be something IntelliJ-specific though, since Racer only cares about one single location, the src root of the standard library sources.

[matklad]

Still, even with a commit hash, how do you figure out which commit hash to put in? Say you have a Rust installation of the latest Stable compiler, and another one with Nightly, how do you find out the correct commit hash for each of those installations?

You can invoke rustc -vV and the commit hash will be there. Here is the parsing code of InteliJ Rust.

How do you add them? Erm, you download and extract the source package (https://www.rust-lang.org/downloads.html), then point your IDE to it?

Yep, currently IntelliJ just downloads the archive with sources (from github, because zip achieves are a bit easier to handle then tar.gz). The code is here. And this is not really pretty

What metadata exactly do you get out of that command?

We learn which which crates inside src/ dir belong to the standard library (we really don’t want to index the sources of the compiler). We also should learn about dependencies (i.e., you can use liballoc from libstd, but not from libcode), but this not currently implemented. The metadata for the stdlib looks something like this.

[matklad]

The metadata for the stdlib looks something like this.

Hm, I’ve just realized that in order to get that output, I’ve just cd src/libstd && cargo metadata. I guess this solves the metadata problem

[matklad]

Still, even with a commit hash, how do you figure out which commit hash to put in? Say you have a Rust installation of the latest Stable compiler, and another one with Nightly, how do you find out the correct commit hash for each of those installations?

Ah, sorry, I’ve misread the question. Generally the IDE should know which compiler is currently in use and be able to switch standard libraries accordingly. I think that the best way to do so is to provide a way to switch compilers from within IDE. You need this anyway to compile the code with the correct compiler.

[bruno_medeiros]

You can invoke rustc -vV and the commit hash will be there.

Ah, interesting.

Yep, currently IntelliJ just downloads the archive with sources (from github, because zip achieves are a bit easier to handle then tar.gz). The code is here. And this is not really pretty

Ok, why did you say it seemed impossible without rustup then?[quote=“matklad, post:26, topic:3490”] We learn which which crates inside src/ dir belong to the standard library (we really don’t want to index the sources of the compiler) [/quote] Ah, good point. Yeah I forgot the sources packages include sources of the compiler too. I can see why this would be a problem. Even for Racer, at some point this will be an issue.

[matklad]

Ok, why did you say it seemed impossible without rustup then?

If sources installed “optionally” with the usual, non rustup Rust installer, then you have to write some code in the IDE to get sources by issuing some HTTP requests, in case Rust is installed without them. But, if you need this code anyway, then what is the benefit of having sources sometimes installed from the beginning?

In contrast, in the world where everybody uses rustup, the IDE can just issue hypothetical rustup downloadStdlib command, without the need to know how sources are actually downloaded.

[bruno_medeiros]

If sources installed “optionally” with the usual, non rustup Rust installer, then you have to write some code in the IDE to get sources by issuing some HTTP requests

Hum, what? You don’t have to write such code in the IDE, the IDE can just ask the user to download the sources manually. It’s not as good as having the IDE do it, yes, but it’s an reasonable option.

In any case, even if the IDE is doing it automatically for the user, it still can be done without rustup. Maybe I misunderstood what you where you said? It looked like you were saying it’s impossible to download the sources for a Rust installation without using rustup …

[matklad]

Of course you can just ask the user to download sources, but this is a bit suboptimal. User should do rustc -vV to figure out what version of the stdlib does she need. And this have to be repeated once every 6 weeks at least. And the user can make mistakes.

The whole point of this thread for me is to try to figure out, how to avoid this manual boilerplate and just to do the right thing using cargo, rustup or some other rust tool which naturally knows how do deal with stdlib and without encoding knowledge about rust repository structure in the IDE. And this completely automagical solution would be impossible if you only have the option to specify “download standard library sources” during the initial installation of Rust.

Asking the user to attach sources or writing some code to download/extract/find stdlib is perfectly possible at the moment and everybody does this (although even this workflow can be improved by providing a pure stdlib download, without the compiler)

Sorry for causing some confusion

[brson]

Sorry I haven’t read through this thread, but my current plan is to add to the rust build system a new build artifact that packages the source up as a rust-installer package called rust-src; then add that package to the manifests produced by our CI; then teach rustup to install arbitrary components (whereas now it only knows how to install the component called ‘rust-std’ via the ‘rustup target’ command). The rust-src package would drop the sources into rustlib/src/rust probably.

If anybody wants to go ahead and do all this then that would be very welcome.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.