Beta testing rustup.rs

[inejge]

I’ve been experimenting with a setup that could make distro rust and rustup peacefully coexist, but leave full configurability to the user, up to and including running their own rustup installation. It involves installing another set of proxy executables in the system path and moving rust and rustup out of the way, but so that the proxies know about them. In more detail:

• The system rust is in /usr/lib/rust, exactly as it would’ve been installed by rustup (it has bin/, lib/ etc.) This could be further subdivided with separate named versions and placed under control of an alternatives scheme, I didn’t try to go that far;

• The system rustup executables are in /usr/lib/rustup or /usr/libexec/rustup, whatever the distro convention is;

• The proxies are in /usr/bin. A Bourne shell prototype of a proxy is available. The same script can be used as rustup, rustc, cargo, and rustdoc by naming it correspondingly. (I haven’t tested debugger invocation.)

The proxies would be in a separate package, and the system rust and rustup packages would depend on it. A proxy checks if there is a valid rustup toolchain directory in $HOME/.multirust or $RUSTUP_HOME. If there is, and the system rustup is installed, the appropriate rustup proxy is invoked (modulo setting USE_SYSTEM_RUST, which requests the use of distro rust.) To use rustup after installing it, the user must first run rustup init, which links the system rust as the toolchain named system and sets it as default.

Note that this a) doesn’t require the modification of $PATH, b) always results in a working configuration, even between installing rustup and initializing it. The need for a separate initialization step is a bit unfortunate, but IMO tolerable.

Frankly, I don’t think rustup belongs in distro packaging.

It’s true that rustup and the distro package manager are a kind of competition, but with the Rust development model in mind, I don’t think it would be realistic to expect a distribution to package all artifacts available to a developer. Rustup fills that niche (and then some!), and I’m reasonably convinced that it can be made to play nicely with what the distro provides.

[jonhoo]

The proxies would be in a separate package, and the system rust and rustup packages would depend on it.

Unfortunately this still doesn’t solve the versioning issue mentioned further above. Specifically, if the user has rustup installed, but hasn’t run rustup update in a while, packages which depend on a newer version of rust will fail to build. And the version dependency cannot be expressed in any way.

[oconnor663]

The idea would be to use the system package manager to install cargo, and cargo to install rustup. This works well if the goal is “never interfere with the version of rustc/cargo that the system provides, but make nightly rust available if you want to invoke it explicitly.” That would be weirder for folks who want cargo to be rustup’s version in their PATH, though, like you pointed out.

[inejge]

Specifically, if the user has rustup installed, but hasn’t run rustup update in a while, packages which depend on a newer version of rust will fail to build. And the version dependency cannot be expressed in any way.

The scheme I’ve described can override rustup and require the use of a system-provided compiler (a dependency that can be expressed), by setting an environment variable, so the state of rustup’s toolchain collection shouldn’t be relevant. (I’m assuming that packages can only meaningfully depend on the system compiler.) Or am I missing something else?

[jonhoo]

Or am I missing something else?

Well, I’d like for the rustup package to “provide” rust, so that if a package requires rust, it is sufficient for rustup to be installed. that way I don’t have to install both rustup and rust, keeping two copies of rust on my system

[brson]

Frankly, I don’t think rustup belongs in distro packaging.

While I tend to agree that there would at least be less friction if people just installed it from us, distros are going to package it - it’s what they do! So I’m inclined to do what I can to make it a decent experience.

Will rustup use a version of rustc/cargo newer than RUSTUP_TOOLCHAIN if one is installed?

It will not. Though I can imagine why one would want that, could you say more about why that is desirable?

No, packages in Arch are fairly often built in an ad-hoc manner as well. Packages that are released in binary form are built through a single, automated system. However, user-contributed packages are built by the users by downloading a tarball with a PKGBUILD file in it, and then running makepkg in the extracted directory. Users also have the ability to do the same for official distributed packages by downloading their PKGBUILD files.

Is downloading and building in an arbitrary location the common way to build both user-contributed and official packages from source? IOW there’s no single command that just does both? Is there any automatic mechanism for enforcing build deps, that is, that the rustc package is installed?

Perhaps ccache is a useful example to follow. On Fedora, it installs all of its conflicting binaries in %{_libdir}/ccache/, then comes with /etc/profile.d/ scripts to put that ahead in the PATH. I could see a rustup package doing something similar, along with a default configuration pointing to the normal /usr toolchain.

Does the /etc/profile.d/ script take effect automatically? I agree that this looks promising.

Was it a deliberate decision not to put a package on crates.io? If it’s to avoid confusion with the standard install flow, maybe we could call the package something like “rustup-minimal” or “rustup-not-the-standard-install-flow-we-swear”?

Publishing the binary isn’t something I’ve considered carefully, but it does result in no proxies, which are a core part of the design, and a scenario I haven’t put any consideration into until now. One could imagine being able to bootstrap the rustup install via the system cargo, then having an option to later generate the proxies. Though off the top of my head I’d prefer in this situation for the proxies to still exist and to default to the system toolchain.

[oconnor663]

Is downloading and building in an arbitrary location the common way to build both user-contributed and official packages from source? IOW there’s no single command that just does both? Is there any automatic mechanism for enforcing build deps, that is, that the rustc package is installed?

The makepkg command is part of the pacman suite of tools, so it knows about dependencies. By default it’ll refuse to build if dependencies aren’t present, though there are flags to either skip that check or (using other parts of pacman) actually install the deps first.

[cuviper]

Does the /etc/profile.d/ script take effect automatically? I agree that this looks promising.

Yes, for new login shells:

• /etc/profile automatically loads all /etc/profile.d/*.sh
• /etc/csh.login automatically loads all /etc/profile.d/*.csh
• other shells may have their own, like /etc/fish/conf.d/

[inejge]

Well, I’d like for the rustup package to “provide” rust, so that if a package requires rust, it is sufficient for rustup to be installed.

That dependency config could indeed fail in a way you’ve described, but what I had in mind is different:

• The compiler is provided by the rust package, which depends on rust-system-proxies.

• The rustup package depends on rust.

• When rustup is first installed, the proxies still execute the system rust binaries. Any attempt to use rustup tells the user to run rustup init.

• After rustup init, the system rust is linked to the system toolchain, which is set to be the default, so there’s still only one copy of the compiler, the system one, and it’s used automatically.

• Only after the user installs another toolchain and sets it as default, running bare rustc will invoke the non-system compiler. The use of the system-provided tools can be requested by setting USE_SYSTEM_RUST.

that way I don’t have to install both rustup and rust, keeping two copies of rust on my system

You wouldn’t have two copies unless you explicitly install another.

[matklad]

ls $(rustc --print-sysroot)/lib/rustlib/src/rust

First, correction: it should be --print sysroot, without the third -

And a question/suggestion: can I hope that I find sources there even if rustup is not used to manage rust installation? It would be really nice if say, distros or the windows installer had an option to install rust sources, and that the sources would be paced to sysroot/lib/rustlib/src/rust.

[brson]

And a question/suggestion: can I hope that I find sources there even if rustup is not used to manage rust installation? It would be really nice if say, distros or the windows installer had an option to install rust sources, and that the sources would be paced to sysroot/lib/rustlib/src/rust.

It could be added as an option to the windows installer, though rustup will eventually basically become the windows installer. For distros, I don’t have any specific knowledge about their plans, but I consider this arrangement of the standard library source code to effectively be part of the broader Rust platform definition, and would strongly encourage them to support it in some way. sysroot/lib/rustlib/src is the canonical location to find the source to the standard library.

This reminds me that wouldn’t it be great to have rust packaging guidelines.

[kornel]

At company where I worked we had system administrators who insisted on using only packages from the distro. Getting an RPM installed was easy (especially if it was outdated enough to be blessed by RedHat ;)). Using curl | sh for something in production has been intentionally made very difficult both politically and technically.

I think there’s a huge value in distros packaging Rust — it adds to its legitimacy, and reduces friction for adoption even in “Serious Business” environment.

[kornel]

I use distro-packaged rustc for “stable” (deploying things to production), and if I need to test something on beta/nightly I use rustup (only in dev).

Having rustup play nicely with distro-packaged rustc is helpful, because sometimes I want to try beta/nightly on VMs that have distro-packaged rustc installed already.

Changing PATH is fine (it’s a minor annoyance, because rustup only modifies .profile, so I have to manually copy it to .bashrc).

I would never use rustup to switch versions of distro’s rustc for use in production. Distros have appropriately versioned packages (e.g. look at PHP — you have both minor versions and families of packages for different major versions), and people using distros already have to have solutions for managing versions, updates and deployments. Having rustup in the mix, doing versioning and updates differently, would only make things harder.

[notriddle]

Honestly, I have to agree with @kornel. The distro’s package building scripts should use the distro’s compiler and avoid using anything installed by rustup (rustup, in turn, should avoid putting its stuff where the package manager will find it, or being marked as providing a copy of the rust compiler). That way, everybody’s package managers don’t need to learn how to interface with it, or it with them.

[dns2utf8]

I do not agree, because it would require an alias to use rustup then.

In my opinion the best would be if there were a package for rustup in the distro. Then the two packages were able to conflict each other.

[hsivonen]

Ideally, rustup would not add rustc and cargo to the path but instead have its toolchains be accessible through rustup itself (rustup rustc, rustup cargo) or through different commands as rustc and cargo. This clearly indicates we are not using the system rustc or cargo but rustup’s rustc or cargo instead.

I think this is backwards in terms of ergonomics. The user who is developing Rust code should be able to type cargo without anything extra and get the fresh Cargo from rustup.rs backed by the fresh rustc from rustup.rs.

It should be the distro’s responsibility to make sure that building its packages from source using its package manager use the Cargo/rustc version that the distro wants to use. It doesn’t matter if those invocations are prefixed or postfixed, since it won’t be the user typing the invocations directly.

[brson]

rustup 0.6.4 is out. Upgrade with rustup self update.

It’s been a while. This is mostly bug fixes, but it does come with one goodie: rustup can generate shell completions for bash, zsh, and fish (thanks @kbknapp).

This should also fix the occasional issues with checksum failures on updates. With that bug out of the way I’m going to focus on releasing rustup 1.0 and making rustup the primary installation mechanism for Rust. This will mostly entail bumping the version number and updating the main website’s download page, but I’ll also probably do a once-over of the issue tracker and see if there’s any last-minute polish to be done.

I do think there’s a whole lot of work left to do to make rustup really great, but it’s about time to bite the bullet and get a major release out. After 1.0, future work will involve getting back to “NDK” support, signing, bugfixing, and packaging of additional tools. In the future I would like to do another pass at the CLI design - I think the basic UI could be cleaner still, the commands more predictable.

0.6.4

• making rustup prepend cargo bin to path instead of append
• Use released version of rustls dependency
• Update OpenSSL
• Made outputting of ANSI terminal escapes codes defensive
• Adjusted rustup-init.sh need_cmd to add uname and remove printf
• Update to error-chain 0.5.0 to allow optional backtrace
• Fix variable naming in rustup-init.sh
• Update clap to fix --help formatting
• Add an FAQ entry about troubles with antivirus
• Clarify how rustup toolchain installation works on Windows
• Do not interpret commas when using “rustup run”
• Fix local declarations for zsh completions
• Fix checksum failures
• Treat an empty CARGO_HOME the same as an unset CARGO_HOME
• Check stdout is a tty before using terminal features
• Add completion generation for zsh, bash and fish shells

Contributors: Alex Crichton, Andrew Koroluk, Brian Anderson, Chungmin Park, Diggory Blake, Guillaume Fraux, Jake Goldsborough, jethrogb, Kamal Marhubi, Kevin K, Kevin Rauwolf, Raphael Cohn, Ricardo Martins

[brson]

rustup 0.6.5 is out. Upgrade with rustup self update.

This release includes a new version of curl, which fixes a number of security issues. Please upgrade.

0.6.5

• Update bundled curl code
• Remove old zsh completions
• Fix two small typos in the error descriptions
• Update README
• Fix name of bash completion directory

Contributors: Alex Crichton, Björn Steinbrink, Brian Anderson, Jian Zeng, Matt Brubeck

[brson]

rustup 0.7.0 is out. Run rustup self update to update.

This release is mostly bugfixes and preperation for rustup 1.0. The big change here is that I’ve finally moved the ~/.multirust directory to ~/.rustup. It should happen transparently, but because of the history of rustup.sh and multirust.sh there’s a lot of branchy special-cases. Let me know if anything blows up. For now ~/.multirust remains a symlink to ~/.rustup for the sake of any tools that expect it to exist, but I don’t intend to leave that forever.

@xenOn has been doing a lot of the difficult work to make rustup support mips. Thanks @xenOn.

If all goes well, rustup will be 1.0 in time for the Rust 1.14 release next week.

0.7.0

• Correctly “detect” host endianness on MIPS
• Add powershell completions
• Update toolchain used to build rustup
• Support probing MIPS64 n64 targets
• Support MIPS architectures in rustup-init.sh
• Automatically detect NetBSD during standard install
• Fix symlink creation on windows
• Search PATH for binaries run by rustup run
• Recursive tool invocations should invoke the proxy, not the tool directly
• Upgrade error-chain
• Add FAQ entry for downloading Rust source
• Rename ~/.multirust to ~/.rustup
• Remove some codegen hacks
• Update libc for MIPS64 host builds
• Default to MSVC on Windows

Contributors: Alex Crichton, Arch, bors, Brian Anderson, Diggory Blake, Kai Roßwag, Kevin K, Oliver Schneider, Ryan Havar, Tobias Bucher, Wang Xuerui

[josh]

To make distribution packaging easier, could rustup automatically install its proxy binaries to ~/.local/bin, which many distributions automatically put in $PATH if it exists? That would let /usr/bin/rustc and /usr/bin/cargo refer to the distro-installed versions (for use by distro packages and users who don’t use rustup), while any user who uses rustup will have the proxies in their $PATH ahead of /usr/bin.

rustup as packaged in a Linux distribution should definitely never have its proxies installed system-wide, where a distribution Rust package could accidentally build with them. Those should stay in a user directory, which the user can then have on their $PATH.

(Also, at least in Debian, we definitely plan to package rustup, because it provides the simplest way for users to test nightly or to install a cross-compiler toolchain for an arbitrary target platform.)