Beta testing rustup.rs

#158

Since non-Arch users may have ideas about the solutions to packaging rustup in distros I’m producing my post there here as well.

Hi. I’d like for rustup to work cleanly when packaged by distros, but haven’t put much effort or thought into it. There are a variety of concerns expressed here and I’m not familiar with the details of arch’s rustup packaging, so let me just brain-dump, and feel free to correct me and respond liberally.

Issues with rustup distro packaging

  • Where does rustup get installed to? Stock rustup wants everything in ~/.cargo/bin, including the rustup bin and its proxies. Distros may be inclined to put them in /usr/bin, and this is probably the correct thing to do from their perspective. I don’t see an obvious reason that wouldn’t work for rustup, but not so for rustc/cargo…

  • If rustup proxies are installed to /usr/bin how can they co-exist with distro-installed rustc/cargo? Both bins want to be installed there.

  • Even if the rustup proxies and the rustc/cargo bins can coexist on disk, how can somebody live in the rustp world while still using the system-installed toolchain? I think it makes sense to reserve a “system” toolchain name in rustup for this purpose but haven’t thought it through.

  • rustup self-updates are incompatible with distro packaging. For this we already have the “no-self-update” feature. Is there any more to be considered here?

  • rustup-init installs some toolchain by default. It is generally desirable to be able to install rustup without installing a toolchain. Having “system” be a valid toolchain would be one way around this limitation, in lieu of having a way to say “don’t install a toolchain”. It looks to me like this is not an issue for distros since their packaging just drops rustup and its proxies directly where it wants without running the rustup installer (basic rustup install is purposely simple like this).

  • Interaction between packaged Rust software that expects particular versions of rustc and rustup is bad. If Rust software wants to depend on rustc 1.10 it can do this traditionally via the package manager, but there is no way to express this if rustup is controlling the rustc version. Tricky issue.

  • The above case is most obviously manifested when building packages from source written in Rust. These expect to be built by the system’s toolchain. There may be ways to signal to rustup that these compilation scenarios should automatically use the “system” toolchain. e.g. if the source of these packages is always built in a specific subtree, there could be an override set on that subtree that tells rustup to use the "system" toolchain. Some complications here in the current design since you can’t just drop a configuration file into the tree itself to configure that like with rbenv.

  • rustup doesn’t have an option to install toolchains globally. Desirable feature, but needs careful design, haven’t put much effort into it.

Thoughts

To the suggestion of installing the rustup proxies as /usr/bin/{cargo,rustc}-rustup. This poses several difficulties, that I’m inclined not to do it. First, usability-wise it means that rustup users need to know to type cargo-rustup when working in Rust. This is pretty counter to rustup being a transparent drop-in for cargo; and I expect that if you install rustup it’s because you want to be in a rustup-world. Second, tools (primarily cargo) expect to be able to invoke rustc, etc. as ‘rustc’, and in rustup-world that rustc should be the rustc proxy. There are potential solutions to this, but I’d rather avoid.

There are some problems where it’s tempting for rustup and the system package manager to communicate about the active toolchain. I’d like to avoid this if at all possible. Very complex.

The idea to install rustup without its proxies is doable, but rustup would be quite crippled. It does break the primary functionality.

There will likely be situations in the future where rustup wants to create and delete proxy bins dynamically. This could change the calculus of the issues here in ways I’m not sure about yet, but istm the problems are not major - rustup will just create those proxies in ~/.cargo/bin as if it was doing cargo install.

I like the idea of a “system” toolchain where rustup can search the PATH for a copy of the tool it isn’t managing and defer to that, but the obvious problem is that both the rustup package and the rust package want their bins to be installed at the exact same place. I think if that issue is solved there is some path forward for solving the others.

Is it possible for the rustup package to modify the system-wide PATH variable? One solution would be to let the “normal” rust packages have the sweet /usr/bin location on the file system, and just put rustup somewhere else, with PATH precedence. Then let rustup defer to the system rustc via the rustup "system" toolchain, have rustup modify the override for the system package build directory to use the “system” toolchain during its install.

#159

Just a quick idea. What if you had the following, when installing both rustup and rust-X.Y (system rust) in parallel?

/usr/bin/rustup - rustup binary
/usr/bin/rustc - rustup rustc proxy
/usr/bin/rustc-X.Y - system rustc
/usr/bin/cargo - rustup cargo proxy
/usr/bin/cargo-X.Y - system cargo

In this case, you could have rustup use the system binaries as the defaults if they are installed and if they aren’t, you could default to an install with no toolchains. If you do that, you can install all the other toolchains in user home directories. I think this would only require a rustup config file in /etc/ to specify the system toolchain, when installed and it would be optional and rustup would never write one.

Edit: I’m mostly familiar with debian and it’s alternatives mechanism, which would allow something like this to work (though I’m not sure how they resolve alternative priorities):

/usr/bin/rustup - rustup binary
/usr/bin/rustc - symlink managed by distro alternatives
/usr/bin/rustc-rustup - rustup rustc proxy
/usr/bin/rustc-X.Y - system rustc
/usr/bin/cargo - symlink managed by distro alternatives
/usr/bin/cargo-rustup - rustup cargo proxy
/usr/bin/cargo-X.Y - system cargo

This would potentially allow you to get symlinks from rustc -> rustc-X.Y without rustup installed and then install rustup and have the symlinks change to rustc -> rustc-rustup, and rustc-rustup would default to calling rustc-X.Y, but would allow toolchain overriding, etc. like normal.

3 Likes
#160

The idea to install rustup without its proxies is doable, but rustup would be quite crippled. It does break the primary functionality.

Installing rustup without its proxies is the mode that would work the best for me. Here’s my thinking, partly repeating what’s in that AUR package thread. Curious whether anyone else’s thinking is similar:

  • Like @brson mentioned above, it substantially simplifies the story around Linux packaging, where rustc and cargo are already installed as independent packages. Some package written in Rust might want to take a build dependency on rustup, to compile with nightly for example, but that’s a drag if the rustup package conflicts with the rustc package.
  • I’m not a big fan of global state in my build tools. Instead of remembering what default toolchain mode I’m in right now, I like to define separate aliases that always refer to a specific version. Something like
alias rustcn="rustup run nightly rustc"
alias cargon="rustup run nightly cargo"
  • Changing what’s in the PATH can have a lot of unintended side effects, if you’re on a system that builds things from source. That includes AUR packages on Arch, for example, but also Vim plugins like YouCompleteMe that have a native Rust component. Usually you’re building those things to use them, not to hack on them, and it’s confusing if your rustup config breaks those builds. (Running rustup default 1.0.0 would probably break everything.)

Of course all of this is totally different on Windows where Rustup is the official way to install the whole toolchain. Can anyone talk about the situation with Homebrew on OSX?

#161

@ahmedcharles Either of those schemes is workable in rustup, as long as they are in the distros. Assuming a distro that doesn’t have Debian’s alternatives mechanism, it seems like it could be problematic for distros since they wouldn’t want all their rustc’s to be called rustc-X.Y when rustup is not installed.

@oconnor663 Thanks for the details!

#162

RedHat family distros also have an alternatives mechanism similar to Debian’s. It’s a bit unavoidable when you want to package VIm and elVIs, or Sun Java and OpenJDK.

#163

If you don’t have alternatives, a distro can package rustc-X.Y with versioned names and have a rustc package that conflicts with the rustup package and the rustc package provides symlinks. I believe that’s a fairly standard way of packaging compilers?

Though, I agree with @notriddle, in that distros probably have a way to deal with alternatives/conflicts in a reasonable way at this point. If they don’t, rust (at least) won’t be alone in causing them grief.

1 Like
#164

rustup 0.6.2 is out. Update with rustup self update.

This release primarily fixes a miscompilation bug that results in segfaults in rustup on multiple architectures in a variety of scenarios. The rustup build is currently pinned to nightly-2016-08-10, it’s not clear where the bug is, and there is no fix upstream yet.

0.6.2

Contributors: Brian Anderson, Diggory Blake, Knight, Marco A L Barbosa

#165

Oh, hey! It looks like installing source code for nightly works now!

Try:

rustup update nightly
rustup default nightly
rustup component add rust-src
ls `rustc --print sysroot`/lib/rustlib/src/rust

cc @matklad @phildawes look at this cool stuff! Very lightly tested…

3 Likes
#166

Wonderful! And this opens the door for the https://github.com/rust-lang/rfcs/pull/1133, which would be even better :fireworks:

#167

Is this only supported for the nightly toolchain? Because I couldn’t get it to work with the stable toolchain.

#168

Yes, only in nightly. The upstream documentation packaging landed this week.

1 Like
#169

It looks like rustup is still segfaulting.

#170

rustup 0.6.3 is out. Upgrade with rustup self update.

This release is just trying yet again to fix crashes in rustup-init.

0.6.3

Contributors: Brian Anderson

1 Like
#171

@brson: some further follow-ups over at https://aur.archlinux.org/packages/rustup/.

The more I think about this, the trickier it seems. The biggest problem seems to be that, when building Rust packages from source (which Arch users do more than many other distros, but which all distros must do in order to provide binary artifacts), it is hard to have any guarantees about what version of Rust you’ll get if you simply run cargo or rustc. If you know that rustup is being used, then it’s easy to override the version, but if you don’t (and a build tool generally shouldn’t need to – it should just require “rust”), then you’re sort of lost.

It’d be great if there was a way to tell rustup “ignore defaults and use at least this version”, but without relying on actually running the rustup command. Perhaps an environment variable that rustup will use. That way, a build script could do something like:

export RUSTC_MIN_VERSION=1.10
cargo build --release
rustc foo.rs

which works regardless of whether a “regular” rust install is used, or a rustup-managed version.

This is still not ideal, because it requires the build script (or the build tool, but that’d be worse) to specify the version dependency outside of the packaging system (in Arch, you’d normally just say depends=(rust>=1.10) in the package’s metadata), but it’s a decent first step.

Thoughts?

#172

@jonhoo I responded on the aur thread to the questions I saw there.

This can be more-or-less accomplished today with the RUSTUP_TOOLCHAIN env var, with the exception that it will not just automatically install that toolchain during the build. But to accomplish this at the project level I might rather have rustup read a config value out of the Cargo.toml metadata section or from a .rustup-version file (ala rbenv).

Is the problem you are concerned about only for the automatic building of packages, and do those packages all get built out of the same directory? If so you could have the rustup package drop the correct value into a .rustup-version file in that directory that contains whatever the correct version for the standard rustc package is; and the rustup package could install that toolchain globally (with yet-undesigned rustup global toolchain support).

#173

Frankly, I don’t think rustup belongs in distro packaging. Don’t get me wrong, it’s a great tool that I do use, but in a distro I think all package updates need to go through the distro’s own package manager. It’s rare that the distro should want multiple versions of a system compiler at all, and when you do it’s usually something like compat-gcc-XY, at least in Fedora (like this).

Personally, I’m using “rustup toolchain link system /usr ; rustup default system”, and then I can override with upstream toolchains as needed. But I’ll be expecting any Fedora-packaged Rust programs to play nice with Fedora’s rustc and cargo.

#174

Will rustup use a version of rustc/cargo newer than RUSTUP_TOOLCHAIN if one is installed?

No, packages in Arch are fairly often built in an ad-hoc manner as well. Packages that are released in binary form are built through a single, automated system. However, user-contributed packages are built by the users by downloading a tarball with a PKGBUILD file in it, and then running makepkg in the extracted directory. Users also have the ability to do the same for official distributed packages by downloading their PKGBUILD files. There is no good location for a .rustup-version file for all packages – one would have to be included with every package that uses rust.

I think it should certainly be possible to install rustup using a distro packaging system. The question is then whether rustup can be then also be used to provide rust for other packages that need it, and whether rustup should conflict with the stable, stand-alone rust package provided by the distro. One could easily create a rustup package that was not labeled as providing rust, and that was still marked as conflicting with the main rust package, but this would be very inconvenient, as you wouldn’t be able to install any rust-based packages (since nothing provides rust).

The Arch rustup package currently both provides and conflicts with the main rust package, and this works fine as long as the rustup default is set to something that can build any package. The discussion is basically what happens if the user has set the default toolchain to be something that is too outdated to build a particular package (e.g., they haven’t run rustup update in a while), or how do you cater for packages that require rust nightly (which rustup can also provide).

A second issue is how to avoid having the rustup package be marked as conflicting with the rust package, but this seems less problematic to me.

#175

Perhaps ccache is a useful example to follow. On Fedora, it installs all of its conflicting binaries in %{_libdir}/ccache/, then comes with /etc/profile.d/ scripts to put that ahead in the PATH. I could see a rustup package doing something similar, along with a default configuration pointing to the normal /usr toolchain.

#176

That’s a good idea for how to avoid the conflict, but that doesn’t fix the aforementioned issues with rustc versioning.

#177

I just realized that my personal preferences (system rust is always stable, rustup commands are explicit) would be very easy to satisfy, if there were a rustup package on crates.io. I already have ~/.cargo/bin added to my PATH to get cargo install working, so doing a nightly build could just be:

cargo install rustup    # This doesn't seem to exist.
rustup install nightly
rustup run nightly ...

Was it a deliberate decision not to put a package on crates.io? If it’s to avoid confusion with the standard install flow, maybe we could call the package something like “rustup-minimal” or “rustup-not-the-standard-install-flow-we-swear”?

1 Like