Based on requirements from the Perfecting Rust Packaging thread, I’ve outlined a set of tasks, divided between Rust and Cargo, that must be completed to make packaging Rust easier.
I’m very interested in feedback, especially from distros other than Debian - many of the requirements here came from Debian.
This is mostly focused on packaging Rust and Cargo themselves, though there are some steps here that serve to make packaging Cargo crates easier.
Once this has gotten review, I’ll make sure there is an issue filed for each and begin working on them myself (though I am happy to have help).
- Perfecting Rust Packaging. The thread on irlo where we discussed packaging requirements.
- How to distro-package a Cargo-using Rust executable. @gus’s early attempts to package crates.
- cargo-bootstrap @dhuseby’s script to bootstrap Cargo without Cargo.
- Fedora thread on anti-vendoring. Other messages in the thread: 1, 2.
- Debian’s packaging guidelines for Go. The Go static linkage model is similar to Rust’s preferred linkage. Other aspects that make Go difficult to package appear less relevant.
- Debian’s hacks to use Cargo. Cargo needs lots of work to make it friendly to distros.
- Debian’s makefile hacks. Distros need to pass common distro-specific linker flags to everything they build and distribute. This applies to their build of Rust, and also to all Rust crates they themselves build, implying Cargo needs similar support.
- Debian’s hardening guidelines. This is the kind of stuff distro’s like to customize.
- Gentoo’s libdir patch. The way rust/rust-installer deal with libdir is broken.
- The Homebrew formula. It appears to have some problems, particularly with how Cargo is packaged. And there is no definitive maintainer for this important package.
- openSUSE’s Ruby packaging guidelines.
Task: Compiler command line customization in the Rust makefile
Distros often have a standard, custom, set of flags that should be passed to all compiler invocations when building the binaries distributed as packages for that platform (think things like hardening options). For example, Debian wants to pass
-Wl,-z,relro during the link step.
To support this, every compiler invocation in the makefile needs to include the appropriate
RUSTFLAGS variable. See the Debian patch for guidance.
Task: Bootstrapping from previous releases
Bootstrapping from arbitrary commits is too difficult for downstream distributors that want to bootstrap from their own binaries.
We’ll retire the current snapshot system, and bootstrap off of the previous stable compiler. This means that upstream will need to wait six weeks between making changes that are incompatible with the bootstrap compiler. At present, the snapshot compiler is regenerated rarely, so this is expected to be ok.
I expect that smaller distros will not be able to keep up with our six week release schedule, but a simple script should allow them to catch up on the bootstrap by rebuilding the intermediate compilers.
We’ll set up CI to ensure that the distro scenario of locally providing the previous compiler works for each release.
Make sure to consider what happens for stable point releases, which we have never done. We probably need to guarantee that all point releases from series
N can bootstrap all point releases from series
Task: Re-bootstrapping from the current release
In some scenarios (I’ve forgotten offhand), distros want to rebuild the compiler using itself. This is more-or-less equivalent to starting the build at stage 2.
We’ll provide a way in the Makefile to do this easily.
Task: Bootstrapping unstable code from a stable compiler
The stable Rust compiler requires the nightly Rust compiler to build. This is no good for distros that want to bootstrap from their own toolchain and don’t want to maintain a nightly toolchain just for building Rust. We’ll teach the Rust makefiles to deal with bootstrapping off the stable compiler transparently (the stable compiler already contains nightly features, just inaccessible).
Task: Dynamic LLVM
Distro’s generally want to use their own system LLVM. Rust’s support for this has traditionally been spotty, but we currently do CI builds with a statically-linked 3.7. There’s demand for using dynamic linking for LLVM as well (Gentoo), so we’ll get that working and set up CI for it too.
We’ll guarantee in the future that Rust always builds with some recent release of LLVM.
See the existing PR.
Task: Add an i586-unknown-linux-gnu target spec
Debian’s 32-bit x86 distro uses i586-level cpu features while our’s is i686. We should be able to teach rustc about the i586 target so Debian can configure it as the host triple.
FIXME: per @eefriedman this may not be the right solution.
Task: Teach the makefile to mix in additional ‘extra filename’ information
Some systems, like Gentoo, want to package multiple versions of the Rust compiler along side each other. There are several obstacles to this, but the obvious one is that the installed Rust crates need to not have conflicting names. While we have a mechanism for this
--filename-extra, the extra strings appended by the current makefile are not sufficient to discriminate between arbitrary compiler revisions.
Most likely we will add a configure switch that specifies an additional string to hash into the filename extra, in addition to what we’re already hashing.
Task: Ensure that SxS installation of crates from multiple compilers works reliably.
For SxS installations of arbitrary Rust compilers, there will be multiple copies of the standard library residing in the same path. Without care, rustc will see them as duplicates.
Neither @alexcrichton or I am confident that the current crate resolver correctly rejects crates that weren’t generated by the same compiler, though we suspect it works correctly in most scenarios.
Tightening this up further could have negative implications for e.g. a distributed Cargo cache, depending on how strict rustc is about rejecting crates it didn’t produce.
I’m not sure how real this problem is but probably needs some thought.
Task: Fix --libdir issues
The way rust and rust-installer handles
--libdir is broken, and at least in Gentoo’s case is unusable without patching.
Task: Disambiguate system-installed crates during resolution
At the moment, if a Rust crate is installed to the system, then rustc builds that depend on that crate will fail with duplicate crate errors - rustc won’t tolerate multiple crate matches. This causes horrible problems for distros that are installing binary Rust crates.
Task: Update Homebrew packages
The homebrew packages don’t have a dedicated maintainer, and the current recipe has flaws: the obvious one is that there is no cargo source tarball for it to use.
While it builds Rust from source it appears to install Cargo from binaries.
It also removes the uninstall script. I’m not clear on why. Perhaps Homebrew has its own uninstall mechanism?
I think we don’t actually need to take over maintenance of the Homebrew package, just update it for best practices once we get them sorted out.
Task: Produce packaging guidelines
Summarize what we’ve learned into some general guidelines for packagers.
- Maintaining independently-bootstrapped Rust compilers.
- Packaging Cargo libraries / applications
- Generating offline docs
Task: Publish source tarballs of Cargo releases
Although Cargo does have tagged releases, we don’t publish source tarballs (the GitHub auto-tarballs are of course broken because submodules). Distros generally prefer tarballs to git.
make dist rules to publish source tarballs consistently with Rust’s own.
Task: Pair Cargo releases with Rust releases
While Cargo does have tagged releases, we don’t actually use those in the Rust releases, instead pairing rustc with an arbitrary recent revision of Cargo.
Update rust-packaging to pair rustc not with Cargo nightlies, but with Cargo stable releases.
Task: Make binary releases of Cargo
To pair Cargo releases with Rust our distribution servers must have versioned releases of Cargo to download. Add new stable dist builders and modify the Rust release process to also do a Cargo release.
Task: Validate that Cargo releases build with the corresponding Rust release
Distros need to build Cargo, and they want to do it with the version of Rust they are also deploying. Presently Cargo builds using some old, arbitrary version of Rust.
Set up CI to ensure that the Rust we’re releasing is capable of building the Cargo it’s paired with. Not sure the best place for this offhand.
FIXME: @alexcrichton thinks this is too much effort and I’m inclined to agree.
Task: Bootstrap Cargo without Cargo
Cargo is self-hosting and difficult to bootstrap on new platforms, moreso than rustc. This causes the most problems for systems that upstream doesn’t directly provide binaries for, like the BSDs.
Create some mechanism that can reliably bootstrap Cargo without running Cargo itself and add testing to validate that it continues to work. Needs design.
Task: Enable Cargo to work without any network access
Build farms need to build Cargo projects without hitting the network. Assuming that distros rewrite their Cargo.tomls to convert dependencies to local paths, then as far as I know the other source of network access is just updating the index.
Task: Compiler command line customization
As with the Rust build itself, when distros build and package binaries of Cargo applications, they want to be able to customize all command lines to all compilers.
In the Cargo case this means at least customizing the rustc command line; it’s not clear whether Cargo itself needs to provide facilities for customizing CFLAGS, etc. or if that’s the responsibility of build scripts.
This requirement seems to be at odds with design goals of the Cargo developers. Design work is needed.
Non-goal: Dynamic-linking support for ‘anti-bundling’
Many distros strongly prefer to use dynamic libraries, and have policies against static linking (see a recent Fedora thread).
While Rust does support dynamic linking, the default is to link statically, and almost all Rust crates do so - the obvious exception being rustc plugins.
The strongest reason for prefering dynamic linking is so that distros can provide security updates without recompiling downstream reverse dependencies.
This use case is not generally supported by current Rust, even with dynamic libraries. Details of Rust’s unstable ABI currently require all downstream code to be rebuilt when upstream changes at all. This is strongly enforced by the compiler itself.
Because of this, I do not see any practical advantage to promoting dynamic linking in Rust - it is likely to surface weird discrepancies between the dynamically-linked distro world and the statically linked upstream world.
Distros will need to adopt mechanisms to handle security updates by recompiling reverse dependencies. Barring major work on the security-update use case, this would be necessary even with dynamically linked Rust libraries. (TODO: any examples of how distros already deal with this - e.g. with Go)
This applies most acutely to the packaging of 3rd-party crates, not the standard library itself, so it will not become problematic until Rust applications begin to be packaged by distros.
I recognize that dylibs-for-security-updates is an important use case for distros, but solving it is a large problem that is out of scope for now.
Non-goal: Redirect crate dependencies to local source installations
Distros that want to package Rust applications built with Cargo generally want to package all their dependencies themselves as well.
This means that, when a distro is building a packaged crate, it wants to use its own package manager to install all the dependencies locally; then when Cargo runs it downloads no additional source code, instead building the deps from the local source.
Presently, we believe an acceptable solution to this problem is that distros that package crates will rewrite the Cargo.toml files, replacing the crates.io dependencies with local path dependencies, according to their own scheme.
Note: Gecko also wants to be able to use its own sources and prevent Cargo from hitting crates.io, but their use case is different - they aren’t repackaging their deps; they just don’t want to depend on external resources. There is still work to do here.