Rust MSRV policy and Linux Distros

internals policy
1

Discussion about MSRV of the libc crate raises the question of what actually is Rust's policy for support of old Rust versions.

  1. Can the Rust project have a clear minimum supported Rust version, including explicit guidance for Linux distributions and crates-io crate publishers?

  2. If Linux distros aren't packaging Rust quickly enough, can the Rust project work around it? For example, provide official up-to-date packages for mainstream Linux distros, or convince Linux distros to package only rustup, and not rustc and cargo.

The current MSRV situtation is all over the place:

  • Rust releases have a 6-week cycle, and AFAIK this technically makes every compiler older than 6 weeks obsolete and unsupported. But I haven't seen any clear messaging from the Rust project stating so and discouraging people from using a compiler older than 6 weeks.

  • In the crates-io ecosystem people choose which Rust version they support, which varies, and there isn't a universal agreement how far support should go, or whether MSRV change is a breaking change. Some projects support only the latest stable, many projects are compatible with a few previous Rust releases (either as a matter of policy or just by chance), and there are some projects that make effort to support very old Rust versions.

  • Linux distros range from "LTS" types that freeze Rust versions so old that they're unusable with a majority of crates-io crates, to more frequently updating and "rolling" distros that try to keep up with Rust, within constraints of their own release schedule. Is a distro with a 6-month-old Rust acceptable, or are they shipping versions past their expiration date?

The problems are:

  • Debian. I don't think Debian's policies can be reconciled with Rust's. Can Debian be convinced to make an exception for Rust? Can Rust work around Debian?

  • There are people (and by extension companies they set policies for) who have "over my dead body" reaction to curl | sh installation method of rustup. Whether that's a justified security policy or an irrational superstition, it's non-negotiable either way. These users want some kind of a "proper" package, and typically prefer to get it from their chosen distro.

  • New users coming from a very slow-moving C don't expect to be updating the compiler often, and expect projects to wait years before using any new compiler features. Such habits and expectations set them up for an awful experience with the crates-io ecosystem, and they typically misattribute that to Rust being "unstable" and "changing things too often". I don't think Rust should change its release frequency, but I'd love to find a solution to onboard such users smoothly.

  • crates-io is not optional for Rust projects in practice. Telling Linux distros to package/fork crates (90K of them!?) is not a serious solution. Leaving manual dependency downgrades to users creates an unpleasant experience.

  • Support for dependency resolution using rust-version field (or other similar features) would help decoupling MSRV of crates-io and Linux distros, but it's not a silver bullet:

    • the question which Rust versions are supported remains, because crates may need to release security fixes, and users of older compilers won't like being left with vulnerable crates, even if they compile.

    • any improvements made in cargo/rustc, even if they were released tomorrow, will still take some time to reach all distros, so there's still a question if/how to support all the Rust versions already released.

25 Likes
2

As I understand it, various distros already package tons of Python libraries, so that you can install Python apps using the system package manager. Does that not work well in practice? Is there something that makes Rust crates less suitable for distro packaging than Python projects?

6 Likes
3

Rustup being what it is, installing it from a distro or other package source isn't an obstacle to using it to obtain a recent compiler. If distros have rustup packaged, and users are willing to install it, that's a no-problem outcome — for the use case of having a Rust development environment, at least (which is different from packaged software written in Rust).

8 Likes
4

As I understand it, those are frozen versions meant solely for satisfying the dependencies of python applications that are shipped by the distro. You are meant to get dependencies for your own applications from pypi using pip. This is the exact same situation as with rust where at least Debian ships with source bundles of all crates used by rust programs part of the repos. The only difference is that in case of python it is easier to accidentally depend on the distro versions due to them being in the default search path, while for rust you need to manually configure cargo to use these sources.

6 Likes
5

The best option would be packaging rustup and let the user choose the compiler version, instead of the distro. That way, we expose users the compiler version they want, not the one the distro wants. I'd be surprised to see rustc-nightly or rustc-beta on debian.

The thing is, that if at any point in time rust decides to expose the download url with another path (e.g. https://rust-lang.org/compiler/$release-cycle/$version instead of https://rust-lang.org/downloadsrelease-cycle/$version) debian users with old rustup versions won't be able to install/update rust altogether until debian releases a new version of rustup. Best-case scenario (rustc already installed) the problem just repeated, worst-case scenario (installing rustc) you don't have a compiler at all.

If we end up "recommending" distros to only package rustup instead of rustc & cargo, we will have to get sure we maintain 1:1 rustup backwards compatibility for several years, which I don't know if it's the case or if it can be at all.

6

IMO download URLs (like, honestly, all URLs) should comprise a stable interface (both in API and UI sense) and any changes should always be accompanied with appropriate 30x redirects. Link rot is not cool.

10 Likes
7

Agree, but that was only an example. Say now cargo needs to be installed on another path, or even better, rustc has to be installed on another path. Say there's a new rustup-installable thing needed for cargo. Old rustup won't do the job, and debian users when updating/installing will end up with a (partially?) broken compiler. If we want to have rustup as the official rust package, we need to hold a lot of guarantees.

8

We already have to hold a guarantee that the existing distribution URLs remain in service; we don't want to break existing installations of rustup. We've backed out changes to rustup/cargo before because they caused existing versions to break. The network protocol is intended to be reasonably forward-compatible[1].

Disclaimer: this post non-sequiturs through several mostly unrelated concepts/ideas.

Disclaimer: my majority interaction with distros is in arguing against people trying to treat Rust like C, and as a Windows user I'm used to the primary way of acquiring software being from the vendor site and not some platform package manager. Even though winget (and scoop/chocolatey) are nice to use in presenting a uniform way to get software, they still amount to metadata pointing to external downloads rather than a central self-contained repository of binaries.

I also have no sysadmin or infra experience beyond just that required to keep my own development (and gaming) machine working.

Spitball proposal: spin up https://apt.rust-lang.org/ (or some other url) as an alternative package repository, and officially provide apt-compatible tarball installation from there for the Rust toolchain. (In a perfect world, perhaps rustup would even use this as its package server for apt-using hosts.) This provides an option for installation that's more checked than curl | sh while not significantly changing the infra packaging burden[2] or what versions the project officially supports.

If disto users are using Distro Rust because they (think they) need the extended support timeline, then they can continue to use the distro's version, but for the ones who are just using it out of inertia/convenience they can just add the rust-lang apt server rather than having to switch to rustup (though that is still of course preferred).

Perhaps this is naive, but: I don't think Distro Rust should be used for development.

As I understand it, the purpose of the distro is to offer software to the OS user. The fact that C development packages ended up going through the distro package manager is then the result of a few combined factors:

  • C lacking any preexisting unifying package manager or build system,
  • the dynamically linked distribution of C libraries to end users as part of binary distribution
    • where the development distribution of the same libraries is nearly identical, merely including a small number of text files describing the linking interface on top of the user distribution, and
  • the linking ABI to libraries and the OS being defined as what the distro-provided platform C compiler produces.

C++ then gets shoved into a C shaped hole, because it was specifically designed to fit there. Templates throw a spanner into the works, but a mostly ignorable one by sticking to the "everything in the header is public stable API" standard.

Because C and distros developed in tandem, they are naturally well matched. But Rust lacks all of the factors which make C a decent[3] match — Rust has a comparatively rapid iteration cycle (both for compiler releases and library adoption), an unstable statically-linked ABI, and a development package which contains significantly more metadata than a for-end-user distribution.

The argument then is targeting Distro Rust so that your binary can be packaged in the distro and compiled with the uniform shared distro toolchain, but that doesn't particularly apply: when the distro is packaging new packages for distribution then they're also packaging a new rustc, so the Distro Rustc at the point they would package your binary is fairly recent.

If the distro is willing to package new binaries, they should also be capable/willing to package a new rustc at the same time.

I agree that distros forking ~all of crates-io is untenable. What may be tenable, though, is a much smaller task: distros providing an alternative registry which mirrors crates-io but hides package versions with a rust-version higher than the Distro Rust version.

This should be a mechanical and trivially automatable overlay (presuming rust-version is in the registry and accurate) and even stateless; the main difficulty is in monkeypatching the returned crates to refer to e.g. debian-crates-io-mirror rather than crates-io.

There's also two choices for how to construct the distro crates-io mirror: it can be a "live service" subset of crates-io, or it can be a snapshot taken at the same time that the distro packages their rustc and other written-in-Rust software packages.

Using Distro Rust becomes a meaningful value-add if that means enabling something which Rustup Rust does not offer. The extended support timeline is one, but one that IMHO comes more from a fear of updates than any concrete benefit. Rust provides strong backwards compatibility guarantees[4], so upgrading to the mainline supported version should be just as guaranteed [5] to work as whatever distro supported LTS patches are provided (which I would hope people asking for LTS would actually use).

The reason I bring this up is that Distro Rust being pinned for a longer timeframe is well positioned to actually offer a benefit of precompiled libraries. Because everything on Distro Rust is using the same version, it would be possible to provide rlibs and dynamic linking (at least in the case where all features are on and purely additive).

No matter the policy that comes out of this discussion, we should get rust-version metadata into the registry ASAP. This data being available allows 3rd party solutions to emerge.

(I am now somewhat interested in the viability of offering pubgrub-based version resolution as a 3rd party generate-lockfile/update/upgrade command. If only I had more time.)

  1. I can never keep it straight when exactly to use forward/backward, and always end up second-guessing myself. It doesn't help that it's two sides of the same coin; forwards compatibility today is to make more things backwards compatible in the future. The point being here that we absolutely do want at a minimum rustup self update to always work, no matter how long it's been since you first installed (a self-updating configuration of) rustup, and as a tiny step above that, rustup update. ↩︎

  2. I think so, anyway; I've not done apt packaging before, but a quick search suggests the server is a static file server, and the official packages would still only interoperate within the same versions, and this wouldn't impact what versions are considered officially supported. The point is solely to provide an alternative installation channel that fits better in a traditional distro workflow. ↩︎

  3. The repeated emerging of source-based package managers for C shows that distros aren't a perfect match for what C developers actually want anymore, though. The distro is a perfect fit for piecewise distribution of the OS, but this also assumes that all of the packaged software is okay with being treated as part of the OS. ↩︎

  4. And they're getting stronger; see the discussions around edition-dependent[6] name resolution for the efforts towards mitigation of some of the allowed breakage. ↩︎

  5. I'm still explicitly ignoring use cases such as a validated compiler (e.g. ferrocene) version which gives additional guarantees. Targeting those is an objectively measurable benefit. ↩︎

  6. Aside: I personally much prefer a rust-version-aware name resolution scheme to edition-dependent, for the same reason that I prefer a shorter MSRV timeline than a distro-scale 3 years. ↩︎

1 Like
9

Not sure if this has been mentioned yet, but NixOS makes excellent use of this: FHS Linux binaries do not run on NixOS unmodified, so NixOS packages rustup with one small addition: auto-patching downloaded binaries/dylibs to function.

So you get rustup from nixpkgs and you use it just as normal (only rustup self update doesn't work, which is normal for it being distro-installed). Also, it means the various wrappers (like rustc) end up in $PATH without extra configuration.

(Whereas the curl | sh approach has many chances to fail as long as upstream doesn't explicitly support NixOS - and even if it did, it would be hackier because it would have to pull the needed dependencies out of thin air, instead of how the rustup package in nixpkgs gets them from its own dependencies)

5 Likes
10

Just a clarification note:

The Rust project's supported compiler version(s) and the ecosystem libraries' compiler support are distinct (though related[1]).

"Compiler (long term) support" for a toolchain is roughly getting (particularly security) bugfixes.

"Library (long term) support" for a toolchain is roughly MSRV; that libraries compile with that toolchain and get (particularly security) bugfixes.

Separately,

A question for packagers: would it fit better into your model to have rolling support windows (e.g. the most recent 6 toolchain versions are supported) or flag day LTSes (e.g. every 6th toolchain version is designated LTS and gets support for 9 toolchain releases)?

The difference for library authors may be minimal, but the difference is meaningful.

  1. Any (aspiring) major library can trivially be expected to work on all officially supported compiler versions at a minimum. (With two caveats: if it's using nightly / otherwise branding off of the bleeding edge, or if the officially supported compiler versions go back further than the loose ecosystem consensus of a reasonable 6 months of compiler support.) ↩︎

11

In the OCaml community, compatibility-conscious package authors tend to follow a common practice of defining the "oldest OCaml version we should make efforts to support" as "the OCaml version provided by Debian Stable" -- which is the latest OCaml version available at the time that particular Debian stable release was cut. For example, currently debian stable proposes OCaml 4.11.1, which was released upstream in August 2020. Many packages work fine on OCaml versions much older than that, but this is the common reference we use when the question arises.

Note: The compiler maintainers do not provide support for older OCaml releases unless in exceptional situations (when ARM M1 was released, we backported the new backend to an older release). The version-support question is mostly for the packaging ecosystem on top of it.

It's useful to keep communication channels open with linux distro packagers for other aspects as well. For example, they tend to develop a patchset of small changes/fixes on top of the implementation, and we try to have a look on a semi-regular basis to upstream what is upstreamable to reduce their workload.

12 Likes
12

It's worth noting that OCaml as a language does have the benefit of being extant since 1996. I expect in 20 years that supporting Debian LTS will also be entirely reasonable, but in reality, we move so fast that even that is very hard-pressing on many maintainers right now.

We do maintain close communication with the Red Hat family of distros thanks in no small part to @cuviper's tireless efforts. If there is anyone "taking point" in the same way for Debian or other distros then it would be news to me. But also according to his report, Red Hat packages us in rolling streams so that we are advanced forward quite quickly compared to Debian. That is what was presented, as I understand it, as the far end of the range from Debian's multi year release cycle.

Similarly, Alpine has a proposal to closely follow Rust open for keeping a very-recent Rust toolchain available instead of trying to work against the Rust model, though work has been stuck on fixing a few issues in rustc and the musl targets for some arches (which were unfortunately always latent since those targets were added).

But it goes further than that. Arch Linux offers the option to install Rustup from the distro, which puts the Rust toolchain into the distro's understanding of its packages, without pinning rustc. But they are, of course, a rolling release model and try to use the very latest patch from everything. And that is, indeed, a model some distros use for everything. Many user-packages in the AUR simply install from a git repo.

NixOS, of course, is... NixOS. It... eddyb covered the details of what nixpkgs does to Rustup, but that in essence entirely rejects this question about MSRV and LTS and even rolling release by simply providing a different answer entirely.

Gentoo and its ilk... uh. Well, they are welcome to ask for more help with building rustc from source, but I am pretty sure we support that use case on the face of it...? Occasionally someone using Gentoo reports a bug, which is cool. Occasionally someone using Gentoo reports that they edited the build for Rust and its dependencies and it surfaces an incompatibility with the newly edited dependency, which is... somewhat baffling?

We can certainly continue on with this discussion of "what do we do about Debian (and Debian-based distros, like Ubuntu)?" But I think that given that there is more than one distro that will in fact install us from the absolute nanomolecular edge of our release schedules, we should at least put into the discussion the completely opposite answer of trying to meet the LTS needs of a specific vendoring of a specific OS, which may be more opposed than one might expect to the needs and wants of other vendors of that same OS.

From my point of view, the Jedi are evil!
From my point of view, the Jedi are evil!773×327 211 KB

8 Likes
13

I really don't think we should treat this as a "versus". Linux distributions have the policies they do for good reason. We have the policies we do (or policies we're working on improving/ratifying) for good reason.

Rather, I think we should consider how we (the Rust Project and Linux distributions and any other redistributors) can serve the users who get toolchains from us, and share work that's in common, while not having to do extra work to cover use cases involving toolchains not obtained from us.

That may mean tooling to obtain compatible packages, or tooling to mirror packages and support offline builds, or other potential solutions.

8 Likes
14

Great discussion! I have some experience with Debian, packaging and security so here are my thoughts:

Can Debian be convinced to make an exception for Rust?

I think this would be the best scenario. (Not latest version ofc, just a bit faster.) Rust has an awesome CI, including crater that makes upgrades very safe. Also the safe nature of the language makes it unlikely that previously-working UB gets turned into vulnerability because UB is uncommon in Rust projects. I was thinking about compiling the list of arguments for and against fast upgrades and mailing it to Debian devs for consideration but I feel it may be too soon.

My guess is that the biggest obstacle is LLVM version. Rustc depends on arbitrary version of LLVM. Debian wants single LLVM library version for Clang and Rust. It's not just about size optimization but it makes cross-language LTO possible! Would it be possible for rustc to be less aggressive about LLVM and perhaps even depend on the version that is in Debian?

package (only) rustup

This doesn't really solve the security issue packaging solves: HTTP servers may get compromised and contain malicious artifacts which historically happened to various projects. Rustup currently doesn't verify signatures so until Tracking: Simple PGP signature verification · Issue #2028 · rust-lang/rustup · GitHub is resolved packaging rustup will not help.

Rustup installs Rust for user but apt installs system-wide. Both cases seem to have their place. An interesting advantage of apt over rustup is that packages can depend on other system packages - e.g. for cross compiling apt packages could install appropriate linker by default (whether they do I don't remember) which is impossible from rustup (today).

Python is an interesting point. In my experience, yes it's possible and not too annoying to develop against system Python packages although it's not 100% smooth. This currently can't be replicated in Rust because there's no way to teach cargo to use system dependencies. And even if it was we still need a separate step to install them which should be made a single automated command. The biggest question remains what to do about external dependencies and Cargo.lock.

Part of the issue is also pressure to update stuff causes unaware people or bots submitting "update dependencies" PRs which break MSRV so need to be rejected. Also bad habit of updating Cargo.toml versions even if no new features are used. This mainly affects projects that want to be compile-able on Debian stable and also without it (most of my projects actually). Not sure how to avoid these.

7 Likes
15

Under normal circumstances, I'd say "Debian changing its stable policies will never happen", but they did for Firefox, and as a result do for rustc and cargo: Debian -- Package Search Results -- rustc-mozilla .

That said, I think the easiest and lowest-friction way to solve this in Debian would be https://backports.debian.org/ .

Sure, that would certainly help.

2 Likes
16

Upstream rustc generally uses the latest LLVM version and pre-releases when they are available to avoid discovering regressions after an LLVM version is released. We support building with LLVM versions several releases back. We do support some older versions. Currently the oldest version is 12. According to Update the minimum external LLVM to 11 by cuviper · Pull Request #90062 · rust-lang/rust · GitHub this is because LLVM 11 (which debian uses) has a bug that resulted in miscompilations after a bootstrap bump: Miscompilation with LLVM 10 and 11 with codegen-units=1 · Issue #90153 · rust-lang/rust · GitHub While it would be possible to work around this specific instance, there is a bug chance that the bug would cause other miscompilations, so no in this case being less aggressive was not an option.

Edit: The repro for the miscompilation was supposedly just a couple of simple lines. This is way too easy to trigger IMO.

fn main() {
    // Start to end, with a RangeTo.
    let mut bytes = *b"Hello, World!";
    bytes.copy_within(..3, 10);

    assert_eq!(bytes, [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 72, 101, 108]);
}
4 Likes
17

Cross-language LTO is a great point for system integration, thanks for mentioning it!

We don't have an exact policy, but we usually try to keep LLVM compatibility for 2-3 release back, which amounts to a year or so. e.g. Currently the compiler supports LLVM 12 to 14, with some effort to also keep up with llvm main (15), and the rust source distribution bundles 14.0.6. But that takes a fair amount of effort for compatibility shims, as the LLVM API is always shifting, so I don't think we'd want to extend that much farther.

4 Likes
18

Thanks for explanation! Interesting issue. Would it be reasonable to backport the LLVM fixes instead of using the latest LLVM? Although given the wide support of LLVM versions it may not be needed that much, maybe something to keep eye on?

19

We can and do backport patches to the LLVM that's bundled with Rust, but for the purpose of older LLVM, it's up to the distros providing that LLVM to backport patches. In that bug where we ran into the issue on CI, we could have asked for Ubuntu to fix it, but we'd be blocked until then.

3 Likes
20

I should note that Debian ships two copies of rustc already, which are allowed to have different versions. One is called rustc, the other is called rustc-mozilla. The latter is one that a Firefox release may be built by and linked against (they also ship a dynamic Rust libstd, you see). They will bump rustc-mozilla's version even for security releases, seemingly "against" their policy of never bumping things in a featureful manner for security patches, if that is what is required to get Firefox patches building without going to greater lengths to backport the patch.

In other words, Debian already makes exceptions for us as they see fit, in order to economize on labor.

Also, there is no obvious "cadence" to the Debian rustc release policy. Their current version of our toolchain on their testing branch is rustc 1.59 (maybe they bumped it due to the CVE against 1.58...? but I am pretty sure that doesn't affect Linux...?), but before that it was 1.48, and before that was 1.41, and before that was 1.34. 7 or 11 versions is a pretty wide gap, and then another 3 just to catch up with us, even on the experimental testing repos.

Clearly some work has to be done with respect to Debian, and it will almost certainly involve some design and feature work on our side, but Debian does not run its show in some predictable and rigid manner that never varies for anyone. They use Rust as they like, and they update rustc when they want to. Which is fine, normally, because they vendor everything they need specifically to get around the problem of anyone updating dependencies, and we pull everything off crates.io and static link everything specifically to avoid having to deal with... well, problems like with Debian! Indeed, really we're in agreement with Debian:

We don't really want anything to do with each other's preferred software distribution infrastructure!

We ALREADY mitigate against interacting with each other's software distribution infrastructure!

Kornelski started off by claiming that crates.io is "not optional... in practice", but apparently Debian does in fact vendor about 200 crates just to work around that. They are quite determined to keep doing things the way they're doing them. I think we could have a nice chat with them to make things easier for both of us, but I think it would be unwise to assume that such a conversation will end anywhere near them changing their overall process of vendoring stuff into apt.

8 Likes