Perfecting Rust Packaging

[alexcrichton]

I’m not sure how to describe the rationale more clearly - after you’ve reread that post, perhaps you could describe how you see the distro security workflow proceeding for a fix to a Rust package?

Ok, it wasn’t clear reading originally whether references were referring to C or Rust libraries, but with the mindset of both this makes more sense. Note that I wouldn’t consider myself an expert at all in packaging, I’m just trying to understand the problem space!

So with all that in mind, I’m still a little confused about how you might be expecting things to work out. It sounds like you want to mirror Cargo.toml as normal Debian build dependencies and not use Cargo for dependency management at all. This would allow you to understand the structure, have each source tree be independent, and if you need to you can patch any crate in the ecosystem. On the other hand it also sounds like you want to use Cargo to build everything without needing modifying any sources and have it just pick up all the dependencies which happen to already be on the system. Does that sound right? Reconciling these two desires will be difficult to do, but it may indeed be possible.

We may also want to continue this discussion on a new thread (as I believe @brson wants to focus this primarily on packaging Rust/Cargo itself at least from the start).

Aside, I find it odd to refer to foreign (non-rust) libraries as “native”. It only serves to present rust as the outsider. If rust is to be a systems language, pervasive throughout the distro, then it has to “go native” too, no?

True! I just use it to colloquially refer to “things normally found in a package manager” vs “normal rust crates on crates.io” kinda

We don’t want the whole world of crate dependencies bundled together, for the same reason we try to avoid “native” library bundling. Each crate that ends up in the distro should be maintained in one place, so they can be easily tracked as necessary. Realistically this means each is its own package.

While I think this makes sense, I think that something may need to budge somewhere on this. Unless all package managers are explicitly willing to entirely duplicate everything Cargo does for dependency management we may need to start assuming this may not happen and strive to find some other solutions.

For example, why do distros want to duplicate Cargo’s dependency management? Or is it fair to say that distros want to do this? Is this only for security updates? If that’s the only reason, then we can probably reach a more targeted solution, but it’d be good to explore this space first. (although like above we may want to continue off-thread to avoid getting too far in the weeds!).

The reason I mention perhaps finding another solution is that I’m not sure that distros really want a package-per-crate. Projects like servo have hundreds of dependencies, many of which are tiny and the likelihood that distros keep up with the rate of change of the entire Rust ecosystem seems… unlikely? The other reason I feel distros don’t want to do this is that it doesn’t currently really make sense to “install a Rust library”. Cargo explicitly will not look at the system for dependencies (e.g. this is how it guarantees reproducible builds), and managing dependencies for a Rust project is idiomatically done through Cargo, not the system package manager.

[cuviper]

So with all that in mind, I’m still a little confused about how you might be expecting things to work out. It sounds like you want to mirror Cargo.toml as normal Debian build dependencies and not use Cargo for dependency management at all. This would allow you to understand the structure, have each source tree be independent, and if you need to you can patch any crate in the ecosystem. On the other hand it also sounds like you want to use Cargo to build everything without needing modifying any sources and have it just pick up all the dependencies which happen to already be on the system. Does that sound right? Reconciling these two desires will be difficult to do, but it may indeed be possible.

I think this is right. Cargo can still verify dependencies, similar to the way a configure script would, and after that it just needs to act like make.

Patching sources for packaging is fine in some cases, but if we would end up repeating the same modification on every single one, that’s probably something the tool should support directly. Overriding opt/debug flags with distro choices, for instance.

We may also want to continue this discussion on a new thread (as I believe @brson wants to focus this primarily on packaging Rust/Cargo itself at least from the start).

Packaging just Rust and Cargo is a great start, of course. And while Cargo itself uses many crates, I think we can get away with bundling those together for the moment, at least while this is the only place they are used in the distro.

For example, why do distros want to duplicate Cargo’s dependency management? Or is it fair to say that distros want to do this? Is this only for security updates?

Security is probably the most important, and generic bugfixing also follows.

Keep in mind also that a distro is tracking dependencies of the entire world; Cargo only knows Rust crates. In a world where Rust programs and libraries are integrated more deeply with the rest of the system, it would be bad if the distro package manager had only a Cargo blackbox for the Rust corner of the world.

The reason I mention perhaps finding another solution is that I’m not sure that distros really want a package-per-crate. Projects like servo have hundreds of dependencies, many of which are tiny and the likelihood that distros keep up with the rate of change of the entire Rust ecosystem seems… unlikely?

We don’t have to package everything on crates.io, just those that prove useful for a program we want in the distro. That still may be a lot of tiny crate packages, yes. But how many of those hundreds used by servo are frequently updated? My hunch is that most of these are probably for one-off functionality that won’t update much.

Compare to a few other languages, with a quick glance at Fedora I see 176 golang, 669 ruby, 1482 python, and 2677 perl packages. And this is just searching at the level of source rpms; some may create multiple installable packages. (Searching this way also includes all historical packages, so some might be retired by now.)

Even though perl has the most, that’s a drop in the bucket of the 154,917 modules on cpan. In the 3,013 crates on crates.io, I’m sure there are much fewer that will find their way into distros.

Anyway, yes, distros keep up with lots of packages.

The other reason I feel distros don’t want to do this is that it doesn’t currently really make sense to “install a Rust library”. Cargo explicitly will not look at the system for dependencies (e.g. this is how it guarantees reproducible builds), and managing dependencies for a Rust project is idiomatically done through Cargo, not the system package manager.

Surely part of that idiomatic way is because these things just don’t exist yet in the system package manager.

I think you’re sort of getting at the disconnect between distro packaging and containers here. You may not be explicitly containerizing, but building all your own exact dependencies into a monolithic static binary is pretty close. I definitely don’t think we should dig into that disconnect here, but let’s allow that both methodologies should be made possible with Rust. If Rust is to be the new systems language, it needs to be able integrate into systems too.

[cuviper]

Projects like servo have hundreds of dependencies, many of which are tiny

Just curious, do you know how many of these are truly external dependencies? At a glance I see a lot of relative paths, which would be fine to keep in the same source package if these are just pieces of servo.

[brson]

I didn’t read the whole thing yet (I plan on doing it later today), but as the rustc packager for Exherbo, the main critical blocker for us atm is https://github.com/rust-lang/rust/issues/16402 (which doesn’t seem to have been mentioned here yet)

Thanks for the link. I’ve noted it.

Currently we also repack the source tarballs to remove src/llvm/* (unneeded, and big) and src/etc/snapshot.pyc (shouldn’t be in the upstream tarball anyway).

Thanks. Filed an issue.

Note that even when we kept the full src/llvm/ source, we still had to remove src/llvm/cmake/modules/LLVMParseArguments.cmake because CC-BY 2.5 isn’t DFSG-free

Thanks for bringing this to my attention. Here’s another issue.

I think this means we’ll need to improve the Debian build to detect version and jump straight into stage2 for the former case, since stage1 won’t build with its own release (until upstream moves away from using #[cfg(stage0)] as a versioning mechanism) - this doesn’t sound too bad, but I haven’t actually done it yet so I don’t know what traps await…

I’ve made a note of this self-re-bootstrapping requirement. We can bake this functionality into the makefiles, including bypassing the feature checks so you don’t need to package nightly.

In a hypothetical future Debian stable release …

Thanks for all this explanation and the link to your proposal.

No, I meant that the binary package naming/dependency metadata should be so that you can install librust-std for the host platform using just its short name without the target triple (using Fedora as an example):

Thanks for the clarification.

Once we get bootstrapped, the Fedora build root will have whatever rustc package is currently stable. So if we’re on rustc-1.4.0-1, and we want to build rustc-1.5.0-1, that’s fine, it’s the previous release. Then a point release appears, we need to build rustc-1.5.1-1 using the buildroot’s 1.5.0 we made earlier. Then if we need to rebuild for any reason, now rustc-1.5.1-2 will be built from 1.5.1-1.

There are ways to override the buildroot, but I don’t think you can tag obsolete packages that way. Usually overrides are used to bring in some new package to chain build others, before sending them all to stable.

It sounds like @gus is also describing the same thing about jumping to stage2, but I hadn’t even considered the problem of feature use. I don’t think multiple --release-channel builds should be necessary – there ought to be an option for even stable builds to use features. There’s no need to be draconian, just let advanced use cases be advanced. IMHO

Thanks for the description, and agree that rebootstrapping shouldn’t require nightly. We’ll solve that in the makefiles.

Edit: Heh, I just realized how terse this message was. ‘Thanks!’. It’s late here.

[alexcrichton]

Ok, cool, thanks for all the info @cuviper! I think for now we can hold off on more details for a later thread (and focus on rustc/cargo themselves here), but it’s really helpful to learn about this space!

Specifically on the topic of Servo I believe they’ve got a handful of external libraries (e.g. skia, harfbuzz, png, etc), but they’d probably know more than I!

[mzabaluev]

The reason I mention perhaps finding another solution is that I’m not sure that distros really want a package-per-crate.

This has been the rule for most other module ecosystems in the Linux distros (Python, CPAN, Eclipse modules to name just a few; Fedora even repackages node.js packages despite the strong preference for bundled dependency installation in npm). They do want to reflect both build dependencies and binary dependencies in the respective source/binary packages, avoid downloading in builds, and discourage library bundling.

I think the main reason for that is maintainability on the distribution level. If a library crate X is found to have a critical bug, they don’t have practical ways to comb through the entire package universe searching for Cargo build manifests listing that library as a dependency, let alone projects that don’t even use Cargo. All they have is their package database, so the bundling of crates should be reflected as build dependencies in the packages.

[SimonSapin]

Just curious, do you know how many of these are truly external dependencies? At a glance I see a lot of relative paths, which would be fine to keep in the same source package if these are just pieces of servo.

components/servo/Cargo.lock currently lists 181 packages, 118 of which are from crates.io and 34 from other git repositories. This leaves 31 crates in the same repository.

[jauhien]

Hi,

I didn’t read the whole thread, sorry (had no time, going to do so), so may be I’ll repeat someones thoughts. Some points from the Gentoo point of view:

• System wide LLVM: it worked for us, but now we’ve switched to shipping shared LLVM libraries only, so we need support for linking with shared libs from Rust upstream /see rust-lang/rust#27937/.

• Installation of multiple rusts. It works for us without multirust, the only thing we need is the ability to install rust libs/binaries to custom dirs. There is one bug in the rust installer that we temporary fix with patch.

• Cargo binary package support: we really need versioned cargo binary releases similar to those versioned rust binary releases (I failed to find any versioned ones for cargo).

• Cargo/rust infrastructure support in general: the main problem is how to make cargo use system libraries during production build (it fetches everything and uses fetched versions). This + not very clean linking model (or may be I just do not understand it well enough) makes shipping cargo based packages in gentoo at the moment near impossible. The ideal solution would be if there existed some ‘production’ mode for cargo when it is used just as a nice build system and uses only packages already installed in the system. If no necessary package is found it should just fail.

• -Werror switched on during Rust build, ideally it would be good to have a possibility to switch it off (I’m just sedding mk scripts at the moment)

• Additional libraries suffix. At the moment I’m sedding with sed -i -e "s/CFG_FILENAME_EXTRA=.*/CFG_FILENAME_EXTRA=${postfix}/" mk/main.mk, it would be good to have some switch in the build system to do this (it is necessary for the possibility of relible side by side istallation of libs).

• Signing of rust release tarballs, so their authentity can be checked.

• Rust bootstrapping: at this point of stability we can switch to bootstrapping by the previously installed compiler I think. There is a switch in the build system that can be used for it (local rustc). But we need guarantees from upstream that compiler will be self compilable by, say, N previous versions.

So far that’s all from my side. I’m going to read the thread and may be I’ll have more points.

[sanxiyn]

-Werror switched on during Rust build, ideally it would be good to have a possibility to switch it off (I’m just sedding mk scripts at the moment)

I agree, but if -Werror is causing any problem for you, I think we are interested in bug reports.

Signing of rust release tarballs, so their authentity can be checked.

I think we already do this, i.e. https://static.rust-lang.org/dist/rustc-1.3.0-src.tar.gz.asc. Perhaps we should document this better? What would be needed?

[brson]

System wide LLVM: it worked for us, but now we’ve switched to shipping shared LLVM libraries only, so we need support for linking with shared libs from Rust upstream /see rust-lang/rust#27937/.

Noted.

Installation of multiple rusts. It works for us without multirust, the only thing we need is the ability to install rust libs/binaries to custom dirs. There is one bug in the rust installer that we temporary fix with patch.

Huh. I’m sad to say I’ve looked at this patch for a while and still don’t see what the bug is. Can you explain in more detail what the patch is working around? Is there a filed issue? (I know that --libdir has always been in some state of broken)

Cargo/rust infrastructure support in general: the main problem is how to make cargo use system libraries during production build (it fetches everything and uses fetched versions). This + not very clean linking model (or may be I just do not understand it well enough) makes shipping cargo based packages in gentoo at the moment near impossible. The ideal solution would be if there existed some ‘production’ mode for cargo when it is used just as a nice build system and uses only packages already installed in the system. If no necessary package is found it should just fail.

This is a common theme!

cc @alexcrichton

-Werror switched on during Rust build, ideally it would be good to have a possibility to switch it off (I’m just sedding mk scripts at the moment)

Ah, interesting point. I’ve opened an issue: https://github.com/rust-lang/rust/issues/28599

Additional libraries suffix. At the moment I’m sedding with sed -i -e “s/CFG_FILENAME_EXTRA=.*/CFG_FILENAME_EXTRA=${postfix}/” mk/main.mk, it would be good to have some switch in the build system to do this (it is necessary for the possibility of relible side by side istallation of libs).

Ah, yes, that makes sense. Although we (try to) tie that extra bit to the version number. Perhaps we’re just not capturing enough metadata to differentiate the versions of Rust you are installing together?

In fact, the code looks wrong. It’s hashing an undefined variable…

Would hashing the version number as intended be enough to distinguish your sxs installs?

Edit: After investigation, this does actually hash the version number correctly before CFG_RELEASE is defined. I don’t understand why offhand.

[jauhien]

Signing of rust release tarballs, so their authentity can be checked.

I think we already do this, i.e. https://static.rust-lang.org/dist/rustc-1.3.0-src.tar.gz.asc. Perhaps we should document this better? What would be needed?

Thank you, my fault, I haven’t seen it. May be adding a link to the signature to the download page would be a good idea. The same for sha256 file.

[jauhien]

Huh. I’m sad to say I’ve looked at this patch for a while and still don’t see what the bug is. Can you explain in more detail what the patch is working around? Is there a filed issue? (I know that --libdir has always been in some state of broken)

The problem with these lines is that they lead to failure in case if libdir which was used during compilation starts with lib/ (our case). Also changing libdir during installation is broken anyway, so makes no sense, as it gets hardcoded in the compiler during build by catching env variable.

Ah, yes, that makes sense. Although we (try to) tie that extra bit to the version number. Perhaps we’re just not capturing enough metadata to differentiate the versions of Rust you are installing together?

In fact, the code looks wrong. It’s hashing an undefined variable…

Would hashing the version number as intended be enough to distinguish your sxs installs?

Edit: After investigation, this does actually hash the version number correctly before CFG_RELEASE is defined. I don’t understand why offhand.

It could work for some environments, but not for Gentoo. We have two Rust packages: binary and source. Now

1. you install binary with already defined suffixes (these hashes)

2. you want to install source package (e.g. for testing or whatever else reason, or may be you were bootstrapping using binary package). If suffixes are the same version hashes as in 1, your installation will not work properly. That’s why we are changing suffixes to the custom ones for source packages this way that we have guaranty that only one package with such suffix can be installed in the system (we have one suffix per slot which corresponds to release upstream channel).

[jauhien]

Installation of multiple rusts. It works for us without multirust, the only thing we need is the ability to install rust libs/binaries to custom dirs. There is one bug in the rust installer that we temporary fix with patch.

Huh. I’m sad to say I’ve looked at this patch for a while and still don’t see what the bug is. Can you explain in more detail what the patch is working around? Is there a filed issue? (I know that --libdir has always been in some state of broken)

I have created an issue with detailed explanation and steps to reproduce: https://github.com/rust-lang/rust/issues/28627.

[brson]

The problem with these lines is that they lead to failure in case if libdir which was used during compilation starts with lib/ (our case). Also changing libdir during installation is broken anyway, so makes no sense, as it gets hardcoded in the compiler during build by catching env variable.

OK. Ugh. The way I want this to work is that setting --libdir during configure doesn’t mess at all with the installer’s layout (so this case wouldn’t trip incorrectly with custom-named lib- prefix files), and passing --libdir to the installer puts it in the right place (this is how --mandir works). The obvious problem here is that the relative path to libdir is hard-coded into rustc. Making this work may be more trouble than it’s worth so I left a comment on the bug about a simpler stop-gap.

you want to install source package (e.g. for testing or whatever else reason, or may be you were bootstrapping using binary package). If suffixes are the same version hashes as in 1, your installation will not work properly. That’s why we are changing suffixes to the custom ones for source packages this way that we have guaranty that only one package with such suffix can be installed in the system (we have one suffix per slot which corresponds to release upstream channel).

Hm, I still think this scenario should work if the release channel were hashed, since a default source build is on the ‘dev’ channel, not ‘stable’. Would hashing the channel help? If not, we could add a configure option to mix in more extras. Feel free to submit a PR in this area if it would help you.

On the issue of side-by-side installs, neither @alexcrichton or I are clear on what actually happens when two copies of std, from two different toolchains, are installed to the same location. We believe this should be workable, but don’t know any reason why it would work reliably with the current crate resolver.

[brson]

My next steps here are to summarize what we’ve learned into a plan to resolve the upstream problems in Q4, as well as some guidelines for packagers. Stay tuned.

[brson]

@gus @jauhien @cuviper @alexcrichton et al.

I’m thinking about the requirements for teaching Cargo to allow the local system to override dependencies, and specifically wondering how a packager might extract the library artifacts from a Cargo build in order to put them at the desired location locally (presumably somewhere under /usr/lib). Are you able today to get the precise information out of Cargo to identify which files to copy somewhere? Does Cargo need to do more to make it clear which of its outputs are the packagable artifacts?

[brson]

Another question: if we modify Cargo so that it can use locally-available rlibs, as packaged by the distro, then the distro is going to accumulate a bunch of rlibs in /usr/lib. Because of the way rustc works now those will all be invalidated when rustc is upgraded - rustc will not be able to use them at all, and will likely just pretend they don’t exist because they ‘belong’ to another compiler. The obvious way I see to fix this is to make all Rust crate packages depend on a specific version of the compiler. I know that sucks, but is it doable?

Edit: oh, @gus I see that in your previous thread, you said that rust packages would include the source in Debian, not just the rlibs, and I also recall this was a subject of the anti-vendoring thread in Fedora. I would not expect distro packages of crates to include the source, but instead the rlibs, just globally invalidated in some way when the compiler version changes. Is this difficult to do?

If Debian packages simply dropped the source code onto the system with the expectation that Cargo rebuilds it every time its needed then there will be lots of extra builds.

Does Debian really want to package the source code, and not the rlibs? If it packages the source code where would it put it on the local system for Cargo to reuse? Either way is going to require cargo and/or rustc changes.

Any other distro maintainers have opinions about - given Rust’s compilation model and ABI limitations - whether their packages of Cargo crates will simply install the source code, or the rlibs?

[brson]

@gus Your use case for distributing packaged cargo apps I think works like this:

• User says ‘apt-get install rust-application’
• apt installs the source of all deps somewhere locally (where?)
• apt installs rust-application, and in the process of doing so compiles it and all its deps locally. Cargo automatically knows to use the local source, not the crates.io source.

So the distro package manager, which is usually distributing binaries and not building locally, is actually distributing source code and building locally. Is this right? Is this really what you want to have? Can we skip straight to distributing rlibs and not do this?

[cuviper]

The ABI issues are exactly what makes shipping source-based libraries seem like a lesser evil. As I mentioned, this appears to be what golang packagers are doing in Fedora already.

Yes, you could make a Rust crate package (with rlibs) depend on a specific rustc version. Then you effectively still have to rebuild them all to get a rustc update out into the distro, so I’m not sure it’s really any help. Even then, is there any guarantee that an rlib from rustc-x.y-1 (original release) is compatible with rustc-x.y-2 (perhaps a recompile with some security patches)?

apt installs rust-application, and in the process of doing so compiles it and all its deps locally.

No, that’s Gentoo – I think these would be statically-built binaries. They would have a build-time requirement on the crate-dev package, with its crate source, but the final product would stand alone.

[brson]

Even then, is there any guarantee that an rlib from rustc-x.y-1 (original release) is compatible with rustc-x.y-2 (perhaps a recompile with some security patches)?

No, the opposite is guaranteed They will be forcibly incompatible.