Perfecting Rust Packaging

There is no automatic way to deduce this currently, but I think this is obvious enough for humans and that's ok. Eg: for a "source based" library you just copy the entire source tree; for some executable app you just copy the (few) output binaries; for some hypothetical "binary rlib/dylib library" package you would just copy the (few) rlibs/dylibs.

If/when packages start including complex application data as well, then yeah we'll need to make a more powerful "cargo install" to work out what goes where - but right now I think this is a low priority.

Yep, this is technically easy to do in the packaging metadata. As you go on to discuss, the implications of this are that we'd need to rebuild every Rust library package when a new rustc was released.

Just to complete this line of thought, we could in fact package and reuse dylibs by using similar tight restrictions at the distro packaging metadata level and it would work just fine with the same caveat that we have to rebuild everything (this time applications too) with each rustc release.

My proposal (and current plan) is to package libraries as source and not distribute rlibs/dylibs at all, for the sole reason that this format is more portable across compiler revisions. The downsides are additional cpu cycles at (application package) build time, we need to rebuild all affected application packages whenever a library is updated, and there will be some library out there somewhere that has a license that won't let us ship source (but I don't care about that right now).

I haven't thought too much about "plugins" yet (both rustc compiler plugins and any project where the "deliverable" is a .so library, like a hypothetical pam module written in Rust). I have a suspicion they might require a tight version requirement on the compiler or std dylibs and perhaps need to be rebuilt on every compiler release. Provided the number of such packages is small, we can deal with that.

Does Debian really want to to package the source code? No. We're just looking for what looks like the best tradeoff within the current limitations of the Rust toolchain and ecosystem. I expect/hope this will evolve quite a bit as we get more Rust applications "in production" and the ABI stability story matures. (My beard is showing, but yes I remember the a.out -> ELF transition that C-on-Linux went through for basically all the same reasons :wink:

As @cuviper also clarified, no this isn't correct. Debian (and just about every other distro - notably not gentoo) have a clear distinction between "source" packages and "binary" packages (Debian nomenclature, but the idea is the same in Redhat, etc).

Note in particular that "binary packages" often include libraries - it's anything that is the "output" of the package build process. The separation of pre- and post- build also results in a sharp distinction of "build-time" ("build-deps" in Debian speak) and "run-time" dependency relationships between packages. I expect the "binary package" jargon and the fact that they might include "libraries" is confusing, and I wish I had a whiteboard handy to draw boxes with arrows.

Each upstream project is bundled up as a "source" package (which typically contains source, duh), and the distro machinery centrally compiles that into (possibly multiple) "binary" packages (which typically contain binary executables, shared libraries, or data/config files of some sort). Regular end users download and install binary packages only. This is good because it doesn't require CPU on the user end, and the run-time dependencies are typically much fewer/simpler than the build-time dependencies.

In "source based distros" (Gentoo is the major example, but also OpenEmbedded/buildroot/etc), users download the source packages and do the compile locally - with the help of the packaging tools. Requires lots of CPU, but allows them to have enormous flexibility in exactly how that gets built. The embedded folks like this because they can get the ultimate size and flexibility in their output. I've never understood why Gentoo users do it :wink:

So: my "source-based Rust libraries" plan is to have:

  • Debian "source" packages that include whatever library/application Rust source.
  • The Debian "binary" package for a Rust library will just be the source, installed in a known directory somewhere (ie: the Debian package build step is basically a no-op).
  • The Debian package for a Rust application will build-depend on (probably several) Rust library packages. The build-deps will ensure the Rust library packages (sources) are installed before building.
  • The application package build step will run rustc (via cargo) to compile the application and all the relevant library sources. The resulting (statically linked) executable goes into the application's Debian binary package.
  • Note the application package has no run-time dependency on the libraries. When the end user "apt-get installs" the application package, they get just the statically linked executables, with no need to ever know about the library packages.
  • A security fix in one of the library packages requires a rebuild of any application packages that build-depend (at whatever depth) on the library package (this is visible in the Debian metadata).
  • A new rustc doesn't require anything to be rebuilt, but we do need to ensure that all applications are able to be rebuilt.

This means: The rust-library "binary" packages will be basically identical to the rust-library "source" packages, except with different path prefix, and probably patches to Cargo.toml applied, etc. rlibs won't be used anywhere, except libstd.

(Apologies for my posts being so lengthy, it's hard to gauge how much background is already understood without body language.)

2 Likes