That was very interesting.
Given that it is a subscriber link, do I have the right to share it on reddit (more specifically r/rust and maybe r/programming)?
The LWN FAQ describes it like this:
If I were you, I’d put it on r/rust, but not r/programming.
I was what I was going to plan, and only if it as a lot of popularity on another subreddit than r/rust. About a year ago, drewdevault posted a similar article but praising how linux distro handle package (and btw It was a good idea that I checked, it was on r/linux not r/programming).
That being said, I find it funny how much better the situation is on archlinux as pointed by one of the comments.
Very interesting, especially in the light of You should prepare for the OpenSSL 3.x secvuln - Xe. OpenSSL will disclose a major vuln on Thursday, so everything linking to OpenSSL statically would have to be re-build.
Luckily, it seems that only 3.x series is affected (though there's some ambiguity about that), and rust-openssl is still on 1.x, so it looks like the Rust ecosystem would be spared, albeit for a bad reason.
But it's a useful mental experiment to imagine what would need to happen if a bunch of Rust binaries binaries needs to be rebuild. For example, running strings ~/.cargo/bin/* | rg "openssl" on my laptop uncovers
cargo-deny
cargo-outdated
sccache
as problematic...
This boils down to Linux package managers being designed around dynamic libraries (which has its benefits, but also limitations) and unwilling to change their architecture to support other approaches.
For Rust, the right model would be something like a database of all build artifacts and their dependencies gathered from Cargo.lock files or compiler's JSON output. That database could be used to look up everything that depends on an insecure dependency, and then rebuild with something like sccache to reduce waste.
On reddit, there was an interesting comment by r/Kuhluh, and I don’t have the answer:
After also reading the linked Emails, there is one thing which strikes me as odd:
This is true, and probably also not "fixable". We need to make some amount of non-upstreamable patches to some crates (most notably, removing Windows- or mac OS-specific dependencies, because we don't want to package those), but in some cases, these are "incompatible" changes, and Rust developers should not be targeting our downstream sources that have these differences with actual upstream sources. This is due to a limitation of how cargo handles target-specific dependencies - all dependencies that are mentioned in any way need to be present for it to resolve dependencies / enabled optional features / update its lockfile etc. But since we don't want to package bindings for Windows and mac OS system APIs, we need to actually patch them out, otherwise builds will fail.
Why exactly do I need the Windows-specific parts when I want to build for Linux exactly?
I think it goes this way:
- Cargo.lock should contain the union of all dependencies. We want
cargo generate-lockfileto give exactly the same result irrespective of the platform the command is being run on. - Even when Cargo.lock already exists, every Cargo operation needs to check that it is fresh.
- To check that Cargo.lock is fresh, Cargo needs to verify that it indeed satisfies all declared dependencies
- To check that, Cargo needs to parse Cargo.toml of all packages (including transitive dependencies), irrespective of whether the dep can be activated for the current platform.
That seems to be false. You can fetch and build with only linux dependencies, in a brand new project:
> cargo add clap
Updating crates.io index
Adding clap v4.0.18 to dependencies.
> CARGO_HOME=$(pwd) cargo fetch --target x86_64-unknown-linux-gnu Updating crates.io index
Downloaded atty v0.2.14
Downloaded strsim v0.10.0
Downloaded os_str_bytes v6.3.1
Downloaded bitflags v1.3.2
Downloaded libc v0.2.137
Downloaded clap_lex v0.3.0
Downloaded termcolor v1.1.3
Downloaded clap v4.0.18
Downloaded 8 crates (900.8 KB) in 0.58s
> CARGO_HOME=$(pwd) cargo check --offline --locked
Compiling libc v0.2.137
Checking os_str_bytes v6.3.1
Checking bitflags v1.3.2
Checking strsim v0.10.0
Checking termcolor v1.1.3
Checking clap_lex v0.3.0
Checking atty v0.2.14
Checking clap v4.0.18
Checking foo v0.1.0 (/tmp/tmp.gpcVDotZ2y/foo)
Finished dev [unoptimized + debuginfo] target(s) in 1.66s
> ls -1 registry/cache/github.com-1ecc6299db9ec823
atty-0.2.14.crate
bitflags-1.3.2.crate
clap-4.0.18.crate
clap_lex-0.3.0.crate
libc-0.2.137.crate
os_str_bytes-6.3.1.crate
strsim-0.10.0.crate
termcolor-1.1.3.crate
> CARGO_HOME=$(pwd) cargo check --offline --locked --target x86_64-pc-windows-msvc
error: failed to download `winapi v0.3.9`
Caused by:
attempting to make an HTTP request, but --offline was specified
Riiight, Cargo doesn't actually need to parse Cargo.toml to learn about dependencies to verify lockfile, it reads this information from a registry. So you don't have to have the source of all crates, just the registry info.
I think Fedora uses a local registry with all packaged crates, not the crates.io, so, for this to work on Fedora, they need to make sure their local registry includes info about windows-only crates and such.
Ah yes, that makes sense, and reminds me of what I had to do to be able to generate a registry DAG by stripping things like development and unactivated optional dependencies.
It would be nice if there was some support for "use-case specific lockfiles" that encode just the subset of the lockfile necessary for a specific operation (stripping out build/dev and target-specific dependencies). Able to be generated from an existing "full lockfile".
Fedora uses source replacement with directory = "/usr/share/cargo/registry", so cargo can dynamically discover which rust-crate-devel packages are installed.
IMHO, this boils down to that system package managers are built to distribute applications. It just so happens that dynamic libraries on *nixes look a lot like applications; the difference is just whether they expose a user interface (textual or graphical) or just a binary interface. The "unix philosophy" is a bunch of specialized applications to perform specific tasks, and dynamic linking is the implementation of that that doesn't require constant de/serialization through the shell.
Rust libraries can't be made (as easily) to look like applications. So trying to individually package Rust libraries the way C libraries are packaged isn't going to work without modification of the packaging process. Monomorphized generics simply break down the clean application boundary that dynamic linking is trying to preserve.
I agree with @kornel here — porting the distro model to Rust libraries looks less like the distro model and more like sccache plus a system for queueing rebuilds of downstream when upstream changes. The statically linked applications can of course still be distributed through the standard distro mechanism.
Part of the problem is that, for example, plenty of crates have unconditional dependencies on winapi, because winapi acts as an empty crate on non-windows (its first line of code is #![cfg(windows)]).
We use GitHub - coreos/cargo-vendor-filterer: Tool to `cargo vendor` with filtering for this problem.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.