Future updates to the rustup distribution format

[mmstick]

You should look into using zstd instead of xz for compression. It has a high compression ratio, almost to XZ-level compression, but it compresses 60x faster than XZ and decompresses 10x faster, getting about a quarter the decompression speed of LZ4. It is truly a next generation compression standard.

[keeper]

The new compression format will be used for all of the available rust releases, so XZ is the only real choice since it’s (almost) everywhere. It will certainly be interesting to see if any distros pick up these new compression formats for their own package distribution, then there might be a new standard.

[Ericson2314]

Don’t forget Cabal too

[Ericson2314]

@brson I just skimmed this for now, but my initial thought is that if we are planning this far ahead, we might as well go all the way and decide rustup is just responsible for delivering rustc, and cargo (optionally) download cached builds of individual crates.

Kind of like what @steveklabnik says, Cargo needs TUF too regardless of rustup, and while it would be great to somehow implement it for both tools I suspect the code reuse gains here are slim as as crates.io and static.rust-lang.org have little in common. So I’d say go for phase 1 as written, but then adopt phase 2 to what I said so we only implement TUF once. (perhaps Cargo can cache executables too, and rustup just installs the binary Cargo acquires in accordance with TUF).

[mmstick]

All Linux distributions already support zstd. Debian, Ubuntu, Fedora, Arch Linux, and Gentoo all offer libzstd. XZ isn’t supported any more than zstd.

[mitchell]

If this can be implemented as a separate package

This https://github.com/Phaiax/rsaltpack library looks like a good start on a reusable NaCl implementation. As long as its data format satisfies TUF’s spec, or something close enough to it, then it could save a comparably significant portion of TUF’s python code for signing, verifying, and generating keys.

[keeper]

Does it come installed in the default minimal installation? It’s the same problem as GPG signing, if it isn’t installed by default, then rustup can’t rely on it.

[mmstick]

There aren’t any minimal installs that I know of which features XZ support by default. Additionally, considering rustup requires network access to install, I fail to see why it’s a problem to add a check for and even automatically install libzstd. There’s official support for libzstd in Rust already, so it’d be trivial to implement.

[DanielKeep]

I might be off base here, but I thought the intention was to add the new compressed archives in addition to the existing ones, not instead of. As such, since rustup is a statically-linked executable, how does it matter whether or not the compression library is likely to already be installed?

[bascule]

You don’t need anything like saltpack. I’d also note that saltpack is “homebrew crypto” (ideally it’d be using something like Rogaway’s STREAM construction), and that crate does not appear to be in anything close to a good state.

All TUF relies on are hash functions and digital signature algorithms. There are already plenty of implementations of the former. I would recommend using Ed25519 for the latter. There’s a pure Rust implementation available in rust-crypto as well as wrappers for the ref10 C code in both sodiumoxide and ring.

[briansmith]

All TUF relies on are hash functions and digital signature algorithms. There are already plenty of implementations of the former. I would recommend using Ed25519 for the latter. There’s a pure Rust implementation available in rust-crypto as well as wrappers for the ref10 C code in both sodiumoxide and ring.

I agree with this.

TUF is a pretty complex thing to implement. I recommend that you sort out the key management issues before you implement TUF. For example, how are you going to manage the HSM/smartcards or whatever that hold the keys? If so, are you going to need to use use RSA or ECDSA signatures since very few (no?) HSMs and smartcards support Ed25519? Or, are you going to use the threshold mechanism as a means to avoid needing HSMs/smartcards. Are keys held by individual developers or are the keys held in a central access-controlled signing server? (IMO, it is better to use both smartcards and the threshold mechanism if there is no central access-controlled signing server.) What happens w.r.t. key replacement when a developer becomes untrusted for whatever reason?

You will probably find that this key management stuff (the stuff in the previous paragraph) takes a surprisingly large amount of time to get right.

[eternaleye]

The difference between XZ and zstd is that while zstd is widely available, XZ is widely accepted. It’s considered an acceptable compression format for the data.tar member of a deb, it’s now the primary compression format GNU uses for source releases, etc.

No one would be surprised at having software distributed in XZ, but zstd is a new and unfamiliar thing. PR is a factor to consider, and the gain from using zstd over XZ is not huge.

[mmstick]

Why should PR even matter in the first place? The compression technology used is just an implementation detail that’s invisible to the user of rustup. In addition, Rust cannot state that it is widely accepted either, so it’s silly to consider what’s widely accepted. Libraries can only become widely accepted when software actually uses the libraries. Let’s be among the next generation applications that are using libzstd so that we can push zstd forward as a widely accepted standard on Linux.

Finally, zstd does offer a huge gain over XZ if you’ve seen the benchmarks, especially with decompression speed. In fact, it’s from the same author of LZ4.

[cuviper]

XZ is slower, but is it slow enough to worry about it? How much time is spent in decompression versus plain IO time writing all the new files?

[arielb1]

You should look into using zstd instead of xz for compression. It has a high compression ratio, almost to XZ-level compression, but it compresses 60x faster than XZ and decompresses 10x faster, getting about a quarter the decompression speed of LZ4. It is truly a next generation compression standard.

rustc decompresses in a few seconds anyway - the only time installation is slow is when network conditions are suboptimal, and in that case we want high compression ratio, not high decompression speed.

And according to zstd’s web-site (http://facebook.github.io/zstd), zstd is still 10% larger than xz. As xz is the current standard and zstd does not provide a significant advantage, we should be using xz.

[keeper]

The compression format being added isn’t just for rustup, it will be used by all forms of rust distribution. And from the sounds of it, just adding a single format is a lot of work. So adding a format that is perceived as a widely accepted standard feels like the right move.

Also just a side-note, rustup should not install packages on the users system. Besides requiring administrative access, it makes too many assumptions about the user’s platform.

[dlmetcalf]

The other issue with zlib, is that it’s not open and use of it involves additional license provisions that could block Rust adoption in many organisations. Facebook grants you patent indemnity over the patents involved in zlib (alone), but in return requires that you provide Facebook patent indemnity over any and every patent claim you might ever have against them (for any cause) in perpetuity. If the choice were to come between (a) use Rust and put all of a company’s patent IP at risk (if the zlib dependency were to be introduced), or, (b) Block Rust and protect the larger IP base; then almost all companies and many individuals, are going to choose option ‘b’ every time.

https://github.com/facebook/zstd/blob/dev/PATENTS

[dlmetcalf]

Correction: I was referring to libzstd, not zlib (as typed).

[mmstick]

Facebook has done away with the patent issue, so that’s no longer a problem.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.