PSA: libz-sys, flate2, and git2 now support the high-performance zlib-ng

libz-sys provides Rust bindings to the zlib compression library, for handling DEFLATE/gzip compression. I've just released libz-sys 1.1.0, which provides opt-in support for the high-performance zlib-ng library in zlib-compat mode.

zlib-ng provides 2-3x performance improvements for compression, and ~20% improvements for decompression.

Most people use libz-sys via the higher-level flate2 crate; you can opt into zlib-ng for flate2 with:

flate2 = { version = "1.0.17", features = ["zlib-ng-compat"], default-features = false }

The git repository format also uses DEFLATE compression extensively, and the git2 Rust crate now supports zlib-ng as well:

git2 = { version = "0.13.9", features = ["zlib-ng-compat"] }

Note that this will build and link a vendored version of zlib-ng, rather than using the system shared version of zlib.

If you build a crate that wraps a C library, and you use libz-sys to provide a version of zlib, please don't opt into zlib-ng by default; instead, use libz-sys = { version = "1.1.0", default-features = false, features = ["libc"] }, which allows zlib-ng but doesn't require it. Optionally, you can provide a zlib-ng-compat feature by adding zlib-ng-compat = ["libz-sys/zlib-ng"] to your features section; alternatively, people who depend on your crate can also depend on libz-sys themselves and enable the zlib-ng feature.

Please give this a try, and comment if it improves the performance of your own crates! If you encounter any issues, please report them via github on the appropriate crates.

15 Likes

Do you per chance have numbers about the performance gap between pure Rust implementation (miniz_oxide) and zlib-ng?

At default compression, miniz_oxide is a little slower than stock zlib; at fast compression, miniz_oxide beats zlib but not zlib-ng; at maximum compression (which you almost never want to use) miniz_oxide is just barely slower than zlib-ng. Here are some performance comparisons I made: https://github.com/Byron/gitoxide/issues/1#issuecomment-672626465

8 Likes