Stumbled upon this blogpost today, and it immediately made me wonder if applying this to Rust tooling would yield significant benifit to compile times, and more importantly - rust-analyzer startup times.
At a glance, it feels like it would be, with how common it is to have the total number of direct+transitive dependencies in the hundreds.
The blog post makes it sound like the gain came from the .tar part of the .tar.gz, rather than the .gz part – the overhead was due to the number of files that needed to be opened, rather than the number of bytes that needed to be read, so having one file with them all concatenated reduced I/O overhead because the kernel could copy them all in one chunk. (Using uncompressed .tar might potentially also make an mmap approach faster than a read approach – mmap is generally slow when working with lots of small files, but using a single .tar gives you one large file.)
I wouldn't expect it to have that much impact on a Rust compile, though – the change improved the speed of the lexer by 2.3×, but lexers are already very fast and would be only a very small non-bottlenecking part of a typical Rust compile. I imagine the total impact taking into account slower parts of the compile would be minimal (probably even with check builds, even though they skip the slowest parts).
Yeah, gzip is slow enough that if you're just going for raw speed it's probably not worth it. That's where something like LZ4 (compression algorithm) - Wikipedia comes in -- though of course doing that in the filesystem instead is often better than making the application layer deal with it.
But of course this isn't really anything new. This is why games -- especially on windows -- have used custom archive formats for assets for 30+ years.
I should probably edit the title of the thread to avoid making an impression like I'm actually suggesting gzip. Though, if that was an option, I know it would see some use for the space savings.
On many file systems you can enable transparent compression on the file system level. BTRFS supports this (with selectable compression algorithm, including zstd), as does NTFS on Windows (with a single fixed algorithm). I think ZFS might support it too, not sure. I have no clue about MacOS X.
This is generally better than handling compression at the application level, as it will be uncompressed into the page cache, which will mean mmap still works and that allows multiple programs to share the same uncompressed pages.
Also: if you are to use archives (primarily useful for Windows and maybe Mac, Linux punishes you much less for many small files), consider something akin to zip instead of tar. Tar doesn't have random access (even when uncompressed). There is no directory listing in it, it is more like a linked list of records.
Zip on the other hand does have a directory listing (though it is at the end of the zip file and a bit weird). I believe each file is compressed individually, which allows for random access as well.
Zip also has uncompressed mode, which means you can get the whole package in one mmap, and then have random access.
Yup, that's how I know of LZ4 -- it's the default compression in ZFS because it's so fast that it's essentially always a win, whereas stronger compression isn't necessarily worth it depending on what you're usually storing in the pool.
There was some talk along these lines previously though not from the performance angle: "Jar" for Rust: single file crate support for `rustc`
Oh, looks like it is about performance also.
Interesting discussion, doesn't look like anyone touched on the idea that this could be particularly useful for Rust-Analyzer.
Also the decompression impact discussion has the blind spot around the middle ground between a bunch of files and decompression in rustc: if .crate is a gzipped archive, the decompression could happen at cargo fetch time, then other tooling would work with contiguous .crate files.
I would like to see benchmarks of this (across all the major host platforms used for development). And consider the downsides too. Currently RA can (recursively) navigate to any code in a dependency or std when I ctrl-click in vscode. Would that still be possible in a zip file? I don't know that all the popular editors are able to seemlessly browse into archives.
I consider this incredibly powerful, and it is something I miss when working in C++, where I get to a header and that is it for system dependencies. I would not consider it worth giving this up for any speed advantage whatsoever.