# Re-squash the crates.io repository **URL:** \ **Category:** libs\ **Created:** [October 15, 2019, 11:36pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121 "2019-10-15T23:36:59Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![orf](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/orf/32/6170_2.png) [@orf](https://internals.rust-lang.org/u/orf)\ **Post date:** [October 15, 2019, 11:36pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/1 "2019-10-15T23:36:59Z")
In September 2018 we squashed the [crates.io](http://crates.io) history into a single commit ([Cargo's crate index: upcoming squash into one commit](https://internals.rust-lang.org/t/cargos-crate-index-upcoming-squash-into-one-commit/8440)). Are there any plans to re-squash the index? It's at nearly 80k commits now, and I'm realizing how slow a fresh `cargo build` is when you're in a place with poor internet. Currently at 25% after 20 minutes. ---
**Author:** ![mark-i-m](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mark-i-m/32/3167_2.png) [@mark-i-m](https://internals.rust-lang.org/u/mark-i-m)\ **Post date:** [October 16, 2019, 5:30am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/2 "2019-10-16T05:30:45Z")
Also, is this a tenable implementation? In a hypothetical future where [crates.io](http://crates.io) is as big as npm, would we need to squash every week? ---
**Author:** ![crlf0710](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/crlf0710/32/1987_2.png) [@crlf0710](https://internals.rust-lang.org/u/crlf0710)\ **Post date:** [October 16, 2019, 11:40am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/3 "2019-10-16T11:40:27Z")
I think the root cause is `libgit2` doesn't support shallow clone. So... here we are~ ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [October 16, 2019, 11:59am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/4 "2019-10-16T11:59:46Z")
I'm surprised that rustup doesn't bundle the index with the installation. I presume Rust package could contain a well-compressed `.git` dir that would be smaller (lzma on top of git packs) and download faster (one file instead of git protocol). ---
**Author:** ![Nemo157](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nemo157/32/11585_2.png) [@Nemo157](https://internals.rust-lang.org/u/Nemo157)\ **Post date:** [October 16, 2019, 12:30pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/5 "2019-10-16T12:30:09Z")
I've been meaning to test an alternative, it's possible to push and pull refs referring to git objects other than commits (e.g. [in this repo](https://github.com/Nemo157/github-testing) I stored a blob object directly). So instead of pulling a ref referring to the entire history of the index `cargo` could pull a ref pointing directly at the tree of the current head, this would allow keeping the entire history without squashing in the repo, while only pulling the necessary objects to get the current index instead of the objects required to look at any point in that history. EDIT: The main thing that needs testing is whether github supports partial updates when you only have refs pointing at trees. It's been a while since I looked at the protocol but I think it operated at the commit level, so it might not be capable of ignoring the objects known to both sides if you're just requesting a tree. A probably simpler alternative would be to keep a ref pointing at a separate always-squashed commit, which should be trivial for [crates.io](http://crates.io) to generate and update on each change. ---
**Author:** ![ehuss](https://avatars.discourse-cdn.com/v4/letter/e/9de0a6/32.png) [@ehuss](https://internals.rust-lang.org/u/ehuss)\ **Post date:** [October 16, 2019, 2:02pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/6 "2019-10-16T14:02:09Z")
The intent is to automate the squashing process with a cron job, and we are just waiting for someone on the [crates.io](http://crates.io) team to set it up. More details can be found at [When should we next squash the index? · Issue #47 · rust-lang/crates-io-cargo-teams · GitHub](https://github.com/rust-lang/crates-io-cargo-teams/issues/47). > [@mark-i-m](#): > > Also, is this a tenable implementation? From our current projections, it shouldn't be a serious issue for a long time. > [@crlf0710](#): > > doesn't support shallow clone AFAIK, this is not an option due to the way GitHub's servers work (though that is fairly old information). ---
**Author:** ![crlf0710](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/crlf0710/32/1987_2.png) [@crlf0710](https://internals.rust-lang.org/u/crlf0710)\ **Post date:** [October 16, 2019, 4:16pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/7 "2019-10-16T16:16:03Z")
GitHub server works fine, and official git implementation can deal with that. The libgit2 library still doesn't support it though. See: > > > git can write a file .git/shallow to indicate that the history is cut off at particular commits. We currently do not... ---
**Author:** ![ehuss](https://avatars.discourse-cdn.com/v4/letter/e/9de0a6/32.png) [@ehuss](https://internals.rust-lang.org/u/ehuss)\ **Post date:** [October 16, 2019, 4:51pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/8 "2019-10-16T16:51:23Z")
> [@crlf0710](#): > > GitHub server works fine It is capable of doing shallow clones, but the concern is performance. CocoaPods had to stop using it for that reason as it was causing CPU problems on GitHub's end. That was 2016, though, so I'm not sure if that's still the case. ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [October 16, 2019, 5:23pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/9 "2019-10-16T17:23:34Z")
> [@Nemo157](#): > > A probably simpler alternative would be to keep a ref pointing at a separate always-squashed commit, which should be trivial for [crates.io](http://crates.io) to generate and update on each change. That seems like a promising approach. `git commit-tree` can make such squashed branch in a fraction of a second (it doesn't need to touch any content). ---
**Author:** ![bluss](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bluss/32/2264_2.png) [@bluss](https://internals.rust-lang.org/u/bluss)\ **Post date:** [October 16, 2019, 7:09pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/10 "2019-10-16T19:09:21Z")
Is squashing on every commit wanted? Don't we lose the ability to download small incremental updates then? ---
**Author:** ![comex](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/comex/32/2587_2.png) [@comex](https://internals.rust-lang.org/u/comex)\ **Post date:** [October 16, 2019, 7:45pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/11 "2019-10-16T19:45:54Z")
In the long run, I think it would be best to move the index off of Git entirely. Even with a single commit's worth of history, a Git checkout has to store all files twice (once in .git, once as the working copy), roughly doubling disk space requirements, for no benefit in this use case. @Nemo157's idea of pulling down objects in a nonstandard way might be able to avoid that while still staying on Git... but even a single copy of the index is 70MB and growing. And that‘s basically a waste of both disk space and bandwidth. Instead, why not have a server with an API that takes a list of crates as input, and returns index data for only those crates and their dependencies? It’s not like there’s any speed benefit in caching this data offline, if `cargo update` has to sync the index every time you run it. ---
**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\ **Post date:** [October 16, 2019, 7:51pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/12 "2019-10-16T19:51:11Z")
> [@comex](#): > > a Git checkout has to store all files twice (once in .git, once as the working copy) Is that fundamental? I thought you could do a bare checkout that doesn't have a working copy and still read blobs out. ---
**Author:** ![cuviper](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cuviper/32/1897_2.png) [@cuviper](https://internals.rust-lang.org/u/cuviper)\ **Post date:** [October 16, 2019, 7:54pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/13 "2019-10-16T19:54:36Z")
> [@comex](#): > > Even with a single commit's worth of history, a Git checkout has to store all files twice (once in .git, once as the working copy), roughly doubling disk space requirements, for no benefit in this use case. Cargo has used a bare repo without a working copy for a while now: > > > This commit moves working with the crates.io index to operating on the git > obje…ct layers rather than actually literally checking out the index. This is > aimed at two different goals: > > \* Improving the on-disk file size of the registry > \* Improving cloning times for the registry as the index doesn't need to be > checked out > > The on disk size of my \`registry\` folder of a fresh check out of the index went > form 124M to 48M, saving a good chunk of space! The entire operation took about > 0.6s less on a Unix machine (out of 4.7s total for current Cargo). On Windows, > however, the clone operation went from 11s to 6.7s, a much larger improvement! > > Closes #4015 ---
**Author:** ![Nemo157](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nemo157/32/11585_2.png) [@Nemo157](https://internals.rust-lang.org/u/Nemo157)\ **Post date:** [October 16, 2019, 8:01pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/14 "2019-10-16T20:01:47Z")
It needs testing, but I think by having a long-running branch that each from-scratch commit is merged into github will still be able to run the algorithm to determine which objects are shared and only send a diff. (The issue may again be that this is only checked at the commit level, but fundamentally the protocol used for fetching is capable of this). ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [October 17, 2019, 12:03am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/15 "2019-10-17T00:03:24Z")
As the index grows it seems inevitable that it will get replaced eventually with some kind of query API, but the current solution has some nice properties: - Cargo finally supports `--offline` flag, and there's also `-Z no-index-update`, so it is possible to edit deps while offline. - The standard index format made it easier to have [cargo-lts](https://lib.rs/lts) that can "rewind" the index to an older version and remove incompatible crates. - Having full history enables tools like [cargo-tally](https://lib.rs/crates/cargo-tally). - The index is very helpful for [https://lib.rs](https://lib.rs). The parts that have to use crates-io HTTP APIs are much slower, since crates-io wouldn't like getting 30000 requests per second 🙂 I'm slightly worried that if Cargo switches to a query API, the unused index will become very low priority for the crates-io team, and will either get shut down outright, or limp unmaintained. ---
**Author:** ![kentnl](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kentnl/32/6165_2.png) [@kentnl](https://internals.rust-lang.org/u/kentnl)\ **Post date:** [October 17, 2019, 6:47am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/16 "2019-10-17T06:47:17Z")
I suspect the [crates.io](http://crates.io) repository could utililize "replacement" refs, where-in, when they create a squash of commit-ASDF -\> root as QWER , they create a replacement ref that maps QWER/ASDF. Then anything that elects to fetch the replacement refs (non-default) will get the full history, and git tools that opt-in to veiwing replacement refs will see the full history uninterrupted. ---
**Author:** ![NickeZ](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nickez/32/2689_2.png) [@NickeZ](https://internals.rust-lang.org/u/NickeZ)\ **Post date:** [October 17, 2019, 7:07am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/17 "2019-10-17T07:07:16Z")
I think you would need to cache it so that it works with `cargo build --offline`, right? ---
**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\ **Post date:** [October 17, 2019, 7:15am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/18 "2019-10-17T07:15:27Z")
Note that a query API isn't everything -- Nuget has one of those, but using Azure DevOps's built-in `dotnet restore` task still somehow ends up running a 1-minute cache update in most of my builds. > [@comex](#): > > but even a single copy of the index is 70MB and growing Hmm, I pulled up WinDirStat, and that 70MB is completely irrelevant compared to the overall size of my `.cargo` folder: ![image](https://us1.discourse-cdn.com/flex002/uploads/rustlang/original/2X/0/06243989af736b41a4b575351ff55027f1b4f4ae.png) Am I in some horrible state that I should be re-installing, or is this normal? ---
**Author:** ![pietroalbini](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/pietroalbini/32/10694_2.png) [@pietroalbini](https://internals.rust-lang.org/u/pietroalbini)\ **Post date:** [October 17, 2019, 7:31am UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/19 "2019-10-17T07:31:42Z")
> [@kornel](#): > > The index is very helpful for [https://lib.rs](https://lib.rs). The parts that have to use crates-io HTTP APIs are much slower, since crates-io wouldn't like getting 30000 requests per second By the way, we started producing daily database dumps of [crates.io](http://crates.io) (only with public data of course), and every use of the API _should_ be covered by it: [https://crates.io/data-access](https://crates.io/data-access) ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [October 17, 2019, 1:05pm UTC](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121/20 "2019-10-17T13:05:04Z")
That's wonderful! Thanks for this. [Next page](https://internals.rust-lang.org/t/re-squash-the-crates-io-repository/11121.md?page=2)