# Crates.io bulk downloads **URL:** \ **Category:** tools and infrastructure\ **Created:** [February 20, 2017, 4:43pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821 "2017-02-20T16:43:41Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![est31](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/est31/32/2551_2.png) [@est31](https://internals.rust-lang.org/u/est31)\ **Post date:** [February 20, 2017, 4:43pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/1 "2017-02-20T16:43:41Z")
Would it be possible to provide dump files containing all crates on [crates.io](http://crates.io)? This would be very helpful for: - Scientists wanting to analyze the data - Future mars colonists. Their ping times will definitely suck! - Archivists - Digital preppers preparing for the next apocalypse where hacked thermostats DDOS level3 or something - Generally anyone who doesn’t want to rely on the internet for everything, e.g. who wants to be able to use Rust while being completely offline e.g. on planes/trains/etc. - People in remote places like behind the great firewall of china, or on the falkland islands (falklands only have satellite internet, can you imagine that?). They obviously can’t download a dump, but one can download it once and then use it to set up a mirror. Other projects like Wikipedia [offer similar dumps already](https://dumps.wikimedia.org/). I’ve recently used [crates-mirror](https://github.com/tennix/crates-mirror) to get 7997 crates of the 8089 crates currently on [crates.io](http://crates.io). But this method is not perfect. You have to download each crate separately, leading to a non trivial overhead in HTTP/TLS negotiations. Also, obviously, it might get in trouble with anti abuse systems. My download of [crates.io](http://crates.io) and the directory with the .crate files right now takes 6 GB storage space. So its of definitely manageable size. In comparison: wikipedia dumps take tens of gigabytes. And this is before very easy optimisations are applied, like only storing the diffs between .crate files. Implementation wise it could be e.g. a weekly job, that creates combined .tar.gz/zip files, and uploads them to static.rlo or somewhere else. As [crates.io](http://crates.io) is addition only, its okay to delete old dumps. ---
**Author:** ![alexcrichton](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alexcrichton/32/4501_2.png) [@alexcrichton](https://internals.rust-lang.org/u/alexcrichton)\ **Post date:** [February 21, 2017, 4:50pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/2 "2017-02-21T16:50:36Z")
FWIW downloading each one individually should be fine, we’re not gonna start blocking your IP or anything like that 🙂 ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [April 29, 2018, 11:30am UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/3 "2018-04-29T11:30:02Z")
What about [crates.io](http://crates.io) metadata that isn’t part of the crates.io-index repository? AFAIK there’s no dataset for owners and download stats. Is it OK to scrape that data? ---
**Author:** ![sgrif](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/sgrif/32/3339_2.png) [@sgrif](https://internals.rust-lang.org/u/sgrif)\ **Post date:** [April 29, 2018, 12:41pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/4 "2018-04-29T12:41:51Z")
We will be publishing an official crawler policy soon. We’re also looking to start producing dumps of our dataset at some point in the future. ---
**Author:** ![harrychin](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/harrychin/32/2986_2.png) [@harrychin](https://internals.rust-lang.org/u/harrychin)\ **Post date:** [April 30, 2018, 11:26pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/5 "2018-04-30T23:26:24Z")
It would be interesting research to see if [crates.io](http://crates.io) + GitHub had enough data to train an idiomatic Rust API snippet creator like this one for Java: [http://www.askbayou.com/](http://www.askbayou.com/) ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [April 30, 2018, 11:36pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/6 "2018-04-30T23:36:27Z")
You can get all of the code already. There’s a crate for downloading _all_ code from [crates.io](http://crates.io): [https://github.com/weiznich/crates-mirror](https://github.com/weiznich/crates-mirror) or [https://github.com/C4K3/crates-ectype](https://github.com/C4K3/crates-ectype) For analysis I suggest modifying it to download only the latest version of each crate rather than all versions. ---
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [April 30, 2018, 11:38pm UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/7 "2018-04-30T23:38:40Z")
Personally I'm interested in download numbers & ownership, as I want to play with crate ranking algorithms and analyze e.g. how much popularity and authors' experience is correlated with various crate quality metrics. * * * ▶ **EDIT: it's here!** [https://crates.io/data-access](https://crates.io/data-access) ◀ ---
**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\ **Post date:** [March 25, 2019, 8:27am UTC](https://internals.rust-lang.org/t/crates-io-bulk-downloads/4821/8 "2019-03-25T08:27:59Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.