Downloading all the crates on crates.io

Pietro Albini Written on April 9, 2020.

There are a lot of reasons you might want to download all the crates ever uploaded to crates.io, Rust's package registry: code analysis across the whole public ecosystem, hosting a mirror for your company, or countless other ideas and projects.

The team behind crates.io receives a lot of support request asking what's the best and least impactful way to do this, so here is a little guide on how to do that!

Getting a list of all the crates

crates.io offers multiple way to interact with its data: the crates.io-index GitHub repository, experimental daily database dumps and the crates.io API.

The way I recommend to get the list of all the crates is to rely on the index: the experimental database dumps are more heavyweight and are only updated daily, while usage of the API is governed by the crawlers policy (limiting you to one API call per second). If you absolutely need to use the API please talk with us by emailing help@crates.io, and we'll figure out a solution.

The index is a git repository, and the format of its content is defined by RFC 2141. There are crates such as crates-index that allow you to easily query its contents, and I recommend using them whenever possible.

Downloading the packages

The best way to download the packages is to fetch them directly from our CDN. Compared to calling the crates.io API, the CDN does not have rate limits and is faster (as the API redirects you to the CDN after updating the download count). The CDN URLs follow this pattern:

https://static.crates.io/crates/{name}/{name}-{version}.crate

For example, here is the link to download Serde 1.0.0. Packages are tar.gz files.

If you want to ensure the contents of the CDN were not tampered with you can verify the SHA256 checksum of the file you downloaded by comparing it with the cksum field in the index.

Keeping your local copy up to date

The best way to keep your local copy up to date is to fetch a fresh list of crates available on crates.io and check if all of them are present in the local system, downloading the ones you're missing. I personally recommend this approach as it's less error-prone, and it heals your copy automatically if for whatever reason some of the changes are lost during a previous update.

Another interesting approach you could implement is to get the difference since the last update of the index with git diff, parsing its output to get the list of crates that were added. There are also third-party crates such as crates-index-diff that automate this process for you. This approach is more fragile and error-prone, but it might be the only sensible solution if checking whether you downloaded a crate or not is slow or expensive.

Common issues to be aware of

While the basics of downloading the contents of crates.io are simple, there are a couple of issues to be aware of when implementing such tooling: