Changes to how crates.io handles index updates

Last Friday (March 8), we deployed a change to crates.io that affects how the index is updated. Previously, we updated the index synchronously on the web server before sending a response to cargo. This is no longer the case.

When you run cargo publish or cargo yank, the index update is now queued to be run asynchronously, off the web server. This means that cargo publish returning with no error does not mean the newly published version can be used from Cargo.toml.

This should not be noticeable the majority of the time. The delay is typically less than 1 second, and the longest we’ve observed is around 3 seconds. We are hoping to lower this further in the near future. Typically the delay between cargo publish returning and the index being updated is less than the the time it takes to type cd other_crate && cargo build. However, if you are publishing multiple dependent crates from a script, you may need to add a sleep or retry behavior to account for the delay. Typically if you wait a second and re-run the command, it will succeed.

Note that in the case of a network issue the delay might be higher than usual. If GitHub or our hosting service are experiencing an outage, it might be minutes or longer. We’re still tuning the retry behavior, so in the event of a spurious issue you may see longer delays in the short term. However, when everything is running smoothly, you should not see a delay longer than 1 second on average and 5 seconds maximum. We’re planning to expose more of this information through the UI in the next few weeks.

Our system is configured to page whoever is on call if a crate goes too long without updating the index. However, if you’re having a problem after publishing or yanking a crate, please don’t hesitate to email help@crates.io

If you have any questions, feel free to ask here or email help@crates.io

8 Likes

Are the changes synchronized in the web server, at least? i.e. if the same crate-version is attempted to be published again before the index has updated, will there be an error?

Will crates be lost if the web server crashes before completing the index update?

1 Like

Are the changes synchronized in the web server, at least? i.e. if the same crate-version is attempted to be published again before the index has updated, will there be an error?

Yes, we update our database immediately and you will be able to see the published crate in our UI even if the index hasn't yet been updated.

Will crates be lost if the web server crashes before completing the index update?

No, they are stored persistently and handled by a completely separate machine. One of the motivations for this change was to fix an extremely rare issue where the server would die after the index was updated, but before the database transaction committed leading to the database and index being out of sync.

With this change the index will always eventually be in sync with the database (unless jobs are failing continuously in which case we get paged)

7 Likes

It makes sense for the API to return immediately, but sleeping against races in upload scripts sounds suboptimal.

Can cargo upload learn which version (ETag for inspection in HEAD, or similar) of the index the processed crate will be in, and (possibly gated behind a --sync option) slowly poll (eg. 1s initial delay, doubling delays) for an index that contains the relevant information?

6 Likes

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.