By quickly glancing at crates.io CI, it looks like the cache is busted on every build:
I think this should not happen in an ideally configured travis cache (ie, travis is optimized for stable caches). Here's how this step looks in an average rust-analyzer build:
The difference is time is 3 minutes!