If it’s local to your machine (or even your physical network), you can (hopefully) trust anyone who would be putting a build in the cache, as they either have access to your machine or your physical network. Both are basically game over security-wise even without a binary cache for cargo.
If it’s on the internet, it’s a similar process to trusting the source code you’re downloading. Check a hash to make sure you have the correct thing, and check to see if people you trust trust that binary (or source code) using something like
If we assume cargo has reproducible builds, you could have a build bot that builds the crate locally and checks against the prebuilt, and if they’re the same, trusts it to be the binary resulting from building that source code.
It’s all a trust problem. For this though, I hope you can trust yourself