The downloads should be shared from ~/.cargo/registry/cache, as long as you're not setting CARGO_HOME to something else.
The build artifacts in each project's target/ directory are a different story, and that's probably what's eating so much of your space. You can see some discussion in this other thread: