The idea to distribute pre-built binaries on crates.io has come up before. But as other people have mentioned, the monomorphism + cross-crate inlining + unstable ABI issues make it hard to do in general.
For procedural macros, which have long compile times but are otherwise self-contained, there's a proof of concept which bundles the macro as WebAssembly: watt.
Cargo also supports pipelining, where we spit out the "header" part first, so that higher up packages can start to build while the rest of the original crate is still compiling. (The linked post says it's experimental, but it's stable and enabled by default now 
)