It should be possible, but there are many things that need to be solved first.
Cargo would need to have a deeper understanding of the relationship between the build environment and the ABI. With some cooperation from rustc, it may be done even without an officially stable ABI, under assumption that same compiler with the same ABI-relevant settings is compatible. Perhaps rustc could export machine-readable metadata about its flags, and in that info include metadata on which flags affect the ABI compatibility?
-sys crates and their downstream users would have to be excluded from pre-built binaries, since they're likely to depend on system-specific system-wide state that Rust/Cargo doesn't manage. Currently -sys crate isn't an explicit pattern, and build.rs is used for much more than sys crates, so Cargo would first need to implement some build isolation and/or declarative alternatives for common build.rs tasks.
There are quite a few different Rust/Cargo versions in use. Multiply this by OS × CPU architectures, and there will be lots of variations to build. Combined with enormous combinations of dependent crate versions, it could result in a huge number of libraries to build and/or a pretty poor cache hit ratio.
There are hundreds of releases of libc, serde, and other usual suspects found in lockfiles. Cargo + crates-io would need to learn how to negotiate a useful common set of dependencies that has pre-built binaries (perhaps the registry index would have extra info listing available builds?)
I'm also concerned about sizes of pre-built artifacts. The target dir of even a small project quickly grows into multiple gigabytes of disk space. Building is slow, but downloading gigabytes of rlibs for every kilobyte of source code could be even slower. For example, debug information of zero-cost abstractions is not zero cost. Rust's MIR and debug info still has traces of all of the calls that have been inlined, all of the arguments that were turned into local variables. Even though the debug format is very efficient, it still adds up to absurd amounts of data.
I think so far the assumption was that disk space is relatively cheap, and accuracy of data and simplicity of the implementations were a priority, but for downloadable artifacts it may be necessary to create a compact representation of rlibs.