Ah, I think I understand your suggestion of 'just build the libraries' now to mean 'build std on demand'. It's tantalizingly possible, but in practice quite hard, and as I described above what we're pursuing now is for the project to make std available in binary form for as many targets as possible.
The main reason it's difficult is just that building std is (I believe) more difficult than building typical Rust programs, since it has some points of close integration with the plattform (like memory management, unwinding, backtraces). It also has a complex and custom build system, and contains unstable code that can't presently be built with the stable compiler.
Building std requires a native cross-toolchain (such as the android ndk) to do the cross-compile. This is true for building any cross-compiled Rust code though. The hardest part of making cross compilation easy on Rust will be making these bits easily available.
I think eventually we'll probably get to a place where cargo can easily rebuild std.