Not really, and it's a major pain. You are not the first person who has wanted to do this and found it surprisingly difficult.
Currently, it can not. We may or may not be able to make it so in the future. Maybe. But also maybe not; using the compiler as a library is arguably the right thing for rustdoc. That said...
Right now no, but I was chatting with @eddyb about this earlier, and he mentioned that there was some bugs that might be causing unneeded building, or at least, that's what I thought he meant 
. This means you'd need to build the compiler, but the first time only.