Let’s discuss how to move fast and make things!
I think “let’s write Code Analyzer for Rust from scratch” approach can allow to both speed up development long-term, and to improve IDE support for Rust mid-short term.
The key ingredients are
- Avoid binary dependencies, if necessary, communicate with other tools via IPC.
- Capitalize on and improve upon existing Racer, by incrementally substituting its heuristics with precise algorithms from Code Analyzer.
- Capitalize on existing save-analysis infrastructure by using it for dependencies.
Specifically, my plan is to expand libsyntax2 to a full Rust Code Analyzer, by adding macro expansion, name resolution and type inference. Implementing full type-inference is hard, but implementing something that works better than nothing for code completion is very surmountable.
A neat trick we can pull along the way is to use save-analysis data for dependencies by invoking cargo check as an external process. That way, we’ll get precise analysis for deps, as well as a bit of performance breathing room. We can focus on “incremental update of the current code” and punt on “batch processing”. Long term, Code Analyzer should be able to produce save-analysis data itself.
I won’t go into drawing boxes and arrows with components, but I already have most of the major moving parts implemented in libsyntax2 (as a reminder, libsyntax2 already includes a language server implementation, and all state management machinery).
For the roll-out plan, a nice first-step would be to switch Racer from heuristics to real parser, type inference and such. This also directly benefits existing RLS which uses Racer.
The language server impl could also be immediately useful as a less feature-complete, but more robust RLS alternative.
I think it should be possible to add Code Analyzer features directly to RLS as well, but I would prefer to avoid focusing on that too much, because adding code to RLS is much more laborious. Instead, I’d rather spike Code Analyzer’s own language server to feature parity with RLS.
Of course, the interesting question here is how to merge this work back with rustc 
? I don’t have a ready answer here, besides “let’s do the work first, and see where we end up”
Broadly, I see three approaches:
The first one is to replace parts of rustc bottom-up, starting with libsyntax -> libsyntax2. This is the approach I originally suggested in libsyntax2 RFC, but now I don’t really like it. The first problem here is that to replace libsyntax with libsyntax2, both libraries should have clean interfaces. While it is hopefully is true for 2, the work to disentangle 1 from the rest of the compiler should be huge. The second problem is that such replacement requires libsyntax2 to be 100% correct, which actually runs contrary to the goal of providing better IDE support now. Currently, libsynta2 is very much usable inside the Code Analyzer, which is developed in itself. Occasionally, I have to fix spurious syntax errors here and there, but that’s not a big deal. In contrast, achieving 100% feature/bug parity with libsyntax1 will require huge effort.
The second approach is to extract reusable libraries from the middle of the stack. For example, we can imagine a type-inference library whose input type is generic, and which is usable with both libsyntax2 and libsyntax1 ASTs. I would very much like this to work out in practice Trait resolution and type inference are complicated, but mostly boring from a Code Analyzer point of view, so I would prefer to remain ignorant about obligation forest and other shenanigans
The third approach is to make a cut upper in the stack. Roughly, make the Code Analyzer capable of producing a checked MIR, and jettison old front-end of rustc completely. In other words, instead of bringing bits of Code Analyzer to rustc, move the backend from rustc to Code Analyzer. I think this is roughly the approach Dotty and Scalac are following.