Keep in mind that it is expected that +/- 50% of compile time is spent in LLVM code generation, because that is a very slow process, not to mention single-threaded (this is a pretty big waste of potential resources anno 2018). As the rest of rustc gets faster, the expectation should be that that percentage is going to increase.
Personally I’d love to see LLVM tossed in favor of a more principled approach (buggy C++ code that unexpectedly breaks stable rustc is not my idea of principled, and just wastes goodwill among users). That principled approach would then enable things like multi-threaded compiler passes where conceptually feasible.
With LLVM, I don’t see this happening for the next 5+ years, because of the architecture alone.
The problem with “swapping out LLVM” is that a lot of time is spent on it, in the order ot 10s of man-years by now. Thus it is not so easily replicated. That said, I think it is still worthwhile as I cannot fathom a world that is still hampered by outdated single-threaded programming models* in about 10 years, and at very fundamental levels to boot (optimization, code generation, linking).
*Contrast this with a problem that is inherently serial, in which case using multi-threadedness can only hurt performance.