2019 Strategy for Rustc and the RLS

[wesleywiser]

Some of the problems I’ve encountered:

• We don’t have a very consistent strategy for assigning owners in crates.io. We should always ensure that all crates have the relevant teams as crates.io owners so that it is easy to publish a new version.
• Similarly we need to ensure that all of our bots are running on all these crates, and that we have a kind of consistent reviewing strategy
  • right now we are very focused on rust-lang/rust

These problems sound like the sorts of problems any large organization (commercial or otherwise) could run into using Rust. Documenting these issues and their solutions or releasing tools to address these issues might also be useful to the broader Rust community.

[arielb1]

This state I think would be much more productive than the current setup: if you want to hack on Rust frontend, you can just cargo test without building any C code, without bootstraping, and without fearing that the tests will fail on some obscure architecture. The tests themselves could be pretty fast as well: proper factoring of rust-analyzer will require a careful isolation of all IO bits, which would mean that tests won’t be doing any IO, which should make it harder to write slow tests.

That’s actually mostly a test problem (rather than a rustc refactoring) today:

1. rustc is actually pretty well separated from LLVM. MetadataOnlyCodegenBackend exists and works modulo bugs.
2. The “slow” bit in the current Rust tests is that they are built as end-to-end tests, which require invoking LLVM. that could be fixed by rewriting them, but I think that would be a fairly large amount of work.
3. The reason rustc tests need bootstrapping are either proc macro tests (which aren’t all tests, and is a technical limitation) or that tests need the metadata for libstd in order to run (which is because they are end-to-end).

The fact that libsyntax2 and RLS repositories have approximately the same number of Rust source code lines, if you exclude tests and generated code, also hints that adding a line of code to RLS is very expensive.

I thought that RLS was fairly close to being a “server API” around Racer, hence its functionality is orthogonal to libsyntax2, which is a Rust frontend.

[zachlute]

The “slow” bit in the current Rust tests is that they are built as end-to-end tests, which require invoking LLVM. that could be fixed by rewriting them, but I think that would be a fairly large amount of work.

That also sounds like the type of work that could be pretty straightforward as a way of getting new contributors involved? Once there are some tests converted as examples. I could be totally wrong about how the process would go, of course, but “do a bunch of the same kind of thing to a lot of different code” is usually a pretty good “sic an army of eager volunteers on it” kind of a problem.

[arielb1]

That also sounds like the type of work that could be pretty straightforward as a way of getting new contributors involved? Once there are some tests converted as examples. I could be totally wrong about how the process would go, of course, but “do a bunch of the same kind of thing to a lot of different code” is usually a pretty good “sic an army of eager volunteers on it” kind of a problem.

Sure, we’ll just have to first get an infrastructure for comparing MIR output, if we want that. Maybe running the rpass tests in a full MIRI will be a better idea.

[matklad]

Let’s discuss how to move fast and make things!

I think “let’s write Code Analyzer for Rust from scratch” approach can allow to both speed up development long-term, and to improve IDE support for Rust mid-short term.

The key ingredients are

• Avoid binary dependencies, if necessary, communicate with other tools via IPC.
• Capitalize on and improve upon existing Racer, by incrementally substituting its heuristics with precise algorithms from Code Analyzer.
• Capitalize on existing save-analysis infrastructure by using it for dependencies.

Specifically, my plan is to expand libsyntax2 to a full Rust Code Analyzer, by adding macro expansion, name resolution and type inference. Implementing full type-inference is hard, but implementing something that works better than nothing for code completion is very surmountable.

A neat trick we can pull along the way is to use save-analysis data for dependencies by invoking cargo check as an external process. That way, we’ll get precise analysis for deps, as well as a bit of performance breathing room. We can focus on “incremental update of the current code” and punt on “batch processing”. Long term, Code Analyzer should be able to produce save-analysis data itself.

I won’t go into drawing boxes and arrows with components, but I already have most of the major moving parts implemented in libsyntax2 (as a reminder, libsyntax2 already includes a language server implementation, and all state management machinery).

For the roll-out plan, a nice first-step would be to switch Racer from heuristics to real parser, type inference and such. This also directly benefits existing RLS which uses Racer.

The language server impl could also be immediately useful as a less feature-complete, but more robust RLS alternative.

I think it should be possible to add Code Analyzer features directly to RLS as well, but I would prefer to avoid focusing on that too much, because adding code to RLS is much more laborious. Instead, I’d rather spike Code Analyzer’s own language server to feature parity with RLS.

Of course, the interesting question here is how to merge this work back with rustc ? I don’t have a ready answer here, besides “let’s do the work first, and see where we end up”

Broadly, I see three approaches:

The first one is to replace parts of rustc bottom-up, starting with libsyntax -> libsyntax2. This is the approach I originally suggested in libsyntax2 RFC, but now I don’t really like it. The first problem here is that to replace libsyntax with libsyntax2, both libraries should have clean interfaces. While it is hopefully is true for 2, the work to disentangle 1 from the rest of the compiler should be huge. The second problem is that such replacement requires libsyntax2 to be 100% correct, which actually runs contrary to the goal of providing better IDE support now. Currently, libsynta2 is very much usable inside the Code Analyzer, which is developed in itself. Occasionally, I have to fix spurious syntax errors here and there, but that’s not a big deal. In contrast, achieving 100% feature/bug parity with libsyntax1 will require huge effort.

The second approach is to extract reusable libraries from the middle of the stack. For example, we can imagine a type-inference library whose input type is generic, and which is usable with both libsyntax2 and libsyntax1 ASTs. I would very much like this to work out in practice Trait resolution and type inference are complicated, but mostly boring from a Code Analyzer point of view, so I would prefer to remain ignorant about obligation forest and other shenanigans

The third approach is to make a cut upper in the stack. Roughly, make the Code Analyzer capable of producing a checked MIR, and jettison old front-end of rustc completely. In other words, instead of bringing bits of Code Analyzer to rustc, move the backend from rustc to Code Analyzer. I think this is roughly the approach Dotty and Scalac are following.

[matklad]

Let’s discuss problems with RLS (this is the last part I think?)

I think there’s an agreement that currently RLS is not quite the IDE experience we strive for. Some people think that this would be fixed by incrementally improving RLS. I don’t agree: steady stream of improvements is impossible if the foundation is wrong, and I think that’s the case with RLS. I’d like to list specific problems.

First and foremost, you can’t fix code that you can’t build, and its next to impossible to build RLS: cargo test works only if stars are aligned, and only the person who build RLS last knows this alignment. CI is perma-red, and applying the not rocket science rule is impossible.

Code Analyzers are interesting in a sense that they are as deep as compilers, but they are also extremely broad. There’s a huge number of relatively shallow features, and this breadth is a significant part of what constitutes a good IDE experience. To give an example, once a colleague at JetBrains had a code like

if (foo() && bar()) {

and wanted to refactor it to

if (foo()) { 
  /* stuff */ 
  if (bar()) { 
    /* more stuff */`

They thought “hm, I guess my IDE should be able to do this refactoring”, placed the cursor at if, pressed Alt+Enter and didn’t see the required intention. Then they moved cursor to && and surely they saw “Split into two if’s” light bulb. That is, ideally one does not ask “can Code Analyzer do X”, one just searches this feature. Given the community-driven nature of Rust, implementing a myriad of such small helpers shouldn’t be hard, if the contribution experience is actually good, and being able to cargo test is a huge part of such an experience.

The second big RLS problem is its “shared mutable state is fine” handling of concurrency. One of the hardest problems of Code Analyzers, which is not typically faced by traditional compilers, is state management. Code Analyzer is a long-lived process which manages state affected by various asynchronous sources: user’s actions, file system events and background analysis jobs. It’s important to provide consistent view of the state to the actual analysis queries: it’s hard to do computations if you can’t rely on repeatable read property. See https://github.com/rust-lang-nursery/rls/issues/962 for details.

Fun aside: LSP, unlike the Dart protocol, includes version numbers for files, which seems misguided? If an edit applies to file A of version X, and A has version X currently, the edit might actually be invalid if the version of other file B hash changed since edit has been calculated. Client and Server must agree on the whole state to be able to make correctness guarantees. Why LSP designers haven’t just ripped of the Dart protocol? It’s so much better for important stuff…

The third big RLS problem is that it is deeply integrated with the build system and runs rustc to get save-analysis data. My and @petrochenkov’s comments in a sibling thread illustrate why this is not a good idea. The main points are that build systems for non-toy projects are non-trivial, and can’t handle on-demand requirements of IDEs. It’s a fine optimization to use save-analysis for readonly dependencies, but using it for the current crate create a lot of complexity (a significant chunk of RLS is build orchestration, which could have been just cargo check otherwise) and poor user experience (for non-trivial projects, RLS takes significant time to answer simple queries after edits). Code Analyzers should not care about build system used, and, as usual, Dart Analyzer is a great example here. It’s build system integration consists of a single abstract class, and a couple of implementations for Bazel and pub.

[matklad]

I thought that RLS was fairly close to being a “server API” around Racer, hence its functionality is orthogonal to libsyntax2 , which is a Rust frontend.

That’s more or less true. My point was not in comparing functionality, but in comparing how hard is it to implement features, and “number of lines of code added per man-hour” as a metric probably works better than a random number.

[gbutler]

@matklad - Thanks for these SIS (Seriously Informative S–t) posts! You’ve done an excellent job IMHO of laying out the state and possible future for RLS/IDE/Compiler related issues.

[qmx]

I’ve seen some talks about trying to use MIR as the integration point for some of the proposed solutions, but wouldn’t that mean that we would probably need to settle on a MIR format and potentially stabilize it?

Not that I think MIR will wildly change from what we have today, but wanted to raise the issue…

[gbutler]

Also, how might something like Cranelift/Cretonne play into all this? Could it be leveraged in the mid-term (probably not short-term) to allow Proc-Macros to execute in a WASM sand-box during Code Analysis? Would that be useful? @sunfishcode @matklad

[earthengine]

More or less, the improvement of rustc and RLS will lead us to freeze some features we currently have. As Rust 2018 is the second major version ever released, I would propose 2019 to be a “feature freeze” period and focusing on documenting/standardlising things we already have right now:

• The Rust language specification
• HIR and MIR specification
• libsyntax(2) API
• Chalk/Polonius API
• Type query API

So by the end of next year we can expect a fully documented language.

Looking forward, I am really expecting rdylib, where a lib crate can be compiled to non-monorphized MIR, being loaded by a binary crate at runtime,

[kngwyu]

The work to maintain racer is large (it has more commits and contributors then RLS)

Though I don’t think racer is very easy to maintain, this comparison is not fair.

Racer’s last commit is 5 years ago.

[matklad]

I’ve seen some talks about trying to use MIR as the integration point for some of the proposed solutions, but wouldn’t that mean that we would probably need to settle on a MIR format and potentially stabilize it ?

I don’t think so. The fact that we extract parts of the compiler as a libraries does not mean that we need to stabilize them. For example, Cargo is published to crates.io as a library, but it is of 0.x version and is explicitly unstable. We do have convos like “- Please provide better document docs for APIs!; - APIs are unstable, you are on your own” once in a while on issue tracker, but that’s it.

Of course, there’s a certain risk that making parts of the compiler usable will lead to folks using them in their tools a de facto stabilization. I think the risks here are low: if someone writes a tool, that’s great, but they’ll just have to keep up with upstream’s pace of development themselves.

[matklad]

Also, how might something like Cranelift/Cretonne play into all this? Could it be leveraged in the mid-term (probably not short-term) to allow Proc-Macros to execute in a WASM sand-box during Code Analysis?

Maaaybe? For Code Analyzer, I’d rather just abstract proc macro interface behind a trait: applying TokenStream -> TokenStream transformations is in scope of CA, but creating such a transformation by compiling Rust code is def out of scope.

As for the impl, I suspect mid-term the solution would be to use a separate native process for proc-macros and IPC mechanism? (loading proc macros into process is ok for batch compiler, but won’t work in CA due to reliability and perf requirements).

[matklad]

More or less, the improvement of rustc and RLS will lead us to freeze some features we currently have. As Rust 2018 is the second major version ever released,

I don’t think so? The feature freeze will be required only when merging two strands of development, but that point seems to be far-off. Optimistically, I’d say two years is a minimal time required to create an IDE foundation, it’s not something we can really reasonably spike. I don’t think IDE-work should affect language planing in a principal way until we’ve have at least the basics working.

[matklad]

HIR and MIR specification

I think stabilization and specification efforts have to be explicitly out of scope to actually ship something usable.

[nrc]

I’m pretty excited about the libsyntax2 work. I see a lot of potential to improve features in IDEs and help the compiler towards very incremental compilation. It would be great to improve Racer with it, for one thing.

However, my impression is that libsyntax2 allows us to do more, but doesn’t help that much with the type-driven actions which the RLS currently provides. Or rather it would let us do something similar and faster, but not as accurate (without implementing all of type checking there).

The perfect end-state I envision is to have a single rust-analyzer library, which is used by both command-line compiler as well as the language server

This is pretty much my ideal end-state too. And I also worry about how to get there. In particular the worry I expressed to Niko is that we use it in the RLS, but then there isn’t the resources to use it in the compiler and we have much increased maintenance costs for the RLS.

the rust-analyzer idea, is basically moving 80-90% of the compiler into a library. Which I think is a great idea, I’m a big fan of moving stuff out of the repo. However, there are issues with doing so (e.g., linking std versions to compiler versions) and realistically, from the RLS perspective, we won’t get anything new, it’s just an implementation detail.

The second reason for this is that working on RLS itself is super hard. …

This is a bit of an exaggeration, if you pull and use that day’s nightly it will work 8 times out of 10. It’s been worse recently because of the increased rate of changes for the edition. However, I totally agree if it would be better if this didn’t happen.

Half of the commits to RLS is “update X”, and there’s also a busy-work process of updating RLS in rust-lang/rust.

This is about right and it is extremely frustrating. I think though that any solution in this space will have these problems. There’s either API compat issues or you have the harder problem of keeping up with language changes without any compiler errors. I believe we can do much better by improving the tooling here - either using a mono-repo or some kind of structured integration of changes to Rust components.

As an example, I’ve implemented “run the test under cursor” feature both …

This isn’t really a fair test. It is something that libsyntax2 should be good at, but which isn’t a goal for the RLS design.

The tests in the RLS are bad, but I think rather than being a reason to throw it all out and start again, we should just write more (and better) tests.

The fact that libsyntax2 and RLS repositories have approximately the same number of Rust source code lines, if you exclude tests and generated code, also hints that adding a line of code to RLS is very expensive

I’m not sure that is a useful metric? If you really want to compare lines of code you should include half the compiler and the RLS support crates. I won’t deny that it is relatively difficult to add features to the RLS, but I expect that two years down the line when libsyntax2 is more complete it will be a lot more expensive to add code there too.

I thought that RLS was fairly close to being a “server API” around Racer, hence its functionality is orthogonal to libsyntax2 , which is a Rust frontend.

This is pretty true, and important.

Capitalize on and improve upon existing Racer, by incrementally substituting its heuristics with precise algorithms from Code Analyzer.

This indeed sounds like a good idea. We’ve also thought about the RLS giving type information to Racer so that it can complete a lot more. It might be useful to do the same for your code analyzer?

Capitalize on existing save-analysis infrastructure by using it for dependencies .

The hard part of dealing with deps is that you have to model the Cargo project. Hopefully some Cargo refactoring will make this much easier, but you still have to deal with some issues in any case (and this is where a lot of the RLS’s nasty concurrency issues come from).

In contrast, achieving 100% feature/bug parity with libsyntax1 will require huge effort.

ISTM that we have to address this eventually whatever approach you take and it will be a huge effort

Some people think that this would be fixed by incrementally improving RLS

I’m strongly in this camp. While most of the points you make are true, none of them are really about the foundation of the RLS and I think they can mostly be addressed without throwing everything away. It is a well-known true-ism in software engineering that you can have a clean prototype, but as you add features you inevitably end up with less clean code. We’ll surely have a bunch of problems with a clean start, we’ll just have wasted a year or so before we get to that point.

The second big RLS problem is its “shared mutable state is fine” handling of concurrency

We can do much better. Fundamentally the concurrency issues are difficult due to wanting to be responsive to the user and still be as accurate as possible. Using snapshots or something like that might be a reasonable improvement.

I think it very much depends on what you want to do as to whether repeatable read is important. AFAIK, nothing in the RLS currently needs it? We might deliver slightly out of date data, but that is usually not a problem and if the user is typing, then it’s the best you can do.

The third big RLS problem is that it is deeply integrated with the build system and runs rustc to get save-analysis data.

I don’t see this as a problem. Effectively (and this will require some work in Cargo) we will ask Cargo how to build a project and then have a very quick way to query "this file has changed what should be rebuilt) and then we do it. Other build systems will use Cargo as an abstraction layer. You never have to do a full build unless you want to (or say the Cargo.toml is changed).

If you can only guess at the project layout, I don’t think you can get accurate type info.

a significant chunk of RLS is build orchestration, which could have been just cargo check otherwise

This is not quite right. The complexity comes from tracking dependencies in workspaces where there might be multiple current crates and having files in memory which have been changed. My hope is that this can be factored out of the RLS and into Cargo.

if someone writes a tool, that’s great, but they’ll just have to keep up with upstream’s pace of development themselves

this has not worked in the past. If a tool appears which make the libs de factor stable, and people use that tool then the compilers are in an awful bind and basically have to ensure back compat.

Maintenance of the RLS

My hope is that by factoring things out to Cargo, and relying on save-analysis and the proposed query system for the compiler, we can ensure that the RLS is a fairly thin intermediary which doesn’t need too much maintenance and that by maintaining the compiler and Cargo, the RLS is also mostly looked after.

In the short-term making incremental improvement to the compiler, Racer, and the RLS is something that can be reasonably done with the resources we have. Rewriting the compiler front-end would seem to need lot’s more people working for a long time

[dan_t]

It is harder to coordinate “cross-cutting refactorings”, since you have to modify a number of components at once. (And if tooling is dependent on you, we may be reluctant to make breaking changes.)

They may be only internal libraries, which are only used by rustc.

So it’s easier to make refactorings over all, but you still have very clear interfaces.

[Xanewok]

I’d just like to throw in my 2 cents here. I agree with above points (including, but not limited to):

1. Settle on more principled approach to concurrency
2. use query/incremental system instead of re-running rustc --emit=metadata ...
3. facilitate tool development + repo coordination (this is frustrating, I agree)
4. …

These are real problems and would I also like to address them, however I honestly don’t like the ‘everything is broken and nothing works’ narrative because I feel like this is counter-productive here and pushes us further away from the goal, if anything.

Most of the RLS problems are growing pains. What first started as a LSP-compatible wrapper around cached rustc invocation has grown to something bigger; it now bundles other tools like Clippy or Rustfmt, reuses Cargo to run customized cargo check and later caches + orchestrates build itself spanning multiple crates and also supports more loosely related features, just like the aforementiond Run test functionality.

The long-term planned direction sounds great and it’d be awesome if we could make it a reality next (or the one after?) year. I just wanted to say that we definitely don’t need to throw out everything; we can focus on actively improving the status quo, instead. I imagine abstracting away the project model (incl. support for other build systems?), some amounts of refactoring, enhancing Racer with save-analysis data and cleaning up the RLS test suite will do wonders and it’s definitely doable now. Frankly, I think the RLS just doesn’t have the manpower to do it as quickly.

[repax]

_{Pipe-dream warning, so I’m hesitant to even ask:}

How feasible would it be, as an end goal, to combine rustc and RLS into a stateful incremental compiler, i.e. one that holds the entire code in all stages from source to MIR in memory (or mmapped)? Source changes would propagate through the structures and incrementally update them. Queries could be run on the structures on-line to provide type information etc to satisfy the needs of the RLS.