2019 Strategy for Rustc and the RLS

compiler
1

So lately I’ve been having separate conversations that all seem to be swirling about a central theme, and I wanted to try and bring those different threads together onto one thread.

Goals

It’s hard for me to clarify what this thread is about. I’m going to use the sort of generic title of a “Strategy for rustc and the RLS” for the time being. Probably a good place to start is to try and enumerate our goals. I have many in mind.

First and foremost, how can rustc help enable a truly excellent IDE experience?

But also a number of other things, that I think are interrelated:

  • How can we make rustc more maintainable and accessible?
    • Related: How can we grow the base of rustc contributors?
    • Related: How can we improve the pace of rustc innovation?
  • How can we support procedural macros that parse and manipulate Rust code?
  • How can we expose parts of rustc as libraries?

Ingredients

We have a lot of pieces floating around. I’ll just list out a few here, but I’m going to forget things.

  • @matklad’s work on libsyntax2:
    • @matklad has been building up a replacement for libsyntax from scratch. The parser is designed from the first to be incremental, fully whitespace preserving, and to do error recovery. As a result, it is well-suited to IDEs, but also to things like procedural macros.
  • rustc’s incremental support and the transition to a query system:
    • rustc now has a pretty powerful incremental system, but it’s not yet usable “end to end” – that is, we always parse, expand macros, run name resolution, and lower to HIR – even if the input files haven’t changed. This needs to change if we want to have the super fast response times that an IDE needs.
  • extracting libraries like Chalk and Polonius
    • One of my interests of late has been extracting out complex problems from rustc into libraries that can be independently supported. These libraries tend to follow a pattern of having a core chunk of shared code along with a wrapper that does unit testing. Chalk (trait system) and Polonius (borrow checker) are two examples. This approach feels very appealing to me as a way to turn rustc into more of a component architecture, but it’s not yet proven (neither Chalk nor Polonius are in use). How far can we take it? Can we make MIR into a library? What about our representation of types? for example)
  • RLS today and tomorrow
    • Right now the RLS communicates via save-analysis. We want it to use the query system (probably?). We also want faster response times and perfect results etc. We have to think about also how to ensure that the RLS is well-maintained – particularly as its usage grows. How can we get from here to there?

Possible directions

Actually, I’m going to stop this post here. I’ll try to add comments with various ideas for possible directions. Or maybe others will do so.

58 Likes
2

One thing that @matklad and I discussed was the idea of starting a fresh, IDE-focused compiler for Rust that builds on their work on “libsyntax2”. In my ideal world, we would manage to pull out and isolate code from rustc — for example, for macro expansion, hygiene, and name resolution — and share it between the two compilers. That may be a total fantasy, since that code today is quite intimately tied with the specifics of how Rust works, and it will be a real effort to separate it. But I think that the end-result would be very useful for other reasons (tying into this strand about Chalk).

On the other hand, when I floated with idea with @nrc, they (righly) raised some real concerns about maintenance burden and so forth. In particular, @nrc emphasized (again, rightly) the view that if the RLS winds up – long term – being a “second compiler”, we’re going to have a huge maintenance head-ache.

I totally agree with that. I always imagined that the idea would be to basically merge this “libsyntax2-based compiler” with rustc at some point so that there is only one. And hopefully that point would be not be that far off in the future. Which raises the question of whether we should be integrating libsyntax2 into rustc somehow from the start.

(All of this assumes, of course, we want to work from libsyntax2. I was pretty impressed with the work that @matklad has done, though.)

13 Likes
3

To elaborate my thoughts here:

Breaking rustc into independent components, like chalk, seems to offer a lot of benefits to me.

From the point of view of contributors, it means that new contributors can come into one of those projects and work on them independently. They can hack without having to go through a bootstrap cycle. We can write unit tests and isolate problems more effectively. And, I think, it will permit us to innovate more quickly, because being forced to come up with an "external interface" to your component often means you can make massive improvements without the rest of the the world having to care.

Independent libraries should also mean unexpected applications. For example, if/when we are able to transition to chalk -- and particularly if we can find ways to move more of chalk's logic into the shared chalk crates -- this implies that Rust tooling may be able to do trait queries more easily.

But this sort of thing is not free. It's a lot of work to find the right interfaces -- and they are may not always exist. It probably means a "less unified" rustc, making it harder to learn everything. It is harder to coordinate "cross-cutting refactorings", since you have to modify a number of components at once. (And if tooling is dependent on you, we may be reluctant to make breaking changes.)

14 Likes
4

About separate versioning, my feeling is that:

  1. The (both tooling and “process”) barrier for working with separately-versioned code is pretty high, and makes anything resembling coordinated changes a huge PITA. Making even small, not-particularly-coordinated changes in LLVM and getting them in is already pretty annoying.
  2. I feel (but I have no proof) that with sufficiently smart use of incremental and the build-system, modifying an end-module should be basically free (--keep-stage 0 almost gets us there, but more love is needed). For example, as of today, working on rustc_codegen_llvm can be done with no unneeded recompilations.
  3. Last time I checked, bors sucked quite a bit, and it feels that it still sucks somewhat. Therefore, getting developers off rustc gets them off bors. Unless we can get a lot of developers off rustc, making bors suck less might be a better option.
  4. I never faced the “rustc initial build” barrier, because I always have a fully-ready rustc somewhere on my PC, but everywhere I was involved, “separate versioning” (i.e., not having a clear single git repo to work with) had deterred me far more than anything involving build time (that’s it, as long as the build system runs on my PC rather than quitting with some obscure error, but I’ve never heard about that being a problem with Rust).
  5. If we can really get true independent usage for a component and have a stable enough API we don’t need any rustc-specific improvements, then getting it into its own library with its own maintainer works well. Similarly with projects that are pure reverse dependencies and want to keep up with rustc asynchronously.
3 Likes
5

On the other hand, I think that it is fairly useful, both for IDEs and for rustc development, to open a compiler session in some “crate context” and ask it some sort of “queries”.

Getting into a state where you can ask useful queries requires at least fully finding and parsing metadata, and you generally also want to work with types, which together are a major part of a Rust compiler. Being able to write-edit-compile-run MIR, or “direct” trait queries, would be pretty sweet.

For example, imagine a world where:

  1. You could create a rustc_polonius crate, and after changing it you can get tests running in only the time it takes to rebuild it.
  2. Something is done to make the bors wait less painful. For example, bors might “join you up” with an existing commit, and only run the tests that belong to rustc_polonius twice.
  3. You can run rustc-query and feed it facts to polonius, and there is a nice interface for handling the parsing, tests, and any integration you might want with the type-system.

Because the “facts part” of Polonius is not that dependent on rustc, there’s not much motivation for keeping it in (except for tooling), but with these changes it feels pretty neutral to have it in/out of rustc, even positive because we avoid double versioning.

3 Likes
6

Can you expand on this one a little bit better? What are the pain points besides waiting several hours for a PR to be tested?

2 Likes
7

That (having to do a compiler run even for a small "documentation change") is a strong pain on "reverse-dependencies" of rustc.

Also, there are still way too many spurious failures.

1 Like
8

I am going to write at least N posts in this thread, on slightly different topics. Here’s the first one!

Let’s discuss maintenance costs!

The perfect end-state I envision is to have a single rust-analyzer library, which is used by both command-line compiler as well as the language server. The library deals strictly with front-end bits of Rust (it takes source code as an input and produces checked MIR as a result), and, as a consequence, does not depend on LLVM and builds on stable Rust. This state I think would be much more productive than the current setup: if you want to hack on Rust frontend, you can just cargo test without building any C code, without bootstraping, and without fearing that the tests will fail on some obscure architecture. The tests themselves could be pretty fast as well: proper factoring of rust-analyzer will require a careful isolation of all IO bits, which would mean that tests won’t be doing any IO, which should make it harder to write slow tests.

The problem with this pretty picture, of course, is that we need somehow to get there first, and there’s certain fear that building a second frontend would be too much of cost.

I’d like to argue though that RLS today already is pretty costly in terms of maintenance.

The first reason for this is that the significant chunk of features that RLS provides is actually handled by Racer, and Racer itself is an alternative rust frontend of sorts. The work to maintain racer is large (it has more commits and contributors then RLS EDIT: this is apples to oranges, kudos to @kngwyu for pointing out) and, when we switch to compiler driven completion, all that work will become useless.

The second reason for this is that working on RLS itself is super hard. RLS links in Cargo, Clippy, rustfmt and uses private rustc APIs, so it is next to impossible to build. Like, the versions of all those things need to align in just the right way, and you also need to have a proper Rust compiler (which might not be even released as nightly yet). Half of the commits to RLS is “update X”, and there’s also a busy-work process of updating RLS in rust-lang/rust.

As an example, I’ve implemented “run the test under cursor” feature both in RLS and in libsyntax2 based language server. The implementation in libsyntax2 is more powerful, was easier to write, and is much better covered by tests.

In RLS, I had to use regular expressions to detect test functions, because there’s no syntax trees in RLS. Then I had to wrestle with a finky test. The test was later commented out, so had to wrestle more and more to actually make it pass. This took about a month IIRC?

In libsyntax2, I’ve wrote a small traversal of AST, (which also handles fn main), a unit test for it, a bit of plumbing for LSP and adding --package arg to command line (which RLS does not do), and an integration test, which checks that this all works together and calls cargo metadata fine.

The fact that libsyntax2 and RLS repositories have approximately the same number of Rust source code lines, if you exclude tests and generated code, also hints that adding a line of code to RLS is very expensive.

37 Likes
9

There is definitely overhead, though I wonder if it is partly that we are not entirely used to it and don't have good "flows' setup. Servo for example has been splitting up things into crates for a long time and I think they've had a lot of success with this approach.

But it's definitely a good idea to drill into the process we would use if big chunks of the compiler lived outside the main tree. I don't think we've really got it down yet. =)

Some of the problems I've encountered:

  • We don't have a very consistent strategy for assigning owners in crates.io. We should always ensure that all crates have the relevant teams as crates.io owners so that it is easy to publish a new version.
  • Similarly we need to ensure that all of our bots are running on all these crates, and that we have a kind of consistent reviewing strategy
    • right now we are very focused on rust-lang/rust
  • The rust-lang-nursery github org is annoying, mostly because we have to keep the teams in sync. Maybe some bots would be helpful here?
  • cargo has good support for things like building a local branch of some package and then configuring rustc to use that instead, but it's not always obvious just how to use it with x.py (we need docs)
2 Likes
10

These problems sound like the sorts of problems any large organization (commercial or otherwise) could run into using Rust. Documenting these issues and their solutions or releasing tools to address these issues might also be useful to the broader Rust community.

14 Likes
11

That's actually mostly a test problem (rather than a rustc refactoring) today:

  1. rustc is actually pretty well separated from LLVM. MetadataOnlyCodegenBackend exists and works modulo bugs.
  2. The "slow" bit in the current Rust tests is that they are built as end-to-end tests, which require invoking LLVM. that could be fixed by rewriting them, but I think that would be a fairly large amount of work.
  3. The reason rustc tests need bootstrapping are either proc macro tests (which aren't all tests, and is a technical limitation) or that tests need the metadata for libstd in order to run (which is because they are end-to-end).

I thought that RLS was fairly close to being a "server API" around Racer, hence its functionality is orthogonal to libsyntax2, which is a Rust frontend.

12

That also sounds like the type of work that could be pretty straightforward as a way of getting new contributors involved? Once there are some tests converted as examples. I could be totally wrong about how the process would go, of course, but "do a bunch of the same kind of thing to a lot of different code" is usually a pretty good "sic an army of eager volunteers on it" kind of a problem.

5 Likes
13

Sure, we'll just have to first get an infrastructure for comparing MIR output, if we want that. Maybe running the rpass tests in a full MIRI will be a better idea.

2 Likes
14

Let’s discuss how to move fast and make things!

I think “let’s write Code Analyzer for Rust from scratch” approach can allow to both speed up development long-term, and to improve IDE support for Rust mid-short term.

The key ingredients are

  • Avoid binary dependencies, if necessary, communicate with other tools via IPC.
  • Capitalize on and improve upon existing Racer, by incrementally substituting its heuristics with precise algorithms from Code Analyzer.
  • Capitalize on existing save-analysis infrastructure by using it for dependencies.

Specifically, my plan is to expand libsyntax2 to a full Rust Code Analyzer, by adding macro expansion, name resolution and type inference. Implementing full type-inference is hard, but implementing something that works better than nothing for code completion is very surmountable.

A neat trick we can pull along the way is to use save-analysis data for dependencies by invoking cargo check as an external process. That way, we’ll get precise analysis for deps, as well as a bit of performance breathing room. We can focus on “incremental update of the current code” and punt on “batch processing”. Long term, Code Analyzer should be able to produce save-analysis data itself.

I won’t go into drawing boxes and arrows with components, but I already have most of the major moving parts implemented in libsyntax2 (as a reminder, libsyntax2 already includes a language server implementation, and all state management machinery).

For the roll-out plan, a nice first-step would be to switch Racer from heuristics to real parser, type inference and such. This also directly benefits existing RLS which uses Racer.

The language server impl could also be immediately useful as a less feature-complete, but more robust RLS alternative.

I think it should be possible to add Code Analyzer features directly to RLS as well, but I would prefer to avoid focusing on that too much, because adding code to RLS is much more laborious. Instead, I’d rather spike Code Analyzer’s own language server to feature parity with RLS.

Of course, the interesting question here is how to merge this work back with rustc :slight_smile: ? I don’t have a ready answer here, besides “let’s do the work first, and see where we end up” :slight_smile:

Broadly, I see three approaches:

The first one is to replace parts of rustc bottom-up, starting with libsyntax -> libsyntax2. This is the approach I originally suggested in libsyntax2 RFC, but now I don’t really like it. The first problem here is that to replace libsyntax with libsyntax2, both libraries should have clean interfaces. While it is hopefully is true for 2, the work to disentangle 1 from the rest of the compiler should be huge. The second problem is that such replacement requires libsyntax2 to be 100% correct, which actually runs contrary to the goal of providing better IDE support now. Currently, libsynta2 is very much usable inside the Code Analyzer, which is developed in itself. Occasionally, I have to fix spurious syntax errors here and there, but that’s not a big deal. In contrast, achieving 100% feature/bug parity with libsyntax1 will require huge effort.

The second approach is to extract reusable libraries from the middle of the stack. For example, we can imagine a type-inference library whose input type is generic, and which is usable with both libsyntax2 and libsyntax1 ASTs. I would very much like this to work out in practice :slight_smile: Trait resolution and type inference are complicated, but mostly boring from a Code Analyzer point of view, so I would prefer to remain ignorant about obligation forest and other shenanigans :slight_smile:

The third approach is to make a cut upper in the stack. Roughly, make the Code Analyzer capable of producing a checked MIR, and jettison old front-end of rustc completely. In other words, instead of bringing bits of Code Analyzer to rustc, move the backend from rustc to Code Analyzer. I think this is roughly the approach Dotty and Scalac are following.

23 Likes
15

Let’s discuss problems with RLS :smiley: (this is the last part I think?)

I think there’s an agreement that currently RLS is not quite the IDE experience we strive for. Some people think that this would be fixed by incrementally improving RLS. I don’t agree: steady stream of improvements is impossible if the foundation is wrong, and I think that’s the case with RLS. I’d like to list specific problems.

First and foremost, you can’t fix code that you can’t build, and its next to impossible to build RLS: cargo test works only if stars are aligned, and only the person who build RLS last knows this alignment. CI is perma-red, and applying the not rocket science rule is impossible.

Code Analyzers are interesting in a sense that they are as deep as compilers, but they are also extremely broad. There’s a huge number of relatively shallow features, and this breadth is a significant part of what constitutes a good IDE experience. To give an example, once a colleague at JetBrains had a code like

if (foo() && bar()) {

and wanted to refactor it to

if (foo()) { 
  /* stuff */ 
  if (bar()) { 
    /* more stuff */`

They thought “hm, I guess my IDE should be able to do this refactoring”, placed the cursor at if, pressed Alt+Enter and didn’t see the required intention. Then they moved cursor to && and surely they saw “Split into two if’s” light bulb. That is, ideally one does not ask “can Code Analyzer do X”, one just searches this feature. Given the community-driven nature of Rust, implementing a myriad of such small helpers shouldn’t be hard, if the contribution experience is actually good, and being able to cargo test is a huge part of such an experience.

The second big RLS problem is its “shared mutable state is fine” handling of concurrency. One of the hardest problems of Code Analyzers, which is not typically faced by traditional compilers, is state management. Code Analyzer is a long-lived process which manages state affected by various asynchronous sources: user’s actions, file system events and background analysis jobs. It’s important to provide consistent view of the state to the actual analysis queries: it’s hard to do computations if you can’t rely on repeatable read property. See https://github.com/rust-lang-nursery/rls/issues/962 for details.

Fun aside: LSP, unlike the Dart protocol, includes version numbers for files, which seems misguided? If an edit applies to file A of version X, and A has version X currently, the edit might actually be invalid if the version of other file B hash changed since edit has been calculated. Client and Server must agree on the whole state to be able to make correctness guarantees. Why LSP designers haven’t just ripped of the Dart protocol? It’s so much better for important stuff…

The third big RLS problem is that it is deeply integrated with the build system and runs rustc to get save-analysis data. My and @petrochenkov’s comments in a sibling thread illustrate why this is not a good idea. The main points are that build systems for non-toy projects are non-trivial, and can’t handle on-demand requirements of IDEs. It’s a fine optimization to use save-analysis for readonly dependencies, but using it for the current crate create a lot of complexity (a significant chunk of RLS is build orchestration, which could have been just cargo check otherwise) and poor user experience (for non-trivial projects, RLS takes significant time to answer simple queries after edits). Code Analyzers should not care about build system used, and, as usual, Dart Analyzer is a great example here. It’s build system integration consists of a single abstract class, and a couple of implementations for Bazel and pub.

17 Likes
16

That's more or less true. My point was not in comparing functionality, but in comparing how hard is it to implement features, and "number of lines of code added per man-hour" as a metric probably works better than a random number.

1 Like
17

@matklad - Thanks for these SIS (Seriously Informative S–t) posts! You’ve done an excellent job IMHO of laying out the state and possible future for RLS/IDE/Compiler related issues.

1 Like
18

I’ve seen some talks about trying to use MIR as the integration point for some of the proposed solutions, but wouldn’t that mean that we would probably need to settle on a MIR format and potentially stabilize it?

Not that I think MIR will wildly change from what we have today, but wanted to raise the issue…

2 Likes
19

Also, how might something like Cranelift/Cretonne play into all this? Could it be leveraged in the mid-term (probably not short-term) to allow Proc-Macros to execute in a WASM sand-box during Code Analysis? Would that be useful? @sunfishcode @matklad

1 Like
20

More or less, the improvement of rustc and RLS will lead us to freeze some features we currently have. As Rust 2018 is the second major version ever released, I would propose 2019 to be a “feature freeze” period and focusing on documenting/standardlising things we already have right now:

  • The Rust language specification
  • HIR and MIR specification
  • libsyntax(2) API
  • Chalk/Polonius API
  • Type query API

So by the end of next year we can expect a fully documented language.

Looking forward, I am really expecting rdylib, where a lib crate can be compiled to non-monorphized MIR, being loaded by a binary crate at runtime,

2 Likes