Lark compiler experiment

[jntrnr]

For the last few months, @nikomatsakis, @wycats, and myself have been working on an experimental compiler to see if we can build one that can handle both straight-line/traditional compilation as well as be able to be used in an interactive environment like an IDE. Still very early days, but it’s showing some promise and we wanted to chat about the project with the wider compiler team.

The project is built on Salsa, with each stage (parsing, typechecking, etc) being salsa-ified. This lets us create a lot of queries that can answer specific questions about entities in the system, like typechecking a single function or grabbing the header of a function without its body.

To prove things out, we also built Language Server Protocol (LSP) support into the compiler, so you can run the compiler in IDE mode from inside VS Code and get interactive features.

If you’re interested, there’s a link to learn more:

• https://github.com/lark-exploration/lark
• For more in-depth overview: https://github.com/lark-exploration/lark/blob/master/docs/internals.md
• Video covering some of the basics: https://www.youtube.com/watch?v=SGY3TZMkOpo&feature=youtu.be

As I mentioned, it’s still very early days, but we’re hoping to prove out enough to show that the approach could potentially work for Rust itself.

[anp]

Oooh! Have you built any sort of latency measurements for this?

[jntrnr]

Not yet, though that’s definitely in the cards as it matures. We’d like to get a sense for how much we are paying as we do more complex things, to make sure we’re not doing analysis we don’t need to do.

Right now we’re doing more of a breadth-first approach to make sure we have at least some coverage of features broadly before we begin improving the speed of things like LSP calls.

[jjpe]

Sounds like perhaps collaboration with @matklad might be a good idea, given his recent work on an IDE-service capable compiler design?

[davemilter]

@jntrnr @jjpe

Sounds like perhaps collaboration with @matklad

https://github.com/lark-exploration/lark and https://github.com/rust-analyzer/rust-analyzer looks almost the same for me:

1. Both are the experiment how to reuse compiler code in IDE
2. Both implements LSP
3. Both uses salsa

[jjpe]

@davemilter I noticed that as well

As an aside, while I’m aware LSP has won mindshare over the last 2 years, I don’t think it’s the end of the road there. But in the meantime it’s better than what came before it, that’s for sure.

[CAD97]

Of note in terms of rust-analyzer at least:

The LSP implementation is separate and on-top of the logic core of rust-analyzer. If some new protocol were to come along, the protocol layer could be implemented on top of the same core and work alongside the LSP protocol interface as well.

(The LSP interface is of course as stable as LSP, but the core logic Rust interface is very unstable.)

[matklad]

looks almost the same for me:

The crucial difference is that Lark is for the Lark language, while rust-analyzer is for Rust. Both are experiments, but you already can use rust-analyzer as a daily IDE.

Another super-important difference from IDE POV is that in Lark everything is a macro, if I understand it correctly. For example, struct is a macro. This “entities decide about their syntax themselves” seems to be a pretty hard nut to crack using traditional IDE syntax trees (like the ones in IntelliJ, Roslyn or rust-analyzer).

EDIT: to make it clear, using something like rowan would be possible, but less useful, because it would be hard to write IDE features like formatting, auto-indentation, typing assists, code assists and refactorings “from the outside”, each specific entity would have to bundle it’s own specific refactorings, etc.