About compiler plugins

[sivadeilra]

I wrote RACC, which is a port of Berkeley YACC to Rust. RACC works as a compiler extension. It consumes tokens and produces an AST, directly into the target program’s existing AST. I really, really liked the experience of writing RACC, and I think compiler plugins could be a very powerful feature for Rust.

I view compiler extensions as tightly-coupled extensions of the compiler, which is an implementation of a language, but is separate from the language. I am 100% perfectly fine with the compiler implementation changing rapidly, being unstable, etc. I expect / hope / want that the compiler will evolve at a much faster pace than the language. And that means that I expect, as the maintainer of a compiler extension, that my extension will be broken many times during the early life of Rust, and I expect to rewrite portions of RACC many times, to adapt to the changes to rustc.

I don’t want to lock the compiler into a rigid API, especially not this early in its life.

[Cyberax]

And I’m working on providing Rust support as an IntelliJ IDEA plugin. This requires me to re-write the language model in Java.

Please, don’t write any compiler extensions because it makes my project impossible.

[cmr]

Compiler plugins are already a thing that are widely used, from macro_rules to derive to custom plugins like compile_msg and other things like glium, gfx-rs, etc. It’s a fight long gone, really. Fortunately, syntax extensions have a well-defined syntactic boundary, and tools like syntex will hopefully allow for pre-processing code to expand syntax extensions in cargo build scripts.

I still think IDEs can be useful in the face of syntax extensions – most code isn’t generated by syntax extensions, and dependencies can be expanded and their information stored…

[Cyberax]

IntelliJ family of IDEs is kinda special - they can provide perfect refactoring, completion and code navigation if language model is complete. And it also has really awesome error recovery support.

Macroses make it impossible to build complete model. Some simple macroses like vec! I can provide as intrinsincs. I also can add support for quasi-quoting, even with refactorings. But compiler plugins are simply impossible. And I think they are entirely a wrong tool for something like parser generators.

I realize that removing macros support from Rust is impossible by now, but it’d be nice if there was some kind of pushback to discourage their use. Unstable API might be a good way to do it.

[sfackler]

It does not seem reasonable to discourage the use of an incredibly powerful part of the language because it will make it more difficult to integrate with one IDE.

[Cyberax]

It’s powerful in the same sense as pointer arithmetic in C is powerful. It can be cool but it’s also extremely dangerous.

And I personally would prefer a language with perfect IDE support to a language with a rich macro system. But then, I spent many days debugging Scala code littered with implicits and macroses.

[sivadeilra]

You should be hosting Rust in IntelliJ, not essentially porting it to Java. If you aren’t hosting the actual Rust compiler (and thus the compiler extensions), then you’re going to be duplicating a huge amount of the language, and your IDE will always be out of date with respect to the “real” Rust.

No, I’m not going to avoid using one of the most powerful parts of the language because one IDE chose to duplicate the language.

[Cyberax]

You can’t realistically host Rust in IDEA. There are multiple reasons:

1. IDEA needs to be able to access the AST of the parsed text. It’s required for features like semantic diffs for version control or structural search&replace.
2. Compiler must have good error recovery support. Rust currently does not fare that well.
3. IDEA keeps a lot of indexes and updates them automatically when dependencies change - it’s something akin to perfect incremental compilation (except that IDEA doesn’t do codegen). Rust has to parse everything every time.
4. Compiler plugins must be FAST. IDEA parses code immediately as you type it, so anything slower than 100-200ms is unacceptable.
5. It’s easy to write inspections in IDEA, with automatic corrective actions.

Yes, writing a language model means reimplementing the language parser and name resolution (but no codegen). However, I consider this valuable in itself - a second implementation helps to identity under-specified spots in any language.

And I feel that if a language needs macroses then it’s actually a pretty weak language. They are a simple, neat and wrong solution.

[cmr]

And I feel that if a language needs macroses then it’s actually a pretty weak language. They are a simple, neat and wrong solution.

I think you underestimate the power of a proper macro system. Type and procedural abstraction aren’t always sufficient, syntactic abstraction can help a lot, and can provide extremely powerful codegen opportunities that would otherwise be infeasible to do statically, require tedius and error-prone manual data entry, or are just plain ugly and verbose. See: html5ever, gfx-rs, derive and serialize/serde, etc.

(Also note: “macro” is singular, “macros” is plural. “macroses” is not a word.)

[mzabaluev]

There are redeeming points for the macro system in Rust:

• Macros are designed to be clearly identifiable syntactically, so that any parsing tool can choose to treat them as opaque. This might be the only option for parser extensions provided by the crate under development, unless the tool implements some kind of instant build-and-use support.
• There has been effort made (and it’s still ongoing) to try to make macro content syntax and expansion predictable and future-proof.

[sivadeilra]

It’s very strange that you say that you want to “support” Rust in some IDE, and yet you want to tell the Rust community what parts of the language are good (worth supporting) and bad (not worth supporting).

If your IDE does not support Rust macros, then I’m not even going to consider using it. Macros like debug!, warn!, println! are vital to all work that I have done in Rust. And in most of the work I’ve done in Rust, I’ve used a handful of carefully-written macros to accomplish reasonable goals.

And being able to write syntax extensions is a profoundly powerful capability. If you want to exclude this by design from your IDE, then I really can’t understand why you’re even considering doing this work.

[Cyberax]

I worked quite a lot with Scheme and its macro system and recently with Scala. It just happened that my job means that I mostly work with other people’s code.

And I do think that in many, many cases code-generating macros can be avoided. An example from gfx-rs:

let var_color = program.find_parameter("color");
program.set_param_vec4(var_color, [0.0, 0.0, 0.0, 1.0]);

This can be solved in some cases by using nameof-like constructs. It’s not applicable to gfx-rs, but macros usage in gfx-rs is also spot-on and very non-evil. They don’t magic-in user accessible fields and are nicely self-contained.

I’m not so sure about html5ever. Imagine for a second that you’d have to deal with match_token.rs but without any comments or documentation.

I understand that many people like macros and I most definitely do not want to tell anybody to stop using them. Just consider, please, that macros can make it very complicated to understand and support your code.

[Cyberax]

Macros based on quasi-quoting are fine. So there’s no problem at all with panic! or println!.

It’s the tree-generating macros can be very problematic. They can do literally everything, up to and including formatting developer’s hard drive and emailing their private photos to everyone in their contact list.

[sivadeilra]

Your “problematic” is my “awesome”.

They can do literally everything, up to and including formatting developer’s hard drive and emailing their private photos to everyone in their contact list.

The same is true of the compiler itself. It’s just a library of code. This is a really misleading thing to say. You’re trying to find reasons to exclude a powerful language feature because it makes your project more difficult. You chose this difficulty.

[tbu]

The same is true of the compiler itself. It’s just a library of code. This is a really misleading thing to say. You’re trying to find reasons to exclude a powerful language feature because it makes your project more difficult.

It’s not misleading. See e.g. there’s a reason people invented regular expressions – it’s their simplicity in matching. Similarly it’s good if tools can only do things that they’re supposed to do.

With the plugin system we have in place we have an unsandboxed and thus probably very hard to sandbox code that needs to run for compilation. Alternative implementations are will have it hard to replicate this system securely.

[huon]

The same is true of the compiler itself. It’s just a library of code. This is a really misleading thing to say. You’re trying to find reasons to exclude a powerful language feature because it makes your project more difficult. You chose this difficulty.

It’s 100% true that macros and especially syntax extensions make IDE integration harder, @Cyberax is not being lazy or trying to making things hard for themselves. That is to say, yes, it is an awesome, powerful language feature, but also, yes, it makes tooling hard.

(Please review the code of conduct, particularly point 3: there’s absolutely no need to be so aggressive to someone who is trying to do something awesome for Rust. )

(That said, I don’t think we should remove the feature, but hopefully users of it can write their syntax extensions so they doesn’t do completely crazy things, to assist with sane IDE integration. IDEs will just have to handle them in a best-effort way. It seems like it would be equivalent to the halting problem to handle them “perfectly”.)

[comex]

Aanyway, amidst all the arguing about whether macros are good, can I bring up a point?

Compiler must have good error recovery support. Rust currently does not fare that well. IDEA keeps a lot of indexes and updates them automatically when dependencies change - it’s something akin to perfect incremental compilation (except that IDEA doesn’t do codegen). Rust has to parse everything every time. Compiler plugins must be FAST. IDEA parses code immediately as you type it, so anything slower than 100-200ms is unacceptable.

All of these would obviously be extremely useful features for rustc. If you’re implementing a fully incremental Rust frontend in Java, that’s great work, but why not write it in Rust instead, with the intent of eventually integrating it into rustc? Then IDEA could eventually simply use APIs provided by librustc, without having to reimplement complex language features like macros, while preserving the performance that is expected of it. I’m sure this is easier said than done, but it would sure be nice to be able to take advantage of the same performance and ergonomics both when compiling on the command line and inside more loosely coupled editors, like vim.

edit: I see that IDEA has some scaffolding for this sort of task, but I don’t think it would be that hard to implement an API strongly inspired by it in Rust ;p

[Cyberax]

Incremental compilation IDEA-style requires a lot of indexes that are updated in-step with the code. It becomes tricky to do it if you don’t have some kind of persistent daemon running all the time.

I’ve explored integration of rustc frontend into IDEA, but it’s probably not possible. Java Native Interface is just too ill-suited for that kind of thing.

Reimplementing the IDEA-inspired API in Rust is an interesting task and it will benefit other IDEs and various integrations with editors (that’s what Racer does really well already). But it’s another project entirely and has its own benefits and drawbacks.

[sivadeilra]

My sincere apologies. I meant to offer cautionary guidance about a direction that I thought would be unproductive.

I do believe that Rust and people who use it would be better served by improving the performance of the rustc front-end, and having all IDEs use that front-end for their language services, instead of having every IDE re-implement the front-end of the language.

In Microsoft’s Visual Studio, the language services (“Intellisense”) used to be completely separate re-implementations of the language. This caused a great deal of inefficiency and inconsistency. They are finally fixing that (for C#) by converging on a single language front-end, called “Roslyn”, which both the compiler and the IDE use.

I would prefer to see Rust / IntelliJ avoid this same mistake. If the IDE is built on the same language front-end as rustc itself, then most changes to the language will be immediately accessible to IntelliJ (and any other IDE built in the same way). I think this is preferable to re-implementing the same thing, many times over.

[nikomatsakis]

On Tue, Jan 13, 2015 at 11:27:53PM +0000, sivadeilra wrote: