Moving all built-in macros to plugins

internals
1

I’ve been meaning to write this up for a while now, ever since I had this proof-of-concept.

The Problem: Rust’s macro/syntax extension system has issues preventing their widespread use in 1.0. I believe this to have potentially dire consequences on Rust’s adoption as a stable language, I would rather keep using nightlies than touch manual code generation.

What are some of these issues?

  • many macros are provided by the compiler
  • those macros have no clear place to attach stability markers or documentation to
  • bootstrapping is hard because built-in macros in the compiler cannot be changed after the fact, to adjust for later changes to libsyntax
  • syntax extensions depend on the internals of the compiler (mostly libsyntax)
  • there is a way to write cleaner and more implementation-detail-agnostic syntax extensions, the quasi-quoting macros, but they’re unusable within the compiler (bootstrapping) and they’re less efficient/flexible than they could/need to be (which can be fixed, but, bootstrapping makes incremental progress very difficult)

The root of all evil: so far, bootstrapping seems to be the main factor that causes or worsens these issues.

  • if we can change the generated code patterns of the snapshot compiler, at the same time with the libs it compiles, that would male changes to built-in macros much less painful.
  • if we can improve quasi-quoting, it can be used almost everywhere, instead of touching compiler internals directly.
  • if quasi-quoting is in a library and if writing syntax extensions doesn’t require direct access to librustc or libsyntax, we could stabilize that (plugin) library prior to 1.0 without preventing changes to the compiler internals or the creation of an improved macro system.

The solution: all of that can be achieved by moving built-in macros to plugins, which are to be compiled by the (stage N-1) compiler that compiled the (stage N) compiler (which will load them in building stage N libs). Thanks to snapshot quasi-stability, stage 0 (snapshot) is usable as “stage -1”, as long as it includes the libs it was linked against.

The branch linked at the top of this post has #[deriving] moved out to a plugin crate, as a proof of concept. I am.unsure about feature gates turning built-in macros on. I presume rustc_diagnostic_macros and quote_macros can be removed. Should macro_rules stay and inject #[phase(plugin)] extern crate macro_rules;?

4 Likes
2

So, I would like to (eventually) move the built-in macros out of the compiler too, but this certainly doesn’t help the stability situation – plugins and syntax extensions are the #1 thing we cannot stabilize yet, due to the lack of any sort of structured API to the compiler.

(It is also true that moving things out of the compiler (almost) always makes it harder to make changes to APIs in the future, since we must accommodate the stage0 phase, so this may be a touch premature. But that seems like a secondary concern.)

3

Accommodating the stage0 phase is the only non-trivial thing I did (also, it was the main issue blocking all of this), and the resulting system handles bootstrapping much better: worst case scenario, the plugins’ code is duplicated, but we don’t need to preserve old functionality in libs.

As for the stabilization, I was under the impression that only the exppsed API would be stable, but not the implementation details.

To be honest, I’m only certain about creating ASTs, with improved quasi-quoting. Allowing AST introspection without exposing implementation details is trickier.

1 Like
4

I think this is a great idea - libsyntax is way too monolithic at the moment. Compiler-provided macros are one area where I feel Rust’s story is pretty messy and could benefit from a tidy up like this.

I don’t think there is time to stabilise macros in any form before 1.0 though, especially not procedural ones. Long term I would like libsyntax to have its own AST which is presented as an API for procedural macros, but that will take a fair bit of compiler hacking…

1 Like
5

I basically agree about this being the right direction as well as not having time to stabilize. I am unsure about exposing the AST for procedural macros or token trees, but regardless we are nowhere near deciding on what’s the right interface (and if we do choose to export an AST, I would certainly not export the current one)

6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.