Good morning Rustaceans! Hot off the heels of the Rust All Hands in Berlin last week we’ve had numerous discussions about the stabilization of Macros 2.0 and what we can have ready for the Rust 2018 release. I’d like to detail what we discussed here as well as solicit for your help in stabilizing a “Macros 1.2” in the upcoming 1.27 release possibly.
In Berlin we had a session specifically about Macros 2.0 and what, if any, we could stabilize for the edition release. While “Macros 1.1” is stable today it doesn’t include features like:
- Macros through attributes (aka
- Custom bang-style macros (aka
- Span information, everything is lost today with
- Rich diagnositics beyond
- Hygiene information to avoid name collisions programmatically instead of “hopefully” via
These features have gone through a number of RFCs to date and have seen quite extensive testing throughout the community. In our discussion, however, we specifically ruled out two large features for the Rust 2018 release:
- Declarative macros (aka
Both of these features were deemed as too high risk to stabilize at this point, but fear not as Macros 2.0 is large and has many other features internally! We came up with a design (which I like to call “Macros 1.2”) which is aimed at stabilizing a subset of Macros 2.0 functionality which is sort of the 80% of what crates use today.
In summary, we concluded that the following subset is possible to stabilize for the Rust 2018 release:
- Importing macros through the module system
- Attribute macros, only attached to items but not modules.
- Custom bang-style macros, but only invoked in module contexts, not as expressions or inside functions.
- No hygiene information through explicitly requesting “call site” hygiene, aka copy/pasting code.
- A large chunk of the
proc_macroAPI to enable preserving span information on tokens.
The subset above we concluded was imminently stabilizable with no major open questions (unlike hygiene and declarative macros). And with that, let’s dive into these pieces!
Macros and the module system
The first item on this list is the ability to import procedural and other macros through the module system (aka
use) rather than
#[macro_use]. This dovetails very nicely with the edition’s attempt to remove the need for
extern crate and is in general much nicer to use than
#[macro_use] as well!
The tracking issue for this issue has now entered FCP for stabilization and the main takeaways are:
- You’ll be able to import attributes via
useas well as procedural bang-style macros
- You’ll also be able to import Macros 1.0-style macros (aka
- Importing 1.0-style macros is unhygienic, so if the macro invokes another macro you’ll have to import that as well
- You won’t use
useto import macros within your own crate but will rather instead rely on today’s scoping rules.
While not a perfect stabilization the warts are specifically related to 1.0-style macros which are long-term going to be phased out with
macro macros. In the meantime the experience of using 1.0-style macros changes very little and we’re mostly just rationalizing
use with the backwards-compatibility of today’s macro system.
Macros and items
We discussed this at the work week and felt that the only part of invoking a macros 2.0 macro was in the item position like a function attribute or an invocation inside of a module scope. This appeared to cover most use cases of procedural macros in the majority.
Concerns related to hygiene were brought up related to expression macros (or those that could expand to expressions). I’m not personally too privvy on these details but @nrc may be able to fill in more if others have questions about this! I think the general gist though is that “copy paste hygiene” is relatively straightforward for items where there’s already not a huge amount of hygiene in Macros 1.0, but expression copy/paste hygiene is tricker.
Lack of hygiene information
We universally agreed that hygiene was a good thing we’d like to stabilize one day. We basically all agreed as well though that it’s realy hard to stabilize hygiene and the compiler is definitely not ready to do so. As a result we wanted to stabilize a forward-compatible solution with hygiene but still not actually stabilize anything hygienic today.
In rustc hygiene information today runs through
Span, where each
Span indicates what syntactical context it came from as well as the literal bytes in the original source it corresponds to. This is expressed as well in the
proc_macro API with
Span having constructors like
call_site (copy/paste hygiene) and
def_site (not accessible by expanded code hygiene).
Our conclusion here was that we would only stabilize the
Span::call_site() function and no others. This way if you ever manufactured a span you’d be explicitly opting-in to copy/paste call-site hygiene. Eventually once we stabilize hygiene it’ll either be through spans or a separate field, and this way we should have avenues for inserting hygiene information, if necessary.
This is also highly related to…
proc_macro API changes
And finally the last piece of stabilization that would be required here would be the
proc_macro API itself. @dtolnay, @nrc, and I took a long hard look at the current API with an eye towards stabilization and came up with a reorganized API which is now available in today’s nightly! The crate should provide everything it did before, only in a reorganized and more forward-compatible and ergonomic fashion. The main changes are:
TokenNodeenum has been removed and the public fields of
TokenTreehave also been removed. Instead the
TokenTreetype is now a public enum (what
TokenNodewas) and each variant is an opaque struct which internally contains
Spaninformation. This makes the various tokens a bit more consistent, require fewer wrappers, and otherwise provides good future-compatibility as opaque structs are easy to modify later on.
Literalinteger constructors have been expanded to be unambiguous as to what they’re doing and also allow for more future flexibility. Previously constructors like
Literal::integerwere used to create unsuffixed literals and the concrete methods like
Literal::i32would create a suffixed token. This wasn’t immediately clear to all users (the suffixed/unsuffixed aspect) and having one constructor for unsuffixed literals required us to pick a largest type which may not always be true. To fix these issues all constructors are now of the form
Literal::i32_suffixed(for all integral types). This should allow future compatibility as well as being immediately clear what’s suffixed and what isn’t.
Each variant of
TokenTreeinternally contains a
Spanwhich can also be configured via
set_span. For example
Termnow both internally contain a
Spanrather than having it stored in an auxiliary location.
Constructors of all tokens are called
Term::internis gone) and most do not take spans. Manufactured tokens typically don’t have a fresh span to go with them and the span is purely used for error-reporting except the span for
Term, which currently affects hygiene. The default spans for all these constructed tokens is
Termtype’s constructor explicitly requires passing in a
Spanto provide future-proofing against possible hygiene changes. It’s intended that a first pass of stabilization will likely only stabilize
Span::call_site()which is an explicit opt-in for “I would like no hygiene here please”. The intention here is to make this explicit in procedural macros to be forwards-compatible with a hygiene-specifying solution.
Final push for stabilization
My hope is that this is the final cut of the
proc_maco API before stabilization. If all goes well I’d like to FCP this subset of macros 2.0 towards the end of the current nightly cycle (1.27) and then stabilize for the 1.27 release. If discussions show that we’re not quite ready for stabilization then we will definitely switch to a different schedule.
I think that this subset is going to enable quite a few use cases seen in the wild today to work on stable Rust while also providing a solid base for us to extend and grow with hygiene/diagnostic/etc information in the future. In the meantime though we need your help in testing out the new APIs. If you’ve got a procedural macro crate or custom derive consider updating to
syn 0.13 or
proc-macro2 0.3 and give us you’re feedback! We’re particularly interested in bugs and ergonomic issues with the current API.
Here’s to hopefully stabilizing Macros 1.2 in Rust 1.27.0!