Help stabilize a subset of Macros 2.0!

[alexcrichton]

The fact that parts of the design space (e.g. hygiene) are still nebulous while we are stabilizing other parts is a bit worrying

Certainly a fair opinion! I’m led to believe, however, that “hygiene experts” (aka @nrc and the compiler team) are confident that the subset we could stabilize is forward compatible with hygiene in the future. We’re in a tough position right now in terms of we don’t know what hygiene will look like long-term but we’d still like to stabilize something ahead of time. I think we’ve done a thorough amount of work exploring the possible design space though and have reached an API which should be maximally forward-compatible to what we’d like to do with hygiene in the future.

In terms of documentation and design, a good point as well! I think we’ll certainly have a good solid round of documentation before stabilization, but I know @dtolnay has done some amazing work especially with custom derive in the syn repository for examples and such.

I suppose macros 1.2 will also do away with the restriction about only allowing a single proc_macro from a crate? (I forget what the exact restriction is, but it was always this weird unergonomic paper cut.)

I believe so yeah, you’ll be able to define multiple attributes/derives/macros in the same crate.

With the new use-based import, one is able to selectively import just one macro, which is great. However, because the macro_rules! macros don’t have hygiene with regard to other macros, all the macros the imported macro expands to also need to be in the caller’s scope.

Indeed a downside! I believe the libs team will recommend having macros that look like:

macro_rules! foo {
   ( /* public interface */ ) => (foo!(@internal /* internal interface*/));
   (@internal /* private interface */) => (/* ... */)
}

and with that sort of construction you’ll still be able to have internal details and be able to work with only one imported macro.

Also note that #[macro_use(foo)] suffers this same problem today in that it doesn’t work with hygiene,

Finally I believe that #[macro_use] also won’t be removed, it’s primarily that we’d like a way to remove the need for extern crate and the attribute, which internal macros and some pain (unfortunately) on macro authors’ part should help solve.

[petrochenkov]

Does this stabilization has strong enough motivation?
Macros 1.1 had really strong motivation - availability of top 10 crates on stable, does this stabilization provides benefits of similar scale?

I’d prefer not to stabilize until we have a maintainer with enough time and detailed understanding of expansion and hygiene, and an RFC describing what exactly is stabilized and how it works. I’ve seen @aturon’s slides from All Hands, shipping fast seems to be the top priority now, but it still feels wrong to stabilize the feature in its current state until there’s really strong motivation.

---

Also, call_site is still hygiene and not “copy-paste” like macro_rules! for items, see, for example, tests in this PR, hygiene-optout-3.rs in particular. There’s difference when you have nested macro calls.

[alexcrichton]

Nah I definitely agree! I do personally feel like there is quite strong stabilization for this iteration of procedural macros. I’ve personally authored macros like wasm-bindgen, cfg-specialize, and stdsimd assertion helpers. Crates like serde_derive will also benefit from the span information to generate more accurate error messages for "does not implement Deserialize" rather than forcing everything to point to the #[derive(Deserialize)] today.

Moreover I see this as a very important stepping stone in the next stage of stabilization of Macros 2.0. The body of Macros 2.0 is so hard it’s hindering its own stabilization I feel. Getting smaller features out the door incrementally is a great way to get more users, experience, and provide further motivation for stabilization.

I think Rocket is also a great example of something that won’t quite be stable with the subset proposed here but should shortly afterwards be able to add one the necessary features and stabilize.

As to an RFC, do you feel that the features haven’t been adequately RFC’d already? Do you feel the OP here has ambiguity about what’s being stabilized?

[mark-i-m]

@alexcrichton Thanks for the reply! It’s good to know that more extensive exploration has been done. However, I would still like some of this knowledge to be written down in the form of an RFC, as @petrochenkov suggests . Having a document detailing exactly what is going on would be really helpful. I think it would be useful for directing the community’s efforts in this area.

[alexcrichton]

Just to make sure I understand first, though, do you have areas in particular you feel are too hazy? I’d love to make sure we’re all on the same page at least before moving to an RFC!

[Ixrec]

At least for me, the biggest issue by far is that I have no idea what “stabilizing without hygiene” means. Does that mean that collisions between “inner” names and “outer” names are completely impossible, and there’s simply no escape hatch syntax/API yet? (this I have no qualms with stabilizing) Or does that mean every “inner” name inside a macro is eligible for colliding with “outer” names in sufficiently unlucky client code? (in which case I’m terrified) And what does this mean for “edition hygiene”; making sure macros from Rust 2015 crates can be used in Rust 2018 crates? Or whatever kind of hygiene it’s called when a macro contains an extern crate that pollutes the client’s namespace?

Other than that, I guess my #2 issue is that I’m not aware of any document I can read to get a reasonable understanding of the current state of macros 2.0 as a whole. I’m pretty sure the RFCs are all outdated by now, but since I’ve never tried writing a macro 2.0 of any kind myself most of the information in this thread’s OP post is gibberish to me.

[mark-i-m]

Just to make sure I understand first, though, do you have areas in particular you feel are too hazy? I’d love to make sure we’re all on the same page at least before moving to an RFC!

At a high level: what is the end goal of macros 2.0 and how does macros 1.2 work towards it? Let me try to take that apart:

• RFC 1584 (macros 2.0) mentions in passing (in the motivation section) that there are a bunch of non-backwards-compatible changes we would like to do, but it doesn’t go into any detail on them.

  • Naming/name resolution: RFC 1561 and RFC 1560 give a bit more information on macro naming and name resolution, but they are TMK the only parts of macros 2.0 that have RFC or are documented anywhere.

    • IIUC, importing macros through the module system (part of macros 1.2) is the first step towards this. What other impacts are there? Are we planning to phase out #[macro_export] and #[macro_use].
    • IIUC macros 2.0 will eventually allow pub macro, but this is not a part of macros 1.2. So how does it interact with importing macros through the module system, which is part of 1.2?
    • How does this interact with the new module system and path clarity work?
    • IMHO, all of this may be worth its own stabilization RFC.

  • Hygiene: This seems to still be not well understood except by a few experts.

    • IIUC, macros 1.2 doesn’t really attempt to make progress on this front for the edition release.
    • But long-term (i.e. for macros 2.0): What is the design space here? What are the design requirements? What are the hygiene experts interested in doing? What are the unresolved questions? IMHO, this warrants its own full RFC since hygiene is one of the things that distinguishes rust macros from other languages.
    • How do proc_macros, custom derives, attributes, and libraries like syn expose hygiene? Is there a mostly-complete vision for what this should look like (it sounds like the answer is “no”)? How do we know that stabilizing macros 1.2 won’t hinder this in the future?

  • Syntax: Is macro the only new syntax proposed for macros 2.0? What do the new declarative macros look like? How are they different from existing declarative macros from a user perspective? Similar for procedural macros and attributes?

    • IIUC, macros 1.2 doesn’t attempt to introduce any new syntax for the edition.
    • The only things I have seen are snatches of proposed syntax in the tracking issue, various discussions, and @jseyfried’s WIP RFC.

• What is the big picture? How do declarative macros 2.0 fit in with proc_macros, custom derive, syn, and the attribute system. They all seem to be secretly tied together in the macro expansion and name resolution parts of the compiler.

  • RFC 1566 lays out the vision for procedural macros, and RFC 1681 (macros 1.1) stabilizes a strategic subset of it. As I understand it macros 1.2 is intended as the next strategic subset, but as mentioned in the previous bullets, it’s not clear what this is a subset of. Specifically,
    • What is the stabilization plan? Are we going to have a macros 1.3, 1.4, etc. until we are done? What deprecations are going to happen? How do editions work into the picture? Personally, I think it would be good to have a high-level picture of how we actually plan to achieve macros 2.0.
    • What remains to be explored? What remains to be implemented? Are there any major changes need to the compiler itself? name resolution? macro expansion? interactions with other features? I think these are all best described by various RFCs and tracking issues, as mentioned in the previous bullets.

[alexcrichton]

At least for me, the biggest issue by far is that I have no idea what “stabilizing without hygiene” means. Does that mean that collisions between “inner” names and “outer” names are completely impossible, and there’s simply no escape hatch syntax/API yet?

A good question! I don’t think hygiene has been super concretely defined in terms of what it means for Rust, but what this “Macros 1.2” proposal is saying is that we’ll do exactly what macro_rules! does today. When you define an item with a macro it enters the same namespace as the surrounding scope, name collisions and all.

To be clear, this is exactly the case already today with derive(Foo) and macro_rules!. This has long been considered a bug in rustc that we’d like to fix, and that’s what the hypothetical “hygiene in macros 2.0” will solve, where you can select what scope the names are defined in.

This proposal, however, is basically saying “let’s hold off on these hypothetical and not concretely defined plans and extend the current system in a familiar way that macros operate in today”.

Dealing with editions doesn’t come into hygiene, these macros will basically allow inserting names into a module which other items can also reach, just like macro_rules! does today. Many questions can basically be answered “what does macro_rules! do today?”

IIUC, importing macros through the module system (part of macros 1.2) is the first step towards this. What other impacts are there? Are we planning to phase out #[macro_export] and #[macro_use].

Yes, do you find that RFC 1560/1561 are lacking details, though? The tracking issue also has details on how #[macro_export] is still needed but #[macro_use] will no longer be needed. There is also information about how it is a known and accepted aspect that macro_rules! macros don’t live in modules when you import them (a limitation of the stable macro_rules! system).

But long-term (i.e. for macros 2.0): What is the design space here? What are the design requirements? What are the hygiene experts interested in doing? What are the unresolved questions? IMHO, this warrants its own full RFC since hygiene is one of the things that distinguishes rust macros from other languages.

Good questions! You’re correct in that macro 1.2 does not make progress in hygiene, it is sticking with exactly what we have today with macro_rules!. In that sense the hygiene afficianados we have agree that the proposal here is adequately forward-compatible with future possible hygiene systems. (this is what we discussed quite a bit in Berlin)

IIUC, macros 1.2 doesn’t attempt to introduce any new syntax for the edition.

Correct.

What is the stabilization plan? Are we going to have a macros 1.3, 1.4, etc. until we are done? What deprecations are going to happen? How do editions work into the picture? Personally, I think it would be good to have a high-level picture of how we actually plan to achieve macros 2.0.

Is the OP unclear in this respect? From what I outlined there the pieces are:

• Importing macros through the module system
• Attribute macros, only attached to items but not modules.
• Custom bang-style macros, but only invoked in module contexts, not as expressions or inside functions.
• No hygiene information through explicitly requesting “call site” hygiene, aka copy/pasting code.
• A large chunk of the proc_macro API to enable preserving span information on tokens.

I can’t necessarily predict the future so I have no idea if 1.3 will happen, but I don’t see hwo that would impact the 1.2 issue?

The set of features unstabilized and yet-to-be-explored is quite amorphous and always changing, but is that necessarily required for staiblizing 1.2?

@mark-i-m you’ve definitely got a lot of great questions, but I feel like the meaty pieces (module system and such) are already RFC’d? The proposal here is explicitly a slice of what we have already RFC’d and implemented today, which is why I’d like to move to FCP later in this cycle. Nothing new is being introduced here other than the idea of what subset should be stabilized I believe.

[RalfJung]

Dealing with editions doesn’t come into hygiene, these macros will basically allow inserting names into a module which other items can also reach, just like macro_rules! does today. Many questions can basically be answered “what does macro_rules! do today?”

So what happens if a new edition introduces a new keyword, and hence parsing depends on the edition. Will the macro-generated code parsed with the edition of the macro, or the file where the macro is used?

[alexcrichton]

Remember though that procedural macros purely receive a list of tokens, it’s up to them to determine how to parse said tokens. In that sense new keywords don’t actually mean anything to procedural macros, they’re just yet another identifier.

[mark-i-m]

Yes, do you find that RFC 1560/1561 are lacking details, though? The tracking issue also has details on how #[macro_export] is still needed but #[macro_use] will no longer be needed. There is also information about how it is a known and accepted aspect that macro_rules! macros don’t live in modules when you import them (a limitation of the stable macro_rules! system).

Oh, sorry. I should have been more clear. What I meant is do you want to stabilize those parts now (as part of macros 1.2 for the edition)?

Also, thanks for the useful links to the tracking issue comments

Is the OP unclear in this respect? From what I outlined there the pieces are:

Sorry, again I should have been more clear. I meant long-term for macros 2.0 overall. You answered my questions, though.

I can’t necessarily predict the future so I have no idea if 1.3 will happen, but I don’t see hwo that would impact the 1.2 issue?

The set of features unstabilized and yet-to-be-explored is quite amorphous and always changing, but is that necessarily required for staiblizing 1.2?

What is the end-goal? Why is the set of features amorphous? What are we trying to reach? All of this is still a bit foggy in my mind. That’s what I’m trying to get at. It would be nice if there was a clean list of goals and design requirements for macros 2.0, along with a well-laid-out plan for reaching it. The impression I have gotten is that we are stabilizing subsets as we are able to simply because we lack such a plan.

I don’t have any objections to the features proposed for 1.2, but I also don’t see where they fit into the big picture.

Anyway, thanks for taking the time to respond

[alexcrichton]

What I meant is do you want to stabilize those parts now (as part of macros 1.2 for the edition)?

Oh sure yeah, and yes it’s intended that the stabilization here would include https://github.com/rust-lang/rust/issues/35896, allowing you to import macro_rules! through the module system as well as procedural macros defined by crates.

All of this is still a bit foggy in my mind.

I agree! It’s a bit foggy to me too

I’m not actually sure what “true hygiene” really means other than “it works like you think it should more often”. I do, however, feel that 1.2 is an important step to take in that it’s well-defined and should help expand what we can do and make the jump to a hypothetical 2.0 a bit smaller.

[durka]

Is it fair to say that “stabilize without hygiene” means that hygienic item names will always be opt-in for macros 2.0? That seems, to me, much more concrete.

[nikomatsakis]

IIUC, it’s more that the API choices we’ve made force you to opt-in to “unhygienic names”, though that’s a lightweight thing to do.

[Ixrec]

I think this post:

what this “Macros 1.2” proposal is saying is that we’ll do exactly what macro_rules! does today. When you define an item with a macro it enters the same namespace as the surrounding scope, name collisions and all.

is saying the exact opposite of this post:

the API choices we’ve made force you to opt-in to “unhygienic names”

So… I’m back to having no idea what we’re proposing. Maybe we need to resort to talking about actual code samples?

macro_rules! x() {
    ($x:ident) => {
        let $x = 1;
        let x = 2;
    }
}

fn main() {
    let x = 42;
    x!(x);
    println!("{:?}", x);
}

Is this valid macros 1.2 code as currently proposed, and if so, what on earth does it print?

[durka]

Yeah, I’m now confused again as well.

Hygienic macros are macros whose expansion is guaranteed not to cause the accidental capture of identifiers. [1]

Macros 1.0 (macro_rules) is unhygienic for item names (and macro names) and hygienic for local variables. Since we’re only talking about item macros at global scope (which I still think is not so great for users), the second part is irrelevant.

What I was saying is, if we stabilize Macros 1.2 as item macros at global scope which put identifiers into the global namespace, then this is baked in forever – however we backwards-compatibly add hygiene will require an opt-in, whether that’s marking the identifiers or putting a #[hygienic] on the macro or whatever. That’s the can we’re kicking down the road, but stabilization sets a default.

So, yeah… I think we’ve demonstrated at least that “stabilize without hygiene” means different things to different people and needs to be fleshed out.

[nikomatsakis]

So, yeah, I think we’ve demonstrated that “stabilize without hygiene” means different things to different people and needs to be fleshed out.

I don’t see the conflict. @alexcrichton wrote:

When you define an item with a macro it enters the same namespace as the surrounding scope

This is what “unhygienic” means – identifiers that appear in the macro are injected into the surrounding context “as if” they were typed there. So if you have a macro like:

macro_rules! foo {
  () => { struct Foo { } }
}

and you do foo!();, that results in a struct Foo that you can name from outside the macro (the struct is not encapsulated within the macro). This is also the behavior you would get with procedural macros 1.2.

In terms of the API, the idea is that when you create that identifier token (the token Foo), it must be assigned hygiene information indicating, loosely, the scope in which the name is defined. All the APIs we will expose will act as though the name were defined at the point where the macro is invoked (the place where foo!(); appears), just as macro-rules does.

In the future, there would be new ways to create tokens (or to create the spans for tokens etc) that may permit other hygiene choices.

[nikomatsakis]

Is it fair to say that “stabilize without hygiene” means that hygienic item names will always be opt-in for macros 2.0? That seems, to me, much more concrete.

That said, to revisit what I wrote, I think this is a fair way of looking at it; it’s just that the “opt-in” that comes later may in fact be a very convenient API. =)

[durka]

I think the confusion may have just come from the phrase “force you to opt-in”. There is actually no action taken to opt-in. Perhaps we could call it “opt-in builtin hygiene” (OIBIH)?

[nikomatsakis]

There is actually no action taken to opt-in.

Basically yes. Unless my understanding is out of date, it’s something like this. When you create the token, you give it a span. We expect (though we could opt for a different approach) to carry hygiene information through these spans, as it often correlates very well with where the token actually appeared in the source text.

Right now, all the spans you can create use the hygiene information of “consider this information to be in scope at the macro-use site” (as opposed to being a name that is private to the macro definition site).