Help stabilize a subset of Macros 2.0!

[steven099]

So… I’m back to having no idea what we’re proposing. Maybe we need to resort to talking about actual code samples?

macro_rules! x() {
    ($x:ident) => {
        let $x = 1;
        let x = 2;
    }
}

fn main() {
    let x = 42;
    x!(x);
    println!("{:?}", x);
}

Is this valid macros 1.2 code as currently proposed, and if so, what on earth does it print?

The key point is that the hygiene being discussed is for procedural macros. IIUC, macro_rules pseudo-hygiene isn’t being changed (correct me if I’m wrong). Full hygiene in declarative macros is slated for the macro feature which isn’t part of macros 1.2. Forgive me if I’m mistaking the source of confusion.

[alexcrichton]

Ah yes to reiterate what @nikomatsakis already said, this is why Term::new in the proposed API requires a Span. That’s the main token for carrying hygiene information, so forcing you to specify Span::call_site() is forcing you to opt-in to unhygienic macros (there’s no other stable alternative)

[petrochenkov]

Tomorrow I’m going to go through the stuff proposed for stabilization and review as much as I’ll be able to.
I have a couple of concerns that can be fixed by tightening few rules before stabilization, but the proposed subset looks safe enough to me (even call_site hygiene, which is a well-defined concept despite all the misunderstanding around it).

UPDATE: Reviewing in progress, will be happening over the next week.

[durka]

If I understand right, @Manishearth’s excellent blog post underscores the dangers of omitting hygiene. Of course, macro_rules is also vulnerable to that issue, and we don’t see lots of people running into it (probably because nobody names their constants __arg_0). But he does tantalizingly say that the derive code (which is basically advanced proc macros) could support hygienic idents, so… maybe it isn’t too far away from being exposed to proc macros?

I think I’m understanding more of what you’re going to do though:

• In the 2018 edition, you’ll be able to write a procedural macro and in places where you create an ident, you have to “choose” a Span but there is only one choice, Span::call_site()
• In a future rustc version after the 2018 edition, other choices will be stabilized, like Span::gensym() or whatever, so macros written for the 2018 edition can be upgraded

That’s a bit better than what I thought originally, which was that we’d be stuck with Macros 1.2 macros that could never be made hygienic. This doesn’t change my worry about trumpeting new macros yet again when they will still have fairly obvious warts. But it seems like y’all are pretty set on the plan, so I’m glad that my interpretation got more positive

[Manishearth]

The fix for derive code is https://github.com/rust-lang/rust/pull/49986 , and it basically works by just calling gensym everywhere. We could expose something similar to proc macros.

I wish we had a &ProcMacroContext argument to proc macros upon which we can expose all these methods.

[Ixrec]

The discussion here makes it sound like Span::call_site() and Span::gensym() are the only two obvious/common options, so it’s not clear why only one of them is stabilizable. Are there problems with gensym? Or a spectrum of subtly distinct definitions of “hygenic” that I’m unaware of? Why can’t in-between options be developed later and both of these simple options be part of “macros 1.2”?

[petrochenkov]

@Ixrec

Gensyms as they exist now are not even hygiene, they are an orthogonal mechanism.

gensym("my_name") effectively creates a symbol with unique string name that can’t be resolved neither at def site nor at call site unless you literally copy it (think of appending a really long hash you can’t guess like my_name_72640a7489c37408342dfe483934393).
On the other hand, if you know that unique name then you can combine it with any kind of hygiene (def site, or call site, or no hygiene) and it will be resolved normally using that hygiene.

https://github.com/rust-lang/rust/pull/49986 is a somewhat hacky solution emulating def site hygiene (the proper solution) with gensyms.

Regarding proper hygiene based on syntactic contexts, I can think of three variants that can be useful:

• Def site - this is what macro does for everything by default and what macro_rules! does for local variables (and also labels).
• Call site (going to be stabilized in macros 1.2) - this is what “hygiene opt-out” in macro PR (https://github.com/rust-lang/rust/pull/47992) does.
• “Transitive call site” aka No hygiene - this is what macro_rules! does with everything except for local variables.

[durka]

Sorry to make this thread even longer but I feel as though something just got “de-clarified” again. What is the distinction you’re drawing between call-site and transitive-call-site?

[petrochenkov]

@durka
Suppose i is an identifier with definition-site context, #i is an identifier with call-site context, and 😂i is an identifier with transitive call-site context, then:

macro level0() {
    let i_def_site = i; // `i` is resolved from this location
    let i_call_site = #i;
    let i_transitive_call_site = 😂i;

    struct S; // Can refer to `S` from this location
    struct #S;
    struct 😂S;
}

macro level1() {
    level0!(); // `#i` is resolved from this location
                   // Can refer to `#S` from this location
}

macro level2() {
    level1!();
}

// ...

macro levelN() {
    levelN_minus_1!();
}

fn main() {
    levelN!(); // `😂i` is resolved from this location
                   // Can refer to `😂S` from this location
}

EDIT: Updated the example with definitions (S) in addition to uses (i).

[petrochenkov]

An issue about invocation syntax for procedural macros 1.2:

Issue: What exactly token streams are passed to procedural macros 1.2

[durka]

Does that mean that a macro using call-site hygiene (say, a Macros 1.2 proc macro) can’t refer to anything (except using absolute paths) if it’s expanded as part of the expansion of another macro? I mean, if #i is resolved from the location of “inside level1!'s expansion”, it seems like there is nothing there to resolve to.

[petrochenkov]

@durka

I don’t entirely understand what you mean. No it’s not necessary to use absolute paths, you can refer to any names that are in scope in the location inside level1.
Yes, in my example there are no useful names to which you can refer from that location except for the level* macros themselves and main (and std prelude, and primitive types!), but in actual code there may be such names.

[petrochenkov]

How to determine hygienic context for multi-token entities.
Not a blocker in general, but it would be nice to decide what token represents call site for macros and attributes, and implement the decision (should be very minor work).

Issue: How to determine hygienic context for "non-atomic" code fragments

[petrochenkov]

Visibilities of “private” items from macros 1.2.
Probably no action required, just something to understand and document.

Issue: Visibilities of "private" items from macros 1.2

[petrochenkov]

How to determine hygienic context for the “crate root” in absolute-by-default paths.

Issue: How to determine hygienic context for the "crate root" in absolute-by-default paths

[petrochenkov]

Proc macro API 1.2 review.

[WIP] Review proc macro API 1.2

[mark-i-m]

@petrochenkov Thanks for posting all of these links here. I appreciate your effort to keep us all informed

[petrochenkov]

Tokens parsed from strings don’t get Span::call_site spans.

I think we need to change this and also change what Span::call_site returns to fix macro backtraces, clippy and edition hygiene for tokens using call-site spans.

Issue: TokenStream::parse does not resolve with Span::call_site()

[jjpe]

Is there anywhere I can read up on how to use procedural (i.e. non-derive) macros 2.0? I want to use it for syntactic manipulation in a way that macro_rules just doesn’t provide: implementing an EBNF-based parser generator. The parser generator already exists, but the macro_based version is still pretty bloated, especially for more complex rules.

[djc]

Are people thinking about applying macros to the module level (so, #![my_cool_macro]), and if so, will that be part of macros 1.2?