Help stabilize a subset of Macros 2.0!

[jjpe]

Is there anywhere I can read up on how to use procedural (i.e. non-derive) macros 2.0? I want to use it for syntactic manipulation in a way that macro_rules just doesn’t provide: implementing an EBNF-based parser generator. The parser generator already exists, but the macro_based version is still pretty bloated, especially for more complex rules.

[djc]

Are people thinking about applying macros to the module level (so, #![my_cool_macro]), and if so, will that be part of macros 1.2?

[reinit]

You won’t use use to import macros within your own crate but will rather instead rely on today’s scoping rules.

Why macros was designed to be not namespaced? Because if they do, then there will be no difference between where the macro is located, regardless it’s in my crate or not.

Maybe everything in Rust should be namespaced so the language is more intuitive (Less things to learn, friendly to it’s user)? Please consider it when you guys start to working on Rust 2.0

[Ixrec]

That is exactly how macros 2.0 already works, which is what this thread is talking about stabilising.

[ExpHP]

Why macros was designed to be not namespaced? Because if they do, then there will be no difference between where the macro is located, regardless it’s in my crate or not.

Maybe everything in Rust should be namespaced so the language is more intuitive (Less things to learn, friendly to it’s user)?

I think local macros still require #[macro_use] so that the compiler knows what order in which to expand them. After all, macros can expand to other macro definitions and even modules!

If you could use them then I think it would probably need weird restrictions like "use foo::my_macro must come after the declaration of mod foo;" (similar to the limitations of #[macro_use]).

(But I haven’t checked myself; maybe @Ixrec is right for macro macros?)

(Update: he is! )

[steveklabnik]

Why macros was designed to be not namespaced?

macro_rules is vestigial; we reserved the macro keyword for a real macro design, which is what macros 2.0 is. We needed to ship Rust 1.0 and had to make some hard choices; this was one of them.

[Ixrec]

(But I haven’t checked myself; maybe @Ixrec is right for macro macros?)

Yeah, I believe the main purpose of https://github.com/rust-lang/rfcs/pull/1560 was to specify how this could possibly work in the face of properly namespaced/module-ized macros.

[ExpHP]

Nah, I think RFC 1560 was the one that made this work:

mod parent {
    use self::child_1::x;
    mod child_1 {
        pub fn x() {}
    }

    mod child_2 {
        use parent::x; // In 1.0.0 this produces "unresolved import"
    }
}

There is an example in 1560 where a glob import appears to import a macro (after the quote “this example assumes modularised macros”), but it isn’t the main focus. I also don’t see any discussion of use on the declarative macros 2.0 tracking issue.

[ExpHP]

Some more sleuthing:

• RFC 1561 (macro naming and modularization) seems to suggest that use should work on declarative macros 2.0.
• Talk in the tracking issue also seems to take this as a given. I.e. it is only macro_rules! macros that should get the short end of the stick.
• The “roadmap” for that issue has every box ticked with no explicit mention of decl macros 2.0.
• declarative macros 2.0 still has no documentation

Baaaahhhhhh, you know what? Documentation or no documentation, I’m just going to copy the first thing I see on the tracking issue, tweak it until it compiles, and see what happens:

#![feature(decl_macro)]

fn main() {
  println!("{}", m::m_helper!());
}

mod m {
  pub macro m_helper() {
    3
  }
}

Output:

3

Whew! So there you have it. Contrary to what I previously said, it appears that use-able, locally-defined macros are not only planned, but they are already implemented (at least partially). However, it is only for declarative macros 2.0 (macro, not macro_rules!)

[mvduin]

In fact pretty crazy things already seem to work:

#![feature(decl_macro)]

fn main() {
    foo::Foo.poke();
}

mod foo {
    super::def_struct!( Foo );
}

macro def_struct( $name:ident ) {
    struct Foo;
    impl Foo {  fn msg( &self ) -> &str { "yo!" }  }

    pub struct $name;
    add_poke!( $name, Foo.msg() );
}

use m::add_poke;

mod m {
    pub fn show( msg : &str ) { 
        println!( "{}", msg );
    }

    pub macro add_poke( $t:ty, $msg:expr ) {
        impl super::Pokeable for $t {
            fn poke( &self ) { 
                show( $msg );
            }
        }
    }
}

trait Pokeable {
    fn poke( &self );
}

Notice the struct $name in def_struct (where $name is Foo) injects a definition into the caller, while the struct Foo in the same macro is invisible and does not conflict with it, yet the latter can be used in an expr passed to another macro that implements a trait for the first struct. Holy cow, and this works right now.

The same hygienic magic (I presume) that’s keeping Foo from conflicting with Foo can also have some unfortunate results though, such as when I tried to access a field of a struct created by a macro:

5 |     foo::X.child.poke();
  |            ^^^^^ did you mean `child`?

I also noticed that m::show has to be public currently. I can imagine why, but I hope that requirement will go away eventually.

[jjpe]

Notice the struct $name in def_struct (where $name is Foo ) injects a definition into the caller, while the struct Foo in the same macro is invisible and does not conflict with it, yet the latter can be used in an expr passed to another macro that implements a trait for the first struct. Holy cow, and this works right now .

While I’m excited to see a macros incarnation in a form that plays nicely with the regular import system, this part really confuses me. Why would 2 struct definitions with the same name in the same scope not conflict with one another, even if there is a difference in how public they are?

This is exactly opposite to what happens on the item declaration level for example, where a struct Foo; and a macro expanding to pub struct Foo; would definitely cause a compile error (granted, it’s macro_rules, but I’d expect the same behavior from any macro system in the sense that they function by means of compile time expansion. It’s then the job of non-macro rustc to compile that expanded code, with the resulting error):

struct Foo;

macro_rules! foo {
    () => {
        struct Foo;
    };
}

foo!();

With output:

Compiling playground v0.0.1 (file:///playground)
error[E0428]: the name `Foo` is defined multiple times
 --> src/lib.rs:5:9
  |
1 | struct Foo;
  | ----------- previous definition of the type `Foo` here
...
5 |         struct Foo;
  |         ^^^^^^^^^^^ `Foo` redefined here
...
9 | foo!();
  | ------- in this macro invocation
  |
  = note: `Foo` must be defined only once in the type namespace of this module

error: aborting due to previous error

For more information about this error, try `rustc --explain E0428`.
error: Could not compile `playground`.

To learn more, run the command again with --verbose.

Or was your point exactly that it’s crazy/undesirable that this is allowed?

[Nemo157]

These two structs are allowed because of hygiene, while they are both named Foo they are named with the identifier Foo from two different hygiene contexts. Current macro_rules! macros are unhygienic for items, so you get the conflict from the two different definitions, compare with the current behaviour for non-item values (playground):

macro_rules! foo {
    ($foo:expr) => {
        let foo = 5;
        println!("foo in macro {}", foo);
        println!("$foo in macro {}", $foo);
    }
}

fn main() {
    let foo = 4;
    foo!(foo);
    println!("foo in main {}", foo);
}

gives

foo in macro 5
$foo in macro 4
foo in main 4

The new decl_macro macros are “more hygienic” than the current macros so have the same behaviour for items as the current ones do for non-items (and will probably have a way to opt-out of hygiene for when you need the current behaviour).

[jjpe]

I see, thanks for the explanation

So would it then be a good intuition to say something like “code in a macro expansion that uses macro arguments can live next to otherwise-identical code in the same expansion that uses literal identifiers instead of the macro arguments”?

And if not, are the new hygiene rules perhaps documented somewhere?

[mvduin]

It simply depends on where the identifier comes from: each invocation of a macro has a unique namespace of sorts for literal identifiers introduced there, while identifiers passed as arguments retain knowledge of their original namespace, e.g.:

macro def_struct_Foo()          {  struct Foo;  }
macro def_struct( $name:ident ) {  struct $name;  }

struct Foo;
def_struct_Foo!();  // ok, this "Foo" is private to the macro invocation
def_struct_Foo!();  // still ok, different invocation hence different "Foo"
def_struct!( Foo );  // error: Foo redefined

[mvduin]

Note that since the private identifiers introduced by macros in this way cannot be directly referenced from outside the macro invocation, these would generally be used for items used locally inside the macro, e.g.:

#![feature(decl_macro)]

fn main() {
    println!( "{}", add_one( 0 ) );
    println!( "{}", add_ten( 1 ) );
}

macro def_adder( $name:ident, $addend:expr ) {
    static ADDEND : i32 = $addend;

    fn $name( value : i32 ) -> i32 {
        return value + ADDEND;
    }
}

def_adder!( add_one, 1 );
def_adder!( add_ten, 10 );

Without hygiene, the two ADDENDs would conflict.

[alexcrichton]

As a heads up, as of the past few nightlies this is now stabilized! The final feature, use_extern_macros was stabilized by @petrochenkov pushing this over the finish line. Congrats to everyone involved!

I’ll be attempting to write up some more extensive official documentation in the coming weeks for the 1.30 release.

[jjpe]

In the context of Rust (as opposed to e.g. Common Lisp) I have come to appreciate the value of hygiene in macros as I attempted to write a macro outer that, upon expansion, defines a second macro inner. The expansion of inner, then, depends on input to outer, so it’s somewhat analogous to the abstract concept of a closure.

The macro_rules construct is insufficient at this, and generates a compiler error. Therefore I had a look at the macro construct. While it seems like it might do the job it raises another question for me: How do I do something like “macro overloading” (i.e. accepting different input => output definitions), similar to macro_rules? Is that even possible ATM?

[matthewjasper]

macro macros can use the same syntax as macro_rules!, as well as the new syntax.

[jjpe]

I’m not sure why, but that’s apparently not 100% true. Consider this code:

pub use ast;

pub macro ast {
    ($name:expr) => {{ // empty node
        // omitted    
    }};
    ($name:expr; [ $($child:expr),+ ]) => {{ // branch / parent node
        // omitted
    }};
    ($name:expr; $data:expr) => {{ // leaf / data node
        // omitted
    }};
}

This gives me this error:

   Compiling parser_generator v0.1.0 (file:///home/jjpe/dev/test)
error: no rules expected the token `(`
  --> src/lib.rs:26:5
   |
26 |     ($name:expr; [ $($child:expr),+ ]) => {{ // branch / parent node
   |     ^

error: aborting due to previous error

error: Could not compile `test`.

This combination of patterns works fine in macro_rules!, but it seems that even having a second pattern is not syntactically valid for macro.

[matthewjasper]

It appears that macro uses , as the separator between arms.