The future of syntax extensions and macros

[withoutboats]

The important thing is whether a macro declaration behaves like an item (function, struct, enum, etc.) or behaves like a macro use. If the latter then we have to be more restrictive about the syntax because it can appear in more contexts. Picking the former means we can have nicer syntax for the declaration.

I want to make sure I properly understand, so let me know if this is incorrect: the reason items can be disambiguated better than macro_uses can be is that they are declared with a keyword and can only appear in certain positions, correct? And the issue here specifically is that in order for macros to have that syntactic flexibility, macro needs to be a keyword.

I think it would be really exciting to have a procedural macro form that “looked like” an item. A lot of examples of DSLs in Ruby are functions that take symbols and blocks to make themselves appear like the Ruby analog for items (e.g. rspec). I myself have wanted to use macros to “create a new kind of item”, such as an actor! macro.

It would be a good goal, not a part of this RFC but a later one, to have item-like macros in the same sense that this proposal has both expression-like macros with #[macro] and attribute macros with #[macro_attribute] (that is, a third kind of macro declared with something like #[macro_item]).

I think there are a number of prerequisites this feature would have to meet. Particularly, it would have to able to take multiple different syntactic forms (e.g. some macro items want to look like functions, some want to look like structs), and it would have to be possible for the compiler to register the new false keyword such a macro has invented. These are the same things that would be required to make syntactic macros work as a procedural macro, so unless one of these prerequisites is impossible (I could see it being the case that creating false keywords would overcomplicate parsing) I think it would be a good idea to include the bang as groundwork for this feature.

[nrc]

That is mostly correct. Whether macro is a keyword or not doesn’t really matter. More that macro definitions are distinguished from macro uses and the former can only appear in item position.

Procedural macros can already act like an item, i.e., you could define foo and then have

foo! {
    x: Foo,
    y: Bar,
}

You can even have an ident form at the moment so you can make it look like foo! bar { ... }, although I would (not particularly strongly) like to remove that.

It gets more difficult to support pub ..., etc., there is no macro syntax for that sort of thing and it would be a bit unpleasant to parse.

The key thing I guess is that we would need macros which can only appear in item position, as opposed to current ones which can appear there and in other positions. I don’t know of any other use case for such a thing, but if there were I don’t see any fundamental reasons not to support it.

[withoutboats]

The key thing I guess is that we would need macros which can only appear in item position, as opposed to current ones which can appear there and in other positions. I don’t know of any other use case for such a thing, but if there were I don’t see any fundamental reasons not to support it.

If it allows them to be more syntactically flexible, there are probably amny macros which create things that act like items that would take the trade off, for the same reason you want to be able to declare macros in this less verbose way.

For example, a good syntax for declaring an actor in an actor library might be might be:

actor! Gostak(msg: Dosh, /*constructor arguments*/) {
    // a procedure describing distimming the Dosh
}

Which would be expanded into a Gostak struct which manages whatever concurrency primitive the underlying actor library uses, and has a constructor and a message receiving method to be used like:

 let distimmer = Gostak::new(/*constructor args*/);
 distimmer.send(dosh);

This hypothetical library would gladly take being declared only in the item position in exchange for being able to take arguments in this manner.

[hobofan]

I think I would prefer the pattern matching syntax of match which requires a comma between the blocks

pub macro foo {  
    (bar) => { ... }
    ($x: ident) => { ... }
}

would become:

pub macro foo {  
    (bar) => { ... },
    ($x: ident) => { ... }
}

From a users perspective I don’t really see a reason why it should be different from match, we are doing pattern matching in both after all.

[Uther]

You don’t need coma in match if you use blocks.

[hobofan]

Damn, I tested it before I wrote the reply to make sure, but where I tested it there was also a attribute between the match arms which then caused a syntax error that I attributed to the missing comma.

[rpjohnst]

Would it be possible to make all macros procedural, so that pattern-based macros are just procedural macros that don’t run any Rust code directly? Perhaps by providing the current token tree match-like syntax as a macro that generates code to operate on libsyntax types?

Since new macros are already changing the syntax, it shouldn’t hurt too much to go all the way and just use the macro keyword for procedural macros. I imagine it might look something like this:

pub macro foo(...) -> ... {
    rules! ... {
        (...) => { ... }
        (...) => { ... }
    }
}

The complication here is forcing all macros to talk about contexts and spans in their arguments, though on the other hand I can see it solving two other problems.

First, one-pattern macros could be simpler if these procedural macro items allow token tree patterns in argument position, like pub macro foo($x:ident). Since macros are a new item type it might also make sense just to refer to contexts and spans differently, though I don’t know how much sense that makes.

Second, the different types of macros could be distinguished by which libsyntax types are specified for arguments and/or return values, rather than by a separate set of attributes.

Overall, I think unifying both types of macros would be a lot more convenient for macro authors and for people learning how macros work.

[nrc]

Another blog post: http://www.ncameron.org/blog/procedural-macros-framework/

On the design of procedural macros

[nrc]

I intend to make available some macros or library functions for pattern matching in procedural macros. Together with quasi-quoting, one could do something a bit like this. However, I believe there is some utility in making macros by example more ergonomic. I view procedural macros as a mechanism of last resort, users should always prefer macros by example where possible. For one thing, it is much easier to introduce all kinds of bugs in procedural macros which are not possible in macros by example.

There are also some practical considerations - procedural macros must be compiled and run. Making that work when they are inline in a crate with normal code will take quite a lot of design and implementation code, but is table stakes for macros by example.

[rpjohnst]

Ah, that’s true about inline macros, though the main thrust of my suggestion was to make procedural macros ergonomic enough that the difference wouldn’t matter. Do you have anything in mind that would make procedural macros more bug-prone, besides the fact that you can leave the by-example stuff behind and venture into procedural territory?

Inline procedural macros is something that would be nice to have in the long run, though. This wouldn’t simplify the user-facing side, but maybe if/when that happens the by-example form could be implemented as a generator of procedural macros to simplify the compiler somewhat.

[colin_kiegel]

@nrc: For eager expansion of macros, I suggest foo!!() instead of $foo!(). This feels most natural to me, because it does not introduce completely new syntax.

I wonder how eager expansion would interact with nested macros. I am not sure: Is there a use case to have super-eager expansion foo!!!() etc. like below?

macro x () {
    macro y() {
        expand_on_use_of_y!();
        expand_on_def_of_y!!(); // .. that is on *use* of x, but after def of x
        expand_on_def_of_x!!!(); // .. is there a use case for this?
    }

    expand_on_use_of_x!();
    expand_on_def_of_x!!();
}

[colin_kiegel]

1) It has already been pointed out, that macro foo { /* pattern */ } { /* substitute */ } would have serious parsing issues. What if the pattern looks like a macro definition ($bar: ident) => { $bar }, or almost looks like a macro definition like ($bar: ident) => { $baz: ident }. And what if the substitute-block could just be interpreted as a normal block inside a function body? This is not only an algorithmic problem to the compiler, but also about readability by human beings.

To disambiguate this I feel like the shorthand notation for macros with only one matcher should always be with parentheses

• shorthand: macro foo ( /* pattern */ ) { /* substitute */ }
• vs. general: macro foo { /* 1-n matchers */ }

I think this limitation improves readability (and parsability) a lot. This means braces and brackets around the matchers could only be used in the general form. I think the readability improvement is worth of this tradeoff. Of course, you could also allow the bracket-style shorthand macro foo [ /* pattern / ] { /* substitute */ }.

2) Another question is, whether the delimiters (), [], {} should be part of the matcher or not, i.e. must the delimiters of the macro usage match with its pattern declaration and should it be possible that foo!() and foo![] expand differently, because the delimiters are part of the matcher?? I am not sure about this.

Currently it has no meaning and the caller can use whatever delimiter (), [], {} he likes, independent of the declaration.

My personal taste would be to let the caller decide which calling syntax he/she prefers and to restrict the delimiters for macro declaration to (..):

1. foo!{..} is just an equivalent to foo!(..); or {foo!(..)} independently of the macro declaration

• foo!(..) and foo![..] are equivalent independently of the macro declaration
• any macro pattern matcher (at the macro declaration) must always be delimited with parentheses (..), but never with braces {..}

Alternatively we could drop (2.) and allow bracket-delimited matchers [..] in (3.). In this case foo!{..} would be equivalent to {foo!(..)} but not to {foo![..]}.

My key point is that the ‘calling’-syntax foo!{..} does something differently at use-site, i.e. independent of the declaration of foo. At least this is my current tendency.

[nrc]

Another blog post - on libmacro - http://www.ncameron.org/blog/libmacro/

[nrc]

Last blog post! http://www.ncameron.org/blog/name-resolution/ This one is about the changes required to name resolution.

The next step will be turning these blog posts into an RFC or two.

[target_san]

My main concern here is that proposed macro system seems to work almost exclusively on token tree. Are there any plans on allowing macros into ‘typed TT’ phase? Or, adding some kind of hook ‘execute when X gets typed’?

[nrc]

There are no plans in this direction at the moment. It might happen in the future, but such a facility is not really a macro any more it is a compiler plugin.

[target_san]

Sure it’s a compiler plugin duty. Though as I understand all meta-facilities are called ‘macros’ in Rust, which are split into syntactic ones (like macro_rule!) and procedural ones (aka compiler plugins).

[nrc]

I should be clearer about terminology. We have in the past used “compiler plugin” (and “syntax extension”) to refer to procedural macros. Macros (whether macros-by-example, currently using macro_rules!) or procedural are purely syntactic, they only interact with the parser, not the rest of the compiler. Type information is not available until much later in the compiler, by which time all macros have been expanded and have disappeared. So, a compiler plugin (in my sense of the concept) with access to type information is a very different beast from a macro (even a procedural macro) which only has access to the syntax.

[Zoxc]

I’d like for whitespace, comments and newline tokens (and other things hidden) to be in the token trees passed to procedural macros. The lexer will produce these tokens and they are trivial to ignore. Trying to recover this after is a lot more (ugly) work and I’m not sure if it possible in all cases (macros in macros aren’t too easy to reason about).

I have written some code to see if there’s whitespace to the left or right in tokens: https://github.com/AveryOS/avery/blob/daae05f04b24b1224b7381c9a416041c3186d1fd/vendor/asm/assembly.rs#L249

It is not pretty and I’d rather not see this duplicated in any procedural macros.

[nrc]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.