Macro meta functions

RFC #3086 defines a metafunction syntax for declarative macros, e.g. ${ignore(...)}. As part of determining exactly what the "arguments" should look like, I've been tasked with collecting a list of currently proposed metafunctionality (included below). I'd also like to ask everyone else: what other kinds of metafunctionality would be useful for writing declarative macros?

metafunction argument kinds purpose source
assign (new) binder, token stream assign a binder for later use in expansion Zulip: macro_rules back to repetition root (comment)
call_site, etc. token tree / token stream specify a named hygiene for token expansion @CAD97 spitballing
concat ? concatenate identifiers into a new identifier rust-lang/rfcs#3086
copy_span binder, token tree apply the span of a binder to a token tree An idea for breaking full macro hygiene
copy_span token tree, token tree apply the multispan of one token tree to another An idea for breaking full macro hygiene
count binder get the number of times a macro binder repeats rust-lang/rfcs#3086
count binder, integer get the number of times a macro binder repeats at a given repetition depth rust-lang/rfcs#3086
eager token stream do eager expansion of macros @CAD97 spitballing
if ? do control flow things in macro expansion like a template engine Zulip: macro_rules back to repetition root (comment)
index n/a get the current repetition index rust-lang/rfcs#3086
index integer get the repetition index at a given repetition depth rust-lang/rfcs#3086
ignore binder treat the binder as used for the purpose of repetition without expanding it rust-lang/rfcs#3086
ignore token stream do expansion of the token stream for side effects (e.g. controlling repetition) but discard the results rust-lang/rfcs#3086
match ? do control flow things in macro expansion like a template engine Zulip: macro_rules back to repetition root (comment)
? ? Cartesian product of repetitions rust-lang/rust#74524
6 Likes

Maybe this (metafunctions) can be used to improve the generics situation for declarative macros. There could be a generics fragment and then some metafunction that does something similar to syn::Generics::split_for_impl.

For example, currently its quite easy to accept simple Rust types in declarative macros:

pub trait Trait {}

macro_rules! simple_derive {
    (
        $(#[$attr:meta])*
        $vis:vis struct $name:ident {$(
            $(#[$field_attr:meta])*
            $field_vis:vis $field_name:ident : $field_ty:ty
        ),* $(,)?}
    ) => {
        $(#[$attr])*
        $vis struct $name {$(
            $(#[$field_attr])*
            $field_vis $field_name : $field_ty,
        )* }
        impl Trait for $name {}
    }
}

simple_derive!(
    /// Attribute
    pub struct Example {
        /// Attribute
        pub value: usize,
    }
);

But it becomes harder when the types are generic, so maybe something like the bellow could be allowed to make it easier:

macro_rules! simple_derive {
    (
        $(#[$attr:meta])*
        $vis:vis struct $name:ident
        $generics:impl_generics $(where $bounds:bounds)? 
        {$(
            $(#[$field_attr:meta])*
            $field_vis:vis $field_name:ident : $field_ty:ty
        ),* $(,)?}
    ) => {
        $(#[$attr])*
        $vis struct $name $generics $(where $bounds)? {$(
            $(#[$field_attr])*
            $field_vis $field_name : $field_ty,
        )* }

        impl $generics Trait 
        for $name ${ty_generics($generics)}
        where
            ${bounds_generics($generics)}
            $($bounds)?
            $($field_ty: Trait,)*
        {}
    }
}

simple_derive!(
    /// Attribute
    pub struct Example<'a, T, const N: usize> {
        /// Attribute
        value: &'a [T; N],
    }
);
4 Likes

Am I the only one who finds that ad-hoc definitely incomplete list of syntax extensions for an already complicated system an abomination?

5 Likes

Note that all of these are just proposed ideas, and every one I could find at that. The purpose of collecting this list is specifically to define what we actually need/want out of declarative macro metafunctions (and if we want them at all or should unaccept that part of RFC#3086).

And if you're saying the list itself is just poorly formatted: sorry, I tried my best :frowning_face:

The idea of providing some of this functionality is that it will allow more macros which are currently required to use the much more complicated proc macro system to become decl macros instead. So in that way, the idea/hope is that by giving decl macros more power we decrease the required complexity of macros which can then move from being implemented as proc macros to being decl macros instead.

3 Likes

Hmm I do not really like the assign operation, instead I would like to see more classic control flow operators on meta variables. When I first learned macros, I asked myself why they were not implemented using those and I have not found a deal breaking problem with them.

So why do we not just allow something like (example from the assign zulip link):

macro_rules! layered_repetitions {
    // Input
    {
        $($path:ident).* {
            $($field:ident),*
        }
    } => {
        // Output
        $@for $field in $field {
            // Repeat complete $path for every $field
            $($path).*
            $field
        }
    }
}

Syntax of course open for bike shedding later, it might even be possible to need no additional prefix.

If you want to iterate over two variables at the same time, then

macro_rules! print_fields {
    {
        struct $name:ident {
            $($fields:ident: $types:ty),*
        }
    } => {
        fn print_struct(s: &$name) {
            $@for ($field, $type) in ($fields, $types) {
                println!("{}: {:?} ({})", stringify!($field), s.$field, stringify!($type));
            }
        }
    }
}

There might even be some more syntax to also add indexing and other goodies. I could also imagine the use of other control flow items such as match, if etc.

3 Likes

To my mind, this work is a result of Rust still firmly stuck in the sunken cost fallacy. We are still trying to add on more workarounds on top of an unfit design. The current macro_rules system as it currently stands has foreign syntax and semantics and simply isn't fit for purpose, given we all hope Rust would be the language of the next 50 years.

We could do much better by taking inspiration from Nemerle and more recently https://www.circle-lang.org/, projects that have much better thought out macro systems. E.g instead of trying to force a scheme like recursive macro definition and trying to add to it metafunctions to count numbers of repetitions, wouldn't it be way WAY easier to just use plain Rust for loops and iterators?

20 Likes

Match? If? Index? Assign?!? Sounds like people are itching to make a poor half-baked informally specified and bug-ridden implementation of half of Rust inside of the template macro system.

It's like someone looked at C++ and said "you know what cool feature Rust misses? Template metaprogramming!"

9 Likes

I feel like there is some middle ground between "We implement C++ templates" and the current macro system.

The macro system has some poor limitations and one is thus required to write a proc-macro. I do not like that, because many people will never want to touch proc-macros and they always are such a hastle to integrate into your projects (add a helper crate etc.). I definitely agree with you that we do not want to create a half baked system, but if we can improve the current marco system in such a way that it is easier to understand and a little bit more powerful then I think we should do that.

For example, why do I need a proc-macro to concatenate idents?

I like the argument destructuring approach where you specify how the input looks like, but expanding the macro itself should be more ergonomic.

iirc in the future macro will be used to declare macros, we could use that opportunity to also define a new macro syntax that would deal with these problems.

6 Likes

Some metafunction to break an AST component back into tokens?

Anyway, I agree with @afetisov that this is probably a bad idea. Resorting to proc macros isn't so terrible; they can be way more readable.

Even ignoring the cost of bringing in syn, I personally find

macro_rules! rpc_impl {
    {
        fn $f($($arg:ident: $Arg:ty),* $(,)?) $(-> $Ret:ty)? {
            $($body:tt)*
        }
    } => {
        fn $f($($arg: u32),*) $(-> ${ignore(Ret) u32) {
            let _ret = {
                $(let $arg = <$Arg>::rpc_decode($arg);)*
                $($body)*
            };
            $(<$Ret>::rpc_encode($arg))?
        }
    }
}

much easier to understand than all of the ceremony required to set up an attribute macro.

Declarative macros are also much simpler for IDEs to support (even with more complicated meta functionality) because they're still purely declarative, so the IDE can know that e.g. you're editing in $($body:tt)* above and that editing there won't change the macro expansion. Proc macros are formally impossible to track in such a fashion because they're defined procedurally.

Then there's also the extra bit that proc macros have to be compiled separately from the main crate. This can be improved in the future such that it can be part of the same package, but it will always need to be a separate build-time part rather than just in the normal crate.

7 Likes

I agree with @CAD97 proc-macros are very useful, but I think we can make decl macros work for many cases where we currently need a proc-macro.

What also came to my mind was named matches (I do not know a better name). So for example:

macro_rules! my_macro {
    ($p(print)? $a:ident + $b:ident) => {
        if ${exists($p)} // how to specify print here?? {
            println!("{} + {}", stringify!($a), stringify!($b));
        }
        $a + $b
    };
}

So I would like to assign a name to some fixed construct in the input. A workaround is at the moment to use ident/tt and then later in a helper macro require the concrete construct, but then the error is less readable and the macro always drags around an ident that actually is only ever something constant.

I do not know if this fits here, but I think that we could use better support for inner helper macros. They should not be callable from outside the outer macro and should be easier to declare (maybe another parameter syntax? but this might be a bit too extreme).

2 Likes
  • The assign one is cute; we'd have to see how it fares: I don't see it as paramount, but I imagine it being quite convenient :slightly_smiling_face:
  • concat is so direly needed that it could indeed warrant being special-cased as a special metavariable function.

But the main thing we need, really, is this so-called eager.

An eager expansion operator such as ${macro!(…)} is a Game Changer

Nothing[1] more is needed. That is, rather than trying to come up with dozens and dozens of metavar combinations, we can just define custom ones with macros and eager.

For instance, to concatenate identifiers:

#[test]
fn ${concat_idents!($fname, _test)} ()

to replace ${concat(…)}, as well as ${respan!(…)}, and so on and so forth.


This means that a very important question to sort out is whether this eager mechanism will be featured:

  • If so, then let's not overcomplicated metavar expressions, and just let them defer to user-defined helper macros (now, the stdlib could consider adding more of those, but then these would just be T-libs question rather than T-lang ones!);

  • If not, then we'd need T-lang extensions for every new usability need, or just a lack of support as we currently do.


Aside: unrelated, but the "named" capture groups idea, which has already been brough before, in this very thread, and in this forum as a standalone thread, is indeed quite important to get rid of the $(@$($name:tt)?)? hacks that are so pervasive nowadays (in other words, the current status quo in this regard is horrendous). But since named capture groups is yet another whole level of macro modification, I think a way simpler and more elegant solution, which was also suggested in that thread (but not in this one!), is to be able to have :empty capture groups.

So, to take @y86-dev's example, this would give:

macro_rules! my_macro {(
    $(print $if_print:empty)?
    $a:literal + $b:ident
) => ({
    $($if_print
        println!(…);
    )?
    $a + $b
})}
  • Currently, we have to write:

    macro_rules! my_macro {(
        $(print $(@$if_print:tt)?)?
        $a:literal + $b:ident
    ) => ({
        $($($if_print)?
            println!(…);
        )?
        $a + $b
    })}
    

  1. EDIT: this term was a bit too adamant, my bad. Obviously there are still meta(var)-based functionality such as index, count, and whatnot which would require dedicated metavar functions, @CAD97, you're completely right :100:. But I suspect there won't be that many of those: the vast majority of use cases ought to be manageable using macro helpers ↩︎

7 Likes

That's not quite true: index and length (the RFC-accepted ones) aren't doable with just macros, as they need access to compiler context. assign is just a cute assistant, but also isn't possible without compiler support. But I think you're correct on the rest.

said macros
macro call_site($($tt:tt)*) { /* proc macro */ }
macro concat_idents($($i:ident)+) { /* proc macro */ }
macro with_span($donor:tt=> $($tt:tt)*) { /* proc macro */ }

macro count($($tt:tt)*) {
    0 $( + 1 ${ignore!($tt)} )*
}

macro if_tt {
    ( if ($($tt:tt)+) { $($then:tt)* } else { $($else:tt)* } } => { 
$($then)* };
    ( if (          ) { $($then:tt)* } else { $($else:tt)* } } => { $($else)* };
}

macro ignore($($tt:tt)*) {}
3 Likes

count! can't be done with a macro like that if you need a literal rather than an expression, e.g. if you want to use count! with concat_idents!.

Then you can use eager expansion twice.

This doesn't work; eager expansion only applies to macro expansion, and the count I provided expands to an expression like 0 + 1 + 1 + 1 + 1.

2 Likes

Ah, correct.