On macros being second class to builtins

[jgarvin]

As a new user I was disappointed to learn rust macros can't replicate syntax of builtins because they can only take one argument set. In order for user-defined flow control to not be second rate compared to built-ins, it should be possible to take a snippet using any of the built-in keywords:

while true {
    ....
}

and to add a ! after the language keyword:

while! true {
    ....
}

and have it be possible to define while! such that both snippets compile and have the same behavior . Currently macro_rules can't express the above because of true appearing before the braces. This would go for all of the major keywords, for, fn, etc. I understand that rust macros are not based on text substitution, I'm not proposing that while! would simply be text substituted with while, but rather that there would be some way to express that the macro can be followed by an expression and then a pair of braces containing statements.

I ran into this trying to port the C++ Catch test framework to rust as an exercise. It features a fairly novel control flow feature, where you write something like:

TEST_CASE("foo") {
    // code common to all tests goes here
    // declare some variables, init some things

    SECTION("a") {
        SECTION("x") {
            ...
        }
        SECTION("y") {
            ...
        }
    }
    SECTION("b") {
    }
    SECTION("c") {
        SECTION("z") {
            ...
        }
        SECTION("w") {
            ...
        }
    }
}

And it then runs the test in a loop, where each iteration runs different sections:

• Iteration 1: a, x
• Iteration 2: a, y
• Iteration 3: b
• Iteration 4: c, z
• Iteration 5: c, w

So each iteration is going to a different leaf in the section "tree." This makes tests much more succinct to write and avoids the need to define test fixtures entirely (all the code the setup that normally goes in a fixture just goes in the test case above all the sections).

So far I have failed to be able to replicate the syntax or something close with macro_rules. Either one of:

test_case! "mytestname" {
    section! "a" {
    }
}

Or:

test_case!("mytestname") {
    section!("a") {
    }
}

would be great, but neither are expressible syntax. Catch is able to get away with it because C++ just uses text substitution, and it's disappointing that Rust's much more sophisticated system can't (I think?) do the same. The closest I can get is:

test_case! {
    "this is kind of like a docstring in Python";
    // now everything else is real code

    section! {
        "a";
        // now the code
    }
}

Which I think is less clear, it's not obvious to a newcomer that the first line is special. Not being able to express this sort of thing would also prevent being able to write any loop construct as nice looking as for or while.

Are there any plans to address something like this? Am I mistaken and there is already a way to do this?

[robinm]

I am not really familiar with the details of macro, but I think that in order to keep a LL(1 or 2) parser, this kind of syntax cannot be valid.

I think you could do something with attributes though.

[CAD97]

macro_rules! itself used to be a "regular" macro (from the grammar perspective), just that no user macro could be defined with that syntax. Currently, it's its own kind of thing in the syntax tree.

I posted a topic a while back proposing the idea of trying to bring back macro_name! foo { /* */ }. It didn't get much discussion, mainly because there's just not all that much demand for it.

If you're willing to dip your toes into procedurally defined macros, they can have whatever syntax within you want, so you could get something like

catch::tests! {
test_case!("foo") {
    // code common to all tests goes here
    // declare some variables, init some things

    section!("a") {
        section!("x") {
            ...
        }
        section!("y") {
            ...
        }
    }
    section!("b") {
    }
    section!("c") {
        section!("z") {
            ...
        }
        section!("w") {
            ...
        }
    }
}
}

because within the proc macro, the only requirement on input is that it tokenizes, and you can manipulate said tokens however you want to output legal Rust code.

A less drastic option would be to look more like the standard #[test] framework:

#[catch::test_case]
fn foo() {
    // code common to all tests goes here
    // declare some variables, init some things

    section!("a") {
        section!("x") {
            ...
        }
        section!("y") {
            ...
        }
    }
    section!("b") {
    }
    section!("c") {
        section!("z") {
            ...
        }
        section!("w") {
            ...
        }
    }
}

though much the same processing would likely be required to implement it.

[jgarvin]

@CAD97 Say for the sake of argument I implemented the last proc macro. And that I had another proc macro test_case2 that introduced section2. Could I stack both test_case and test_case2 on foo, and use both section and section2 inside? Or would the inner one error from not knowing the syntax of the outer one? (I.e. do proc macros compose?)

[CAD97]

Proc macros compose... if they've been designed to facilitate it.

Just as a quick example, given

#[outer]
#[inner]
fn f() {}

the proc macro outer will receive the token stream

# [ inner ] fn f ( ) { }

and have to re-emit whatever #[inner] is going to run on (or even decide to not emit #[inner] and prevent that attribute from evaluating).

I should definitely note, though, that in the attribute version the fn item has to parse as a normal item. I messed up and that specific syntax isn't allowed. You can get close (and why I thought this might've worked before testing it) by abusing parsing the { ... } as blocks and just fixing up semantics in the proc macro.

[Manishearth]

Note that while! expr {} as used in your examples would not work for things like, it's ambiguous with while !expr

As @CAD97 mentioned there was indeed an internal concept of foo! identifier {} macros (called IdentTT). There was some prior discussion to maybe add them as first class features, but it didn't really go anywhere. I don't think anything has changed since then, so I wouldn't expect such a feature to get accepted.

[jgarvin]

Your 2 examples look identical, only diff is stable vs nightly compiler choice. Is that what you meant? Also two periods vs three, but I only ever meant the periods to be placeholders for real code.

[jgarvin]

Why is while !expr ambiguous? Shouldn't exclamations have to occur right after the macro name to be part of the name?

[jgarvin]

Could I possibly do:

#[test_case("foo")
{
    #[section("a")]
    {
        // code here
    }
}

the documentation seems to say that attributes can apply to blocks. I don't know if blocks are allowed at the top level though (for test_case).

[Ixrec]

The short answer is that parsers (as opposed to lexers) completely ignore whitespace, i.e. all the parser sees is something like [Keyword::While, Sigil::Exclamation, Expr(...)] so it has to pick what the Sigil::Exclamation is operating on based solely on what's before and after it.

But yeah, in practice just making the string part of the attribute seems like by far the best solution to this use case. Trying to emulate C/C++ macros exactly would wreak total havoc on Rust's grammar (as it already did on C/C++'s grammar).

[Manishearth]

@jgarvin

Why is while !expr ambiguous? Shouldn't exclamations have to occur right after the macro name to be part of the name?

Not in the way stuff is currently parsed/lexed, you can already do println !("foo") and while!true {} and while! true {}. Rust's lexer doesn't really care about spaces between things unless absolutely necessary.

[CAD97]

@jgarvin

Your 2 examples look identical

One used section!("a") and one used section!{"a"}. Yes, that matters for where you're allowed to use macros.

[bill_myers]

Isn't this trivially ambiguous?

while! (a + b) * c {bar();}

Currently this would be parsed as just while! (a + b), but the change would introduce an ambiguity.

Don't see a solution other than introducing new syntax like while!! expr {block} or accepting while! {cond {body}} or similar, unless we want to make parsing depend on the actual macro definition, in which case the definition can specify the flavor (but this seems a really bad idea).

[jgarvin]

@bill_myers see first comment by @CAD97 , this syntax actually used to exist, so it must have been handled somehow.

[CAD97]

To be clear, the "item like" syntax is

$:ident ! $:ident $:block
// e.g.
test_case! it_works { .. }

This could easily be extended to also accept

$:ident ! $:lit $:block
// e.g.
test_case! "it works" { .. }

but not to an expression for the reason @bill_myers states: it would be ambiguous with the existing "function like" syntax.

As a general guideline: try to write a macro_rules! matcher that matches the existing "function like" syntax in one arm and your proposed "item like" syntax in the other. If it works, then it's likely acceptable (from a grammar perspective). If it doesn't, it's likely problematic.