Pre-RFC: Macro-expansion of input for proc-macros (derives primarily)

language design
1

There’s a bit of an uncomfortable situation going on with the expansion of derives (compiler plugin or proc_macro). #[cfg] attributes on sub-items are expanded:

#[derive(MyTrait)]
struct MyStruct {
    unconditional_field: Foo,
    #[cfg(feature = "some_feature")]
    cfg_specific_field: Bar,
}

If some_feature is not set, the derive for MyTrait will only see struct MyStruct { unconditional_field: Foo }.

However, we don’t currently expand any other attribute or macro invocations inside item definitions before expanding their derives, which makes #[cfg] special when it really shouldn’t be. We could just fix that so #[cfg] isn’t special at all, and just macro-expand derive input, but there’s a problem.

Many derive crates seem to be either depending on this behavior, or have simply accepted it in a forward-incompatible manner, as they expect macro invocations in their input: https://github.com/rust-lang/rust/pull/48465#issuecomment-377732109

However, as I stated in that comment, it seems that they’re not so much relying on this behavior, but have just come to expect it. In that Crater run I saw multiple breakages where a derive was expecting stringify!() in its input. I haven’t looked closer but I expect they’re manually stringifying the input when they really shouldn’t have to.

However because of this reliance/acceptance, this change would be breaking unless it was explicitly opt-in.

We’ve worked out three-ish possible solutions:

  • Allow #[proc_macro_derives] to opt-in to expanded input by declaring it up-front:
#[proc_macro_derive(MyTrait, expand_input = true)]
pub fn derive_my_trait(input: TokenStream) -> TokenStream {
    // the derive can assume no macro invocations in its input
}

This is the most well scoped solution, affecting only derives that ask for it. However, I think proc-macros and proc-macro-attributes might appreciate this behavior too. We could make them opt-in the same way as well, I guess, but that’s strictly less flexible than:

  • Expose a function in libproc_macro to macro-expand a TokenStream:
pub fn expand_macros(tokens: TokenStream) -> TokenStream { ... }

Then any proc-macro may simply ask for it programmatically, even limit expansion to a subset of its input or macro-expand some intermediate tokens:

#[proc_macro]
pub fn macro_expand_input(input: TokenStream) -> TokenStream {
    proc_macro::expand_macros(input)
}

I’m not entirely sure what the problems are with this approach, it seems like it could get a little complex WRT resolution and hygiene, like if something like this was passed to the proc-macro:

mod foo {
    macro_rules! bar {
        () => ()
    }

    bar();
}

but the proc-macro passes just bar() to expand_macros(). I suppose we’d mostly just say “well, would that macro invocation be legal anywhere else in the module in which this proc-macro was invoked?”

  • Add this behavior to the Rust 2018 epoch.

Unless I’m drastically misunderstanding their purpose, epochs were created exactly for this kind of breaking change. However, this does leave the utility for proc-macros and proc-macro-attributes unaddressed.

cc @jseyfried @petrochenkov

3 Likes
2

Wild guess: That's proc-macro-hack.

I should also point out this RFC:

3

Some precedent here is that built-in derives are executed before custom derives.

One question about approach 2 is how context would be handled? Proc macros don’t have any context about the crate/module they’re being expanded in, but this would be needed to perform expansion. So this API would have to get the context from some TLS or whatever. And if that’s considered acceptable, why not let the proc macros get some more of that context themselves? :wink:

Are you talking about expanding macros anywhere in custom derive input, or just in attributes? Isn’t this also tied in with the debates about eager expansion (this is morally equivalent to eager expansion as far as I can tell) and macros in attributes? I see there was a thread about the latter, but it somehow got turned into the above linked RFC for an eager expansion API for proc macros, which doesn’t solve the original issue (e.g. #[path=concat!(...)], #[doc=stringify!(...)]) at all.

4

How horrifying.
I missed the whole macros 1.1 story, but the more I learn about it the more I'm convinced that macros 2.0 should not be stabilized until they are actually soundly implemented.

5

I referenced the macros-in-attributes case here, which looks at it from a “the author of the attribute proc macro should write it to handle macro calls” perspective, but that approach could also work here. Consider:

#[my_attr_macro]
{
    #[some_other_attr_macro(concat!("hello", " ", "world"))]
    let x = 0;
}

If we don’t expand some_other_attr_macro before passing the input to my_attr_macro, and some_other_attr_macro doesn’t do its own macro expansion of concat!, then my_attr_macro can do that expansion ‘for’ some_other_attr_macro before it expands. This helps deal with the macro ecosystem fracturing into two worlds of “handles unexpanded macro calls” and “doesn’t handle them”.

@durka do you have any tips for improving the macros-in-attributes section of the RFC?

Edit: ah, I see you asked a question here. Yes, the idea is to migrate the compiler builtins to become more of a “standard library of normal attr macros” rather than “actual magic” - I should update the RFC to make that clear.

6

I agree. The issue with approaches 1 and 2 (but especially 2) is inconsistent behavior between different macros, which leads to Principle of Least Surprise violations for macro users.

I see. That makes sense, but I still don't really see your RFC as a resolution to the original issue. "RFC: change #[path]/#[doc]/etc to use this new function that expands macros" (which would be the next RFC after yours is accepted) seems pretty much the same as "RFC: change #[path]/#[doc] to expand macros".

7

That's a fair assessment, in the sense that if the RFC is accepted that doesn't imply the attr macro issue would be resolved.

On another note, I just remembered an issue with the "macro assistance" example I gave above: the attribute system explicitly avoids mandating a syntax for the tokens in an attribute argument, which means you can't safely pre-expand an attributes arguments 'for' it. As an extreme example, if foo is an attr macro which allows and expects full expression syntax, consider:

#[my_attr_macro]
{
    #[foo({let s = concat!("a", "b"); println!(s)})]
    let x = 0;
}

Expanding the argument of foo here would probably be incorrect, especially not knowing what foo does. I guess we're forced to respect argument macro calls as a do-or-die boundary: it really is up to the author to handle them or not.

However, if attr macro expansion order is well-defined it's not hard to imagine an expand macro which at least handles expansions in the attr body. For example:

#[expand]
#[some_attr_macro]
x = concat!("a", "b");

Here, some_attr_macro should just see x = "ab".

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.