[Pre-RFC] Macros in Attributes


#1

Summary

Allow expression macros in attributes. For example:

#[path = concat!(env!("OUT_DIR"), "/hello.rs")]
mod foo;

Motivation

Users sometimes attempt to use macros inside of attributes such as #[path], #[export_name], and #[link_section]. These are commonly used in code generation where being able to programmatically control their values is useful. Additionally, #[doc = include_str!("file")] would allow large explanatory texts to be moved outside of the current source file.

Currently, expression macros inside attributes cause parsing errors when included directly (rust-lang/rust#18849), and attempts to get around this restriction using macro indirection causes ICEs (rust-lang/rust#42164).

This RFC proposes to allow expression macros inside attributes to allow these use cases.

Guide-level explanation

Built-In Attributes

The previous use cases are probably mostly encountered by experienced Rust developers, who might be more surprised that macros don’t work here.

One consideration is making sure error messages are informative, for instance we should expect similar errors to when existing macros receive unexpected tokens. For example, this:

fn main() {
    let s = "hello!";
    println!(s);
}

Gives the following error:

error: expected a literal
 --> bin.rs
  |
3 |     println!(s, 5);
  |              ^

Which inspires the following:

#[path = include!("file")]
mod foo;

Should error with:

error: expected a literal after macro expansion
 --> bin.rs
  |
1 | #[path = include!("file")]
  |          ^^^^^^^^^^^^^^^^

Procedural Macros and Custom Attributes

Procedural macros can specify custom attributes which are passed to the macro before macro expansion. We should update the proc macro reference with guidelines on expanding macros in their input.

Reference-level explanation

A relatively small change is sufficient to address the #[export_name] and #[link_section] cases outlined above. Specifically:

  • Allow key-value attributes to have a macro as their value node, updating the parser and AST accordingly.
  • After macro expansion, confirm that the macro nodes are expanded into literal expressions.

Unfortunately, #[path] and #[cfg] are interpreted too early by the parser for this to work (see unresolved questions below).

Drawbacks

  • Expanding macros into one part of attributes (key-value values) but not others (attribute lists, attribute identifiers themselves) may be confusing or surprising.
  • Cargo currently ignores environment variables when determining when to recompile parts of a project. This may make env! behave surprisingly when used as above. We can mitigate this by either documenting the issue in the reference page for attributes, or updating Cargo to take a snapshot of environment variables during builds.

Alternatives

For some use cases there are existing solutions in the crate ecosystem using proc macros (see here), however this is rather heavyweight.

There is always the option of using build.rs to generate files with the relevant contents, but this seems like a code smell.

Unresolved questions

  • Do we extend the legal positions of expression macros to include list items (e.g. #[name(macro!(...))])?
  • Are ident macros stable enough to be supported, or is restricting to expression macros sufficient?
  • The implementation of #[path = <value>] tries to use <value> as a string literal long before macro expansion takes place. Can we delay parsing paths until after that? This seems hard to do, since #[path] is used to find files to parse.
  • Similarly, the feature-gated #[cfg(name = <value>)] syntax also tries to use <value> before macro expansion. How do we handle this?

Make it easier to include generated files as module?
#2

For modules this may work:

mod foo {
   include!(concat!(env!("OUT_DIR"), "/hello.rs"))
}

If you’re using code-generation, could you make names configurable as part of your customo code generation process?

Macros can match attributes (and rewrite them too, I think), so something like this may be possible:

prefix_export! { name, {
   #[export_name="foo"] …
}}

#3

I think this is a great idea. Since anything to do with modules is controversial these days, something like #[export_name = concat!("foo", $bar)] might be a better motivating example.

@kornel: Refuting specific examples of a general facility is always hazardous, but anyway both of the solutions you point out are not complete: include! only works for a single statement, and attempting to construct #[export_name] via a macro ICEs as already mentioned.


#4

Thankfully, I don’t think the module system changes are going to sweep in and invalidate the above motivating #[path] example for at least a few months, although you’re right that the timing isn’t great.

It’s also the only use I could find which doesn’t involve defining a macro (which would be a bit of a distraction), although I guess it’s not totally unreasonable to have #[export_name = env!("CONFIG_VAR")].


Pre-RFC: Macro-expansion of input for proc-macros (derives primarily)
#5

I’ve always wanted to be able to do:

#![doc=include!("README.md")]

#6

In general attributes are themselves macros which get passed their parameters pre-expansion. That means that it is up to the attribute what it does with any macros it finds, and if it chooses to expand them, in what context. The libraries for procedural macros should make this easy and I hope that most attributes do this (and we should probably have guidelines for macros which recommend doing it). That leaves the built-in attributes. It is probably worth examining these on a case by case basis, and where the arguments are treated like an expression, expand macros there.


#7

@nrc I’m probably showing off my unfamiliarity with this area, but what do you mean by ‘general’ versus ‘built-in’ attributes? I don’t see any reference to them in the reference - are you referring to proc macro attributes?

As far as I can tell, concat! and friends are implemented as a syntax extension in libsyntax_ext, run after parsing but long before some of the built-in attributes get used: it seems #[link_section] and #[export_name] are evaluated in librustc_trans, but #[path] gets interpreted in libsyntax itself. Is there a way to bring the concat! evaluation ‘forward’? I can’t quite get my head around what libsyntax::ext::expand is doing…

Looking in libsyntax::ast, the "value" in #[name = "value"] gets parsed into a MetaItemKind::NameValue(Lit). If we let NameValue hold either a Lit or Mac (macro call) and update fold, is that enough for proc macro attributes, #[link_section], and #[export_name]?


#8

Beware that at least currently Cargo is unaware of user env vars. If you change the var, the relevant parts of the project will not be recompiled, and you’ll get a mixed result.


#9

@fitzgen @kornel Motivation and discussion updated, thanks!

I’m looking at how #[cfg] and #[path] get interpreted, and it’s looking less and less easy to allow macro expansion inside them (or at least, expansion with macro $variables), since they can (for instance) change the available macro definitions. Anyone have any ideas?


#10

For anyone still interested, after discussion with @nrc this turned into an (in-review) RFC for a macro expansion API, which you can find here. I’d love to hear any thoughts on what’s missing or how to improve what’s there before I make the PR!

Edit: RFC now live.