Pre-RFC: Allow Custom Syntax Parsing for Attribute Macro Inputs

language design
1

​​Problem:​​ Currently, attribute macros require their input to be valid Rust syntax before macro expansion. This causes false errors in tooling (rustc, rust-analyzer) when the macro expects a custom DSL. For example:

#[proc_macro_attribute]
pub fn dummy_attr(_attr: TokenStream, input: TokenStream) -> TokenStream {
   //do something
}

fn main() {
    #[dummy_attr]
    {key:value} // now this give error
    #[dummy_attr]
    {key::value} //but this is allowed
}

Tools like rust-analyzer flag {key:value} as invalid Rust syntax during analysis, disrupting developer workflows even though the macro would accept it. Function-like macros don’t have this limitation because they participate in token-tree expansion.

​​Proposal:​​ Enable attribute macros to declare a custom grammar for their input body. This would instruct tooling to use:

  1. Rust syntax by default (backward-compatible behavior)
  2. A macro-specified grammar when explicitly declared

​​Suggested Implementation:​​ Add a new attribute to mark macros with custom syntax:

#[proc_macro_attribute]
#[custom_grammar = "grammar_identifier"] // New annotation
pub fn my_macro(_attr: TokenStream, input: TokenStream) -> TokenStream { /* ... */ }

Where grammar_identifier could resolve to:

  • A ​​built-in grammar​​ (e.g., json, toml, ron) supported natively by tooling
  • A ​​user-defined grammar​​ via:
    • ​​Syn-powered parser​​: Share logic between the macro and tooling
    • ​​TextMate grammar​​: Leverage existing editor syntax definitions
    • ​​WASM-compiled parser​​: Portable syntax engine

​​Workflow Integration:​​

  1. ​​Compiler​​: Ignore custom syntax validation in early passes (treat input as token stream)
  2. ​​rust-analyzer/LSP​​:
  • Use custom parsers for syntax highlighting/validation
  • Provide diagnostics based on macro-specific rules
  • Use the macro’s grammar for auto-completion in attribute bodies

​​Motivation:​​

  • Eliminate erroneous errors in IDEs for valid macro input
  • Enable rich tooling support for domain-specific syntax
  • Unify compiler/tool behavior without changing macro expansion rules
  • Backward-compatible via opt-in syntax declaration

​​Challenges:​​

  1. Standardizing Grammar Definitions: Need consensus on cross-tool grammar format
  2. Parser Sandboxing: Security implications for user-provided parsers
  3. Performance: On-demand grammar loading to avoid tooling slowdown

​​Alternatives Considered:​​

  1. Do nothing: Forces macros to accept Rust syntax or tolerate tooling noise
  2. IDE-only solutions: Fragmented support without compiler awareness
  3. Full syntax plugins: More complex and conflicts with stable Rust

​​Open Questions:​​

  • Should grammars be definable inline in proc macros?
  • How to handle versioning for user-defined grammars?
  • Preferred grammar format priority (Syn, TextMate, WASM, etc.)?

​​Why this approach?​​

  1. ​​Clear problem statement​​ with concrete code example
  2. ​​Backward compatible​​ via opt-in annotation
  3. ​​Tooling-focused​​ while preserving current compiler behavior
  4. ​​Practical deployment path​​ using existing ecosystem (Syn/TextMate)
  5. Exposes ​​tradeoffs and alternatives​​ transparently

This structure aligns with RFC conventions by separating motivation, design, and open questions. The grammar format flexibility increases adoption potential without mandating a single solution.

2

We have to parse source files before we are able to macro-expand them, or even know which proc macro a macro invocation refers to. Having parsing itself depend on macro information is incompatible with that. For custom DSLs you should use bang macros (my_macro!()) rather than attribute macros as for those there is a much more flexible (but still independent of the macro it will resolve to) grammar accepting everything that tokenizes as rust code and has balanced parens.

11 Likes
3

I've been thinking about a related problem, of giving correct syntax highlighting and IDE support to macros in general. If a library like leptos or dioxus could provide some custom grammar files and IDEs supports some of these, it might improve usability of macros and allow library-specific IDE plugins. But requiring this feature at the language level feels wrong, because new grammars shouldn't break public interfaces if rust ever chose to use a new grammar language for everything, else rustc would forever have to support the grammars we chose to stabilize today

4

I can't think of a single actual use case where you would want to have non-Rust syntax specifically in an attribute macro. IMHO, attributes are for Rust code.

5

I once ran into a situation where I wanted to write a proc-macro that would automatically call some functions in its impl based on the enum definition, or something like that.

I no longer remember what exactly I wanted to do, but it was perhaps something like this:

#[my_attribute]
enum Message {
    OnAction1 = handlers::on_action1,
    OnAction2 = handlers::on_action2,
}

I only remember the compiler telling me that the proc_macro needed to have valid Rust syntax as an input and me thinking it was a bummer. So there are use-cases.

6

Perhaps this will be "fixed" by Declarative `macro_rules!` attribute macros by joshtriplett · Pull Request #3697 · rust-lang/rfcs · GitHub

As far as I remember, declarative macros had no such syntax restrictions.

7

You can represent that today with

#[my_attribute]
enum Message {
    #[my_attr_action = handlers::on_action1]
    OnAction1,
    #[my_attr_action = handlers::on_action2]
    OnAction2,
}
2 Likes
8

No, that RFC is about something else, not about lifting syntax restrictions.

1 Like
9

As an alternative, one could just allow anything in blocks. It should be comparatively simple to just find the matching brace to continue parsing from. This would only allow custom grammar inside of braces. To expand that further to outside of blocks, it might be possible to end the custom grammar area upon a semicolon (or block). This would at least not necessitate allowing actual custom grammar implementations in the compiler.

Edit: This might actually be relatively simple, since rustc is capable of continuing past grammar mistakes (to check for other errors). Just braces, parens, brackets and quotes have to match.

1 Like