Pre-RFC: Allow Custom Syntax Parsing for Attribute Macro Inputs

language design
1

​​Problem:​​ Currently, attribute macros require their input to be valid Rust syntax before macro expansion. This causes false errors in tooling (rustc, rust-analyzer) when the macro expects a custom DSL. For example:

#[proc_macro_attribute]
pub fn dummy_attr(_attr: TokenStream, input: TokenStream) -> TokenStream {
   //do something
}

fn main() {
    #[dummy_attr]
    {key:value} // now this give error
    #[dummy_attr]
    {key::value} //but this is allowed
}

Tools like rust-analyzer flag {key:value} as invalid Rust syntax during analysis, disrupting developer workflows even though the macro would accept it. Function-like macros don’t have this limitation because they participate in token-tree expansion.

​​Proposal:​​ Enable attribute macros to declare a custom grammar for their input body. This would instruct tooling to use:

  1. Rust syntax by default (backward-compatible behavior)
  2. A macro-specified grammar when explicitly declared

​​Suggested Implementation:​​ Add a new attribute to mark macros with custom syntax:

#[proc_macro_attribute]
#[custom_grammar = "grammar_identifier"] // New annotation
pub fn my_macro(_attr: TokenStream, input: TokenStream) -> TokenStream { /* ... */ }

Where grammar_identifier could resolve to:

  • A ​​built-in grammar​​ (e.g., json, toml, ron) supported natively by tooling
  • A ​​user-defined grammar​​ via:
    • ​​Syn-powered parser​​: Share logic between the macro and tooling
    • ​​TextMate grammar​​: Leverage existing editor syntax definitions
    • ​​WASM-compiled parser​​: Portable syntax engine

​​Workflow Integration:​​

  1. ​​Compiler​​: Ignore custom syntax validation in early passes (treat input as token stream)
  2. ​​rust-analyzer/LSP​​:
  • Use custom parsers for syntax highlighting/validation
  • Provide diagnostics based on macro-specific rules
  • Use the macro’s grammar for auto-completion in attribute bodies

​​Motivation:​​

  • Eliminate erroneous errors in IDEs for valid macro input
  • Enable rich tooling support for domain-specific syntax
  • Unify compiler/tool behavior without changing macro expansion rules
  • Backward-compatible via opt-in syntax declaration

​​Challenges:​​

  1. Standardizing Grammar Definitions: Need consensus on cross-tool grammar format
  2. Parser Sandboxing: Security implications for user-provided parsers
  3. Performance: On-demand grammar loading to avoid tooling slowdown

​​Alternatives Considered:​​

  1. Do nothing: Forces macros to accept Rust syntax or tolerate tooling noise
  2. IDE-only solutions: Fragmented support without compiler awareness
  3. Full syntax plugins: More complex and conflicts with stable Rust

​​Open Questions:​​

  • Should grammars be definable inline in proc macros?
  • How to handle versioning for user-defined grammars?
  • Preferred grammar format priority (Syn, TextMate, WASM, etc.)?

​​Why this approach?​​

  1. ​​Clear problem statement​​ with concrete code example
  2. ​​Backward compatible​​ via opt-in annotation
  3. ​​Tooling-focused​​ while preserving current compiler behavior
  4. ​​Practical deployment path​​ using existing ecosystem (Syn/TextMate)
  5. Exposes ​​tradeoffs and alternatives​​ transparently

This structure aligns with RFC conventions by separating motivation, design, and open questions. The grammar format flexibility increases adoption potential without mandating a single solution.

2

We have to parse source files before we are able to macro-expand them, or even know which proc macro a macro invocation refers to. Having parsing itself depend on macro information is incompatible with that. For custom DSLs you should use bang macros (my_macro!()) rather than attribute macros as for those there is a much more flexible (but still independent of the macro it will resolve to) grammar accepting everything that tokenizes as rust code and has balanced parens.

11 Likes
3

I've been thinking about a related problem, of giving correct syntax highlighting and IDE support to macros in general. If a library like leptos or dioxus could provide some custom grammar files and IDEs supports some of these, it might improve usability of macros and allow library-specific IDE plugins. But requiring this feature at the language level feels wrong, because new grammars shouldn't break public interfaces if rust ever chose to use a new grammar language for everything, else rustc would forever have to support the grammars we chose to stabilize today

4

I can't think of a single actual use case where you would want to have non-Rust syntax specifically in an attribute macro. IMHO, attributes are for Rust code.

5

I once ran into a situation where I wanted to write a proc-macro that would automatically call some functions in its impl based on the enum definition, or something like that.

I no longer remember what exactly I wanted to do, but it was perhaps something like this:

#[my_attribute]
enum Message {
    OnAction1 = handlers::on_action1,
    OnAction2 = handlers::on_action2,
}

I only remember the compiler telling me that the proc_macro needed to have valid Rust syntax as an input and me thinking it was a bummer. So there are use-cases.

6

Perhaps this will be "fixed" by Declarative `macro_rules!` attribute macros by joshtriplett · Pull Request #3697 · rust-lang/rfcs · GitHub

As far as I remember, declarative macros had no such syntax restrictions.

7

You can represent that today with

#[my_attribute]
enum Message {
    #[my_attr_action = handlers::on_action1]
    OnAction1,
    #[my_attr_action = handlers::on_action2]
    OnAction2,
}
3 Likes
8

No, that RFC is about something else, not about lifting syntax restrictions.

1 Like
9

As an alternative, one could just allow anything in blocks. It should be comparatively simple to just find the matching brace to continue parsing from. This would only allow custom grammar inside of braces. To expand that further to outside of blocks, it might be possible to end the custom grammar area upon a semicolon (or block). This would at least not necessitate allowing actual custom grammar implementations in the compiler.

Edit: This might actually be relatively simple, since rustc is capable of continuing past grammar mistakes (to check for other errors). Just braces, parens, brackets and quotes have to match.

1 Like
10

Dear Rust Developers and users.

While experiencing a macro limitation inside Rust, targeting and customized and alternated language subset for domain-specific usage, I've thankfully encountered this particular page, containing a feature discussion which really would help in my case. Also, I'm feeling relieved to reach out here in time, long before the discussion function is going to be removed (due to topic closing, a year after the last post is submitted).

A short prehistory: after gaining inspiration from Flutter and Jetpack Compose, I decided to create a similar GUI approach inside Rust (I have to admit Rust became the primary and only choice for a compiled low-level language half a year ago. Thanks for providing an alternative better than C++... But that's information rather to discuss inside another topic, hehe). I also won't tell more about the exact framework I'm designing; you just should know the so-called "Compose Syntax" slightly differs from Rust's, underlining the specific domain it's addressed to.

Additionally, I understood Egui's and ICED's notation and construction methods look too unlikely to seem attractive (I'm expressing it mildly, because those remain good frameworks for graphical applications anyways). I'm also not aiming to build an alternative to Slint (with it being more a hybrid out of Flutter and QT, carrying its own runtime and markup language).

My request finally results in a fundamental change inside the Rust Compiler, particularly the macro preprocessor, mainly containing the methods showed up here inside this discussion. Allowing custom grammar definitions would take away the problem of coding lexer, parser, updater, LSP server and different IDE extensions for highlighting errors/suggestions at design time. (Also imagine taking permanent care of these different systems, which could be more challenging than helping out). I would like to remind: my aimed language implementation JUST SLIGHTLY differs from Rust, therefore a much wiser strategy would be to adapt the evergreen built-in parser system inside Rust's own tool chain, maintained and extended by the wonderful language's development team / foundation.

Dear Developers and Users. Let's create such a grammar extension, in order to achieve similar goals to those other programming languages solved as well:

  • SwiftUI required a specific compiler modification for native parsing of the alternated syntax, recommendes for GUI design. Apple implemented it.
  • Jetpack Compose was requested by Google as a primary feature inside the Language primarily chosen for Android development: Kotlin. The Compiler was modified to fit.
  • Rust has the best macro implementation of any language so far, made for unlimited compile-time task management. I really adore such a feature, being missing or just misimplemented (like in C++) quite often. Also, it's proven not only to be limited for a single use-case like "Compose", but also capable of customized usage, exampled by many yet existing cargo crates.

I don't believe the feature of "custom language grammar implementation inside macros" would badly influence the Compiler development, especially if the Rust's built-in parser/lexer/grammar is being used anyway (this time, in an extended way).

Remember: this is the chance to DO IT RIGHT.

I would thank all readers for insighting till the end of this message; I appreciate their (/your) patience. Also, I really hope this plea is going to be taken seriously, as it underlines the need of the discussed feature. I would love to assist realizing it, as far as I can or have power to.

Sincerely.

11

We do have custom grammar inside macros, just not in attribute macros. That's because attribute macros are meant to act on Rust syntax, unlike function macros which operate on their own syntax.

What you're describing sounds like a problem completely separate from that of the OP.

1 Like