Unrestricted_attribute_tokens feature status

3

Tracking issue: https://github.com/rust-lang/rust/issues/55208

There are three important aspects with regards to tokens in attributes:

  • Who parses and interprets the attribute tokens?
  • What tooling is used for parsing attribute tokens?
  • Whether we care about this kind of attributes or not?

Let’s classify our attributes using these aspects:

  • Proc macro attributes. Tokens are interpreted by proc macro authors, using proc macro API and higher level libraries on top of it. These attributes are stable and we care about them.

  • Derive helper attributes. Tokens are interpreted by proc macro authors, using proc macro API and higher level libraries on top of it. These attributes are stable and we care about them.

  • Tool attributes. Tokens are interpreted by tools, using whatever, probably libsyntax/MetaItem APIs right now. These attributes are stable and we kinda care about them, but it’s mostly tool’s responsibility to parse these attributes correctly and define what that “correctly” means, and tools also have their own compatibility policies with regards to accepted attribute syntax.

  • Built-in attributes. Tokens are interpreted by the compiler, using libsyntax/MetaItem APIs. These attributes are stable and we care about them.

  • Legacy proc macro attributes. Tokens are interpreted by proc macro authors, using libsyntax/MetaItem APIs. These attributes are unstable, deprecated and we don’t care about them.

  • Legacy proc macro helper attributes. Tokens are interpreted by proc macro authors, using libsyntax/MetaItem APIs. These attributes are unstable, deprecated and we don’t care about them.

  • Custom attributes (feature(custom_attribute)), these are basically legacy proc macro helper attributes that are not whitelisted in any way. Tokens are interpreted by proc macro authors, using libsyntax/MetaItem APIs or perhaps proc macro API. These attributes are unstable, deprecated and we don’t care about them.

So, we see that the non-builtin non-macro attributes we care about are interpreted by macro authors using token streams and proc macro API.
This means that they are already prepared to deal with anything that can appear in macro attributes, and also that any compiler’s internal issues with libsyntax/MetaItem APIs won’t affect them.
Basically, for non-builtin attributes we can immediately extend the syntax to what proc macro attributes accept, namely:

PATH
PATH `(` TOKEN_STREAM `)`
PATH `[` TOKEN_STREAM `]`
PATH `{` TOKEN_STREAM `}`

with every inert macro-helper or tool-helper attribute defining its own little grammar restricted only by the macro or tool author’s imagination, similarly to macro attributes. (Legacy unstable stuff will be affected as well, but we don’t care about it.)

This leaves us with two issues - built-in attributes and key-value attributes #[PATH = SOMETHING].

Built-in attributes are in very sad state.

First, the MetaItem API is both outdated and buggy, and not prepared to deal with any tokens beyond meta-items in their “classic” Rust 1.0 flavor, this leads to issues like https://github.com/rust-lang/rust/issues/55168. Second, even if all footguns of the MetaItem API are avoided, most of built-in attributes are simply not validated in any way beyond checking their name!
So, accepting more nonsensical tokens in addition to already accepted nonsensical meta-items is not something we should strife for.

I suggest to just issue an error for built-in attributes with non-meta-item tokens instead of feature gate, none of these attributes should accept those tokens anyway.

Now, regarding #[PATH = SOMETHING], this is an open question.
I think we do need an RFC to discuss what we want from them.

What we strictly need for backward compatibility with stable code is 1) literals and 2) NtExpr for stuff like #[doc = $my_doc_str] in macros, I think that’s all, but we can check with crater.

We can certainly accept #[PATH = IDENT], perhaps we may want to extend the value syntax to arbitrary expressions #[PATH = EXPR], or arbitrary types #[PATH = TYPE], some people also mentioned macro invocatios #[PATH = my_macro!(tokens)].
On the other hand, all this is already available with delimited attributes - #[PATH(IDENT)], #[PATH(EXPR)], #[PATH(TYPE)], #[PATH(my_macro!(tokens))], so perhaps we don’t need to extend key-value attributes at all?
I honestly don’t know.

@CAD97
In the near future I’m going to submit a PR relaxing the rules for non-builtin attributes and restricting the rules for built-in and key-value attributes, as described above.
After that you’ll be able to proceed with an RFC deciding the fate for key-value attributes.

1 Like