Question: Does rust as a language *require* derive macros be kept in order?

I have a situation (see Sort derives · Issue #4112 · rust-lang/rustfmt · GitHub) where I want rustfmt to sort derive macros into an order other than the order that they are specified in within the code. Right now, derive macros are required to be in a particular order because each macro only sees the output of macros that follow it (see enhancement request - derive sorting · Issue #1867 · rust-lang/rustfmt · GitHub). But when I read through the derive-macro docs I couldn't find anywhere where it specifically stated that rust as a language required this to be true. That is, it feels like the requirement that derive macros be in a particular order is an accident and not something that was intentionally planned.

So, with that background out of the way -

  • Is there a place in the documentation that explicitly states that it is a requirement of rust as a language that the derive macros be order sensitive? If so, can someone please link to where I can find it?
  • If there is no such requirement, and this behavior is an accident of the design of the compiler, can we choose to lift this restriction and not have it be considered a breaking change?1
  • What would be involved within the compiler to support this change? That is, how would we specify that derive macros can be put in any order and their output needs to be semantically the same regardless of the order the derive macros were declared?

1I expect that doing this would require lots of testing, crater runs, etc. before everyone is comfortable with supporting the change, especially if it isn't on an edition boundary. That's not what I'm concerned with here, I'm concerned with the language definition itself.

4 Likes

It's not a written requirement somewhere, but it's observable behavior and in principle changing it could break stable code. We've avoided changing it in the past precisely because it could break stable code.

3 Likes

Is proc-macro-attribute application order defined? · Issue #692 · rust-lang/reference · GitHub seems related

How can derives observe other derives? They only get the type input afaik, the impls from the other derives are not given to them.

1 Like

For example a macro could write to a file that the other one reads or maybe make some http requests.


To be honest this seems like a hack that abuses implementation details of the rust compiler and I don't see any legitimate use for having derive macros that depend on the order they are derived in.

I can not really imagine that anyone would abuse the ordering of derives in real code.

4 Likes

IIUC, what it is is that

#[derive(A, B)]
struct S;

is equivalent to

#[derive(A)]
#[derive(B)]
struct S;

so A gets (the equivalent of)

#[derive(B)]
struct S;

and B gets

struct S;

due to the normal attribute expansion rules.

3 Likes

For separate #[derive] attributes that's true, for composite ones it's not, a little testing gave:

#[derive(bar::Foo, bar::Bar)]
struct Baz;

#[derive(bar::Foo)]
#[derive(bar::Bar)]
struct Quux;
Foo sees: struct Baz ;
Bar sees: struct Baz ;
Foo sees: #[derive(bar :: Bar)] struct Quux ;
Bar sees: struct Quux ;

(EDIT: And, I'm not sure how much of this Rust actually guarantees anyway, it feels like something that should not be guaranteed to keep working even if it does work currently).

8 Likes

How hard would it be to tweak the compiler to randomly reorder the derive attributes, and then do a crater run? I'd love to see how much code depends on the current behavior that's out in the wild.

That does not bring me joy. The following also doesn't bring me joy:

#[derive(Debug, Debug)]
struct Foo;

Compilation output from the playground:

  |
1 | #[derive(Debug, Debug)]
  |          -----  ^^^^^ conflicting implementation for `Foo`
  |          |
  |          first implementation here
  |
  = note: this error originates in the derive macro `Debug` (in Nightly builds, run with -Z macro-backtrace for more info)

But at least the compiler lets me know what's going on so I can fix it. BUT is there a legitimate reason to try to derive something multiple times in a row? I know that it's possible to write a macro that will generate code that will compile, but it feels like a mistake that should be discouraged.

My take on this issue

Given that everyone that's answered on this topic so far is (AFAIK) an expert in the language, the fact that there is ambiguity of interpretation makes me... concerned. So, can we resolve the ambiguity in a sane manner?

My vote is for the following rules:

  1. Attributes in a composite derive are treated as an unordered set. The compiler is free to reorder the attributes as it sees fit, and is required to turn the attributes into a set. Attributes within a composite set are not permitted to see the output of attributes in the same composite derive. All permutations must result in the same output.
  2. Separate derives are effectively within nested scopes. Outer scopes can see the results of execution of inner scopes, but not the other way around.
  3. The innermost scope is 'executed' first, replacing the scope with the execution results. This continues until all derives are executed.

So, given that, we can look at the following example:

#[derive(Clone, Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]
#[derive(Quux)]
struct Foo;

is semantically similar to the following (if braces could be used this way)

#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)] // Scope 2
{
    #[derive(Quux)] // Scope 1
    {
        struct Foo;
    }
}
  • By rule 2, each derive line defines a separate scope.
  • By rule 3, scope 1 will be executed first, producing the output of Quux macro.
  • Once the Quux macro is done, its output becomes available to the derive macros in scope 2.
  • By rule 1, the duplicate Clone derives are reduced to a single instance. Also by rule 1, we can reorder the derive macros in whatever manner is convenient to the compiler. Finally, note that since the macros within a composite derive can't see each other's output, it is entirely possible to spawn futures for each derive macro, and construct their output concurrently. I'm not suggesting that is necessary or a good idea, just that the rules permit it.

The one issue that I see with this is how deriving the same macro multiple times in a row should be handled. E.g.:

#[derive(Quux)]
#[derive(Quux)]
#[derive(Quux)]
#[derive(Quux)]
struct Foo;

My vote is to not try to do anything clever, and just expand the macros using the rules above. Once the macros are all expanded, if there are any conflicts that the compiler will emit an error as normal.

Given all of that, are the rules unambiguous? Are there any real-world issues that anyone can think of by implementing these rules? More importantly does anyone disagree with the rules, or should I start writing an RFC along these lines?

FWIW, since this is a change to the current behavior, even though that behavior appears to be an accidental quirk of the compiler as it currently is, my personal feeling is that settling these rules will require waiting for the 2024 edition before they could be stabilized (assuming that everyone here is interested in doing so, the RFC is accepted, etc., etc., etc.).

1 Like

The current behavior is certainly not accidental, and is not a quirk.
IIRC, some popular crate (proc-macro-hack?) generating macro_rules items from derives depended on the current behavior.

I see that two issues are mixed in this thread - the expansion order of derives in a single list (which is indeed unspecified), and the order of output items (which is well defined due to macro_rules).
I'll try to look through the thread and answer in more detail today.

3 Likes

This is exactly backwards from the way that all other attribute macros work, which is outside-in. This is guaranteed by the language, and in fact a necessary property.

(E.g. #[cfg(FALSE)] #[derive(Invalid) struct Foo; compiles fine, because #[cfg(FALSE)] is (effectively) an attribute which takes its input and discards it, passing along an empty token stream for further processing.)

The outer attributes should see any inner attributes. This is consistent with the existing and plain outside-in processing order.

What's not immediately clear is the interaction of multiple derives within a single #[derive()] container. We've shown that derive macros within a container don't see each other, so if that were the only potential way to observe ordering, it'd be valid to reörder them. However, derive order is observable one other way: macro_rules! macros using legacy scoping rules (i.e. all of them, currently). This means that reördering derives is potentially behavior changing. (However, our research here also indicates that rustfmt's default setting of merging derives is also behavior changing.)

It's also worth noting, @ckaran, that #[derive] macros are special in that they're "inert;" they do not (and can not) impact the definition of the type they decorate. Instead, they emit some extra items which are inserted after the decorated item.

This is exactly the output I would expect. Deriving a trait twice is no different to implementing it twice.

@CAD97 basically already did this.

Multiple derive attributes (#[derive(A)] #[derive(B)] struct S;) follow the same rules as multiple of any macro attributes.
The first macro attribute, derive or not, is expanded first and takes the rest of the item as input tokens.
So if rustfmt is merging multiple #[derive]s into one, or splitting one derive into multiple, it's indeed behavior changing in theory, but not much in practice.
(With one exception - it's better to not split Copy from other built-in derives, it will lead to pessimization, or even build failures in case of #[packed] structs, because Copy modifies behavior of other derive macros in the same #[derive] container).

Multiple derive macros in a single derive attribute (#[derive(A, B, C)]) are expanded in unspecified order (as soon as their resolutions become ready in practice), so the tricks with writing a file in one derive and reading it in another is not guaranteed to work.
What is guaranteed is that the code generated by A will be followed by code generated by B and then code generated by C.
If you change that, you'll have issues like Macro-generated macro can no longer be called · Issue #63651 · rust-lang/rust · GitHub (apparently extracted from a version of futures-rs relevant at that time).
I'm pretty sure that it's not a common issue in practice, and rustfmt can implement the reordering at least under an option, if some #[rustfmt::skip]-like opt-out is provided.

2 Likes

Yup, you're right, my mistake, thank you for catching it! Does this also guarantee that outer scope derive macros can't see the output of inner scope derive macros? Otherwise the whole system needs some kind of 'up-and-down' behavior, and possibly be executed multiple times before the macros are fully expanded (if they are fully expanded at all).

So, does this imply that Copy should be the first attribute in the list? Is that documented anywhere? Are there any other order dependent derive macros that we need to be concerned with in core/std?

OK, but this part is invisible to the end user, correct?

Does this also guarantee that outer scope derive macros can't see the output of inner scope derive macros?

Yes, the outermost macro attribute sees everything below it as unexpanded tokens, so it cannot see the outputs, that would be eager expansion.

So, does this imply that Copy should be the first attribute in the list?

Copy should either inside the same #[derive(...)] with the other built-in derive macro (e.g. Clone) or in one of the #[derive(...)]s immediately above it.

#[derive(Clone, Copy)] // OK
#[derive(Debug)] // OK too
#[derive(PartialEq)] // Still OK
#[repr(packed)]
struct S { ... }

See fn resolve_derives in the compiler for the exact implementation.

This property of Copy was originally introduced to optimize behavior of other derives like Clone, then reused again to keep supporting built-in derives on #[repr(packed)] structures, and now we need to continue support it for backward compatibility and to not lose performance.

Is that documented anywhere?

Maybe, but likely not.

OK, but this part is invisible to the end user, correct?

It's visible if the derive macro generates macro_rules items, like in the issue linked above.

OK, but what about:

#[derive(Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Clone, Copy)]
#[repr(packed)]
struct S { ... }

versus

#[derive(Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Clone)]
#[repr(packed)]
struct S { ... }

Are there performance implications here?

Thank you, I will.

This is probably a problem. I use LaTeX which has a real problem with packages being required to be imported in a particular order in order to work correctly. The issue is that the documentation for different packages can be unclear, and may lead to cycles in the dependency graph (if you can call it that). Learning this sharp edge of LaTeX takes a fair amount of effort, and is in general a waste of time and brain cells. The fact that rust's macro system may be heading down that path is extremely concerning to me, and is one of the reasons why I wanted to change the behavior using the rules that I originally described in my earlier post (modulo the fixes that @CAD97 and @petrochenkov described). I acknowledge that trying to do so would be a breaking change, and therefore it could only be done on an edition boundary, if it is done at all. However, I think it's something that should be considered before we end up having weird dependencies in the macro system that aren't obvious to users. I mean, the fact that the ordering of Copy could make a difference in packed structures is something I would never have thought about.

2 Likes

I believe the macro_rules! cases from the issue are all from #[derive] attributes generated from other related macros, so those macros are in charge of ordering the derives in the correct order not the user. For general purpose user-written derives there should never be any ordering requirement between them.

No, position of Copy inside a single list is not important.

OK, to summarize what I understand at the moment about how derive macros currently work:

  1. Macros are order-dependent. Macros that are earlier in a list are aware of macros that are later in a list.
  2. Macros are expanded from outer scopes to inner scopes. Thus, an outer-scope derive macro will only see a token stream of the inner contents, not the results of expansion of the inner contents.
  3. #[derive(A, B)] may yield different results than #[derive(A)] #[derive(B)].
  4. Copy is special-cased and needs to be in the same list as the rest of the built-in derive macros to work correctly, but it doesn't matter where it is in the list provided that it is in the outer-most scope.

Have I missed anything here? Are there any other special cased derive macros?

Looking the list over, the first two points seem to have relatively low cognitive complexity. The last two points feel like they can easily lead to overwhelming cognitive complexity. Is it possible to fix those points?

Given the proc macro APIs are stable, not for the currently implemented proc macro APIs. But then again, proc macros can already make arbitrary syntactical differences semantic, so I don't see that as that much different from proc macros just being poorly written and reliant on syntax specifics they shouldn't be.

Personally, I'd like to have proc macros receive the derive attribute that their derive is in (like how attribute proc macros receive their attribute), but starting with their derive name, keeping the look-forward-only property. (Yes, this would make it more ordering sensitive.) A really nice (imho) effect of this is that a derive macro now can know the path to the proc macro (thus likely, via reëxports, the path to the trait it's implementing) without having to hardcode crate names. (This unfortunately doesn't help getting to helper modules, unless we add the (very interesting) feature of $ty::super paths.)

(Please give me some way to know the path to the runtime crate from my attribute proc macros, so they don't break when the runtime crate is renamed :pleading_face: (NB: workaround for function-like macros: take a #![crate($path)] at the beginning, then expose it in the impl crate via a wrapper macro_rules! which adds #![crate($crate)].))

With a bit more special casing, yes: in addition to Copy looking for other built in derives and changing their semantics, have the other built in derives look for Copy and update their semantics. Then, using the extra built-in power of looking within the same derive container, it doesn't matter which is first; the one that comes first will see the latter one, and adjust semantics as required.

Doing so in the type system rather than compiler built-in special powers would require a very big new metaprogramming feature to have #[cfg(T: Copy)] and I think such metaprogramming would easily become unsolvable at the item level. (And as block statements can contain items...)

2024 Edition! (I hope...)

Is there a current collection point for enumerating all of the 'broken' things with the various bits of the current macro system? Or a working group? I'd like to see this part of the language really cleaned up and ironed out before we end up with C++'s template metaprogramming craziness (anyone that has dealt with an error because of Boost knows what I'm talking about, you make one tiny mistake, and have 10,000+ errors in the compile, fix that error and they all go away).