`pub` on `macro_rules!`

language design
1

Macros defined with macro_rules! have weird scoping. As I understand it, this is due to historical compiler limitations, and the compiler could support normal scoping now. I understand that there's some sort of effort to add a new macro construct that would not only fix scoping, but also do some other things, but I assume that's a long way away. Also, there could be reasons someone would want the old kind of macro definition, but the new scoping. I want to see if there's an appetite for a “quick fix” for the current scoping problems.

Background

Currently, there seem to be three patterns for the scoping for a macro_rules! macro.

  1.  macro_rules! foo {
     …
 }

    The macro is only available outside the module with #[macro_use] and doesn't have normal item scope at all.

  2.  macro_rules! foo {
     …
 }
 pub(crate) use foo;

    The macro gets normal item scope, but it's only visible within the crate and can't be made fully public (pub use foo; is an error).

  3. Publicly exported from the crate root:

    #[macro_export]
macro_rules! foo {
    …
}

    The macro gets normal item scope and is fully public, but it will be exported from the root of the crate. You can re-export it from the current module, but you can't hide the re-export.

I try very hard to keep my code clearly organised and well-documented. When writing macros, I want to group them together with related functions, types and constants in a module with an appropriate name, documentation and position in the hierarchy. But none of these options work well with that in mind:

  • The #[macro_use] approach means you usually can't see where a macro comes from when reading a module that uses it. It does at least show up in the correct section of rustdoc output, but intra-doc links don't work. [EDIT: Actually they do work, but forward references require inserting use statements! More importantly, these show up with :lock:.] :frowning:
  • The pub(crate) use approach means the macro will show up with :lock: (“restricted visibility”) in documentation, suggesting it shouldn't be used outside a narrow context, and it can't be used by other crates. :frowning:
  • The #[macro_export] approach means the macro will show up in the crate root rather than an appropriate module when reading the rustdoc output, and IDEs will automatically insert use statements with the “wrong” path, which I can only catch with manual code review. It subverts my attempts to organise things into appropriate modules. :frowning:

Proposal

What I'd really like is to be able to write:

pub macro_rules! foo {
    …
}

The macro foo! would be a public item with normal scoping, exported from the current module rather than the crate root.

If this specific syntax is doable, this would also provide an alternative to the weird pub(crate) use foo; idiom:

pub(crate) macro_rules! foo {
    …
}

But more than that, it would also work for any other visibility level you can express with pub(…), e.g. self or super. I'd probably use the latter quite a bit.

What do you think? Does anyone know if there's some technical limitation that would prevent this?

11 Likes
2

Even if it isn't possible, I dont see why you shouldn't be allowed to do pub use foo;. Why would it only be allowed at the crate level? That makes no sense to me (syntactical).

I can see that perhaps visibility macro! could be a problem for the parser (after all, macro_rules! itself looks a bit like an invocation of a macro!) depending on what is resolved at what stage. But I could be wrong, I'm just a rust user, not a developer of rust.

3

Even if it isn't possible, I dont see why you shouldn't be allowed to do pub use foo;. Why would it only be allowed at the crate level?

When you re-export something, you can't give it a greater visibility than it already has, e.g. this is not allowed:

mod private_module {
    pub(super) struct Foo;
}
pub private_module::Foo;

This is why pub use some_macro; doesn't work, as I understand it. It's treated like a re-export of something, and the macro has crate visibility. If you use #[macro_export] on the macro, then it has public visibility, but with the problem of it being exported from the crate root already.

Technically a special case could be made for this, but it's an ugly one. I'd rather change macro_rules!.

1 Like
4

pub macro_rules! was attempted once before, but it was dropped back then because it didn't interact well with name resolution at that time. Now that the integration of interleaved macro expansion and path resolution has developed some, it might be easier to do.

Right now, there are options like macro_pub (transparency: my crate[1]), macro-vis, or macro-v (have not reviewed) which provide an attribute shimming the workaround.

  1. And also, the crate was and is an experiment/example in writing a proc macro without syn, just with proc_macro directly. ↩︎

6 Likes
5

Ah, thanks for the context. Did the previous attempt come before macro_rules! some_macro { … } use some_macro; was possible?

6

It's long enough I don't quite recall. I think use was possible at that time, but the issues were IIRC more about processing the macro_rules! itself than use. Scanning PR history, #88019 (2y ago) seems like a relevant change that would make a relevant impact (roughly, make macro_rules! exist in HIR instead of disappearing in the source->HIR lowering). #91795 (2y ago) also seems helpful (I think at some point #[macro_export] also looked like a pub use to rustc?).

I'd check a number of rustc versions with pub macro_rules! syntax to see which recognize it in order to try to refind the relevant timeline, but can't do so from mobile, unfortunately.

1 Like
7

Name resolution interactions was fine.

It was dropped because edition lints for replacing macro_use and macro_export with pub macro_rules required an order of magnitude more work than pub macro_rules itself, and nobody was willing to do that work.
(The original implementation also had one easy to fix metadata encoding bug.)

If the feature can be supported without all that migration machinery, it can be done very easily.

2 Likes
8

Edition lints? Like, checking that the code is allowed in the current edition? I would have thought this was a backwards compatible change and could be allowed even in the 2015 edition.

1 Like
9

The reasoning was like this: pub macro_rules will work on all editions -> but it also gives an opportunity to remove macro_use and macro_export on the next edition (2021 back then) -> so we need to write migration lints that will suggest using pub macro_rules instead of macro_(use,export) -> no resources to write the lints before the edition release -> pub macro_rules was scrapped as well.

10

Ah. That sounds rather perfect-is-the-enemy-of-the-good to me…

7 Likes
11

If we actually wanted to deprecate macro_use and macro_export, we'd need such migration machinery. However, adding a new mechanism as an available option for people to use does not seem like it should require any kind of migration.

Yes, it's potentially an opportunity to migrate, but if the lack of migrations is blocking the ability to introduce the feature and make macros substantially simpler for many people to deal with, let's skip the deprecation-and-migration.

(IIRC, there was also an issue about macros within a crate? Would pub(crate) macro_rules! work as expected for accesses from within a crate?)

5 Likes
12

I would expect this to behave the same as macro_rules! foo { … } pub(crate) use foo; does right now. If that has some limitation, this would inherit it.

1 Like
13

Hmm. I'm not familiar with deprecate-and-migrate cycles. Suppose we decide to remove macro_use and macro_export in a future edition, which is a long way from releasing. Do we have to add a deprecation warning immediately, or is it just a requirement that it's been warned about for some undetermined length of time?

1 Like
14

That depends, and there isn't a universal rule for how far in advance. The primary rule we've been somewhat enforcing is that if we're migrating from one syntax to another in an edition, such that the old syntax won't work anymore, there needs to be a migration lint.

1 Like
15

What would it take to revive this? If part of the answer is "a decision that we don't want to deprecate the old syntax", for instance, I'd be happy to help arrange that decision.

5 Likes
16

Regarding the proposal of $pub:vis macro_rules!, it's still stuck on the meaning of /* no vis */ macro_rules! (henceforth referred to as "pub-less macro_rules!"):

  • the idea was for it to mean the same as pub(self) macro_rules!, but this breaks certain old recursive macro definitions which just assume to be in scope even in parent modules (through #[macro_use]) or whatnot;

    • obviously the "breaking change" here was imagined in an edition boundary, but on the condition that there be nice compiler-generated suggestion to fix it. And that turned out to be a rabbit hole of edge cases that blocked the whole thing.

One thing we could do, thus, is to punt on pub-less macro_rules! meaning the same as pub(self) macro_rules!, and still enable the non-pub-less macro_rules! (pub macro_rules!, pub(crate) macro_rules!, pub(self) macro_rules!, etc.):

  • the compiler machinery for all this is already available, it's just a matter of switching the knob to enable it;

  • as the rest of my post will show, the world of macros is already plagued with hacks. So having pub(self) macro_rules! not be exactly the same as macro_rules! will:

    • be hardly noticed except for people writing pub(self) macro_rules! … at the end of a module (at which point they'll probably be doing so deliberately);

    • still be order of magnitudes better than the current plethora of hacks;

    • remain in line with the ideä of an on-edition-boundary change for pub-less macro_rules! definitions;

That is, the expected workflow I could imagine is:

  1. Somebody writes macro_rules! macro_name … at the beginning of a module, as they do now. The macro works for the current module, and submodules defined after the macro.

  2. they reälize they want the macro to be called from a parent module (or downstream crate). They then slap pub(crate) (or pub) on it:

    • the parent module (or downstream crate) can now refer to it by path;

    • current stuff Just Works™

      • but for the submodules now needing a not-so-surprising use super::… (:point_left: this is the only difference with the "perfect" (but stuck) proposal of having pub-less macro_rules! behave exactly like pub(self) macro_rules!: the use super::*; would have been already needed to begin with).

That seems like a rather seamless workflow, despite the legacy behavior for pub-less macro_rules!.

FWIW, in the public case, now that rustdoc lets #[doc(inline)] override a #[doc(hidden)], the current approach is to do:

#[doc(hidden)] #[macro_export]
macro_rules! __foo { … }

/// …
#[doc(inline)]
pub use __foo as foo;

This yields a properly-scoped and pub-visible macro, like a pub macro_rules! would (or pub macro does).

  • the only caveat is that this still requires you to avoid using the same foo name multiple times per crate (since the #[macro_export] will make them clash at the root of the crate :pensive:). If this is something you need, then using one of the more feature rich aforementioned helper crates may be warranted.

The non-pub case is then the

macro_rules! __foo { … }
pub(restriction) use __foo as foo;

you mentioned (and we almost always can use foo instead of __foo, thereby simplifying it down to your:

I'd say that beyond immediately-called macros,

(+ #[macro_use]) ought to be avoided.

All this does mean that it is possible to define your own meta-macro to handle this properly, much like the aforementioned third-party libs, except now we can reduce it down to the pervasive paste!:

scoped! {
    pub macro_rules! foo { … }
    // and/or
    pub(crate) macro_rules! bar { … }
    // and/or
    macro_rules! baz { … }
}

#[macro_export]
macro_rules! scoped {(
    $(
        $( #$attr:tt )*
        $( pub ($($restricted:tt)+) )?
        $( pub $(@$if_pub:tt)?      )?
        //     ^^^^^^^^^^^^^^^
        //     `$if_pub:empty` matcher when? 🥺👉👈
        macro_rules! $macro:ident $rules:tt
    )*
) => (::paste::paste! {
    $(
        $( #$attr )*
        $($($if_pub)? #[doc(hidden)] #[macro_export] )?
        macro_rules! [< __ $macro >] $rules

        $( #$attr )*
        #[doc(hidden)]
        #[allow(unused)]
        $(pub ($($restricted)+))? $($($if_pub)? pub)?
        use [< __ $macro >] as $macro;
    )*
})}
12 Likes
17

Oh, wow, thanks for the very detailed reply! I'm especially thankful for you telling me about the documentation trick, as that finally gives me a good immediate solution to my problems. But I think the complexity and undiscoverability of it all speaks for itself: being able to just use a visibility qualifier would make things easier for everyone.

Regarding pub-less macro_rules! use: I was imagining leaving its behaviour as is. It would be nice if it could be changed in some future edition, but in line with my whole theme here of not letting perfect be the enemy of the good, I do not want that to block a potential improvement in the nearer future.

6 Likes
18

Right, thank you! The behavior of macro_rules! without a visibility was indeed a major issue, and making it (over an edition) mean pub(self) macro_rules! got blocked on the challenge of doing migrations (which would need to know if anything in the crate invoked the macro).

I agree that, for now, we could decouple the two by having macro_rules! keep the same meaning and requiring pub(self) macro_rules! for truly private macros. I do think we eventually want to make that transition (whether or not we can fully automate it), but let's not block on that to make pub and pub(crate) macros much easier to understand by having them appear in the module they're declared in rather than the top of the crate.

6 Likes
19

We can deprecate macro_* and replace it with macro. So breaking change may be defer to some of the next editions

20

macro is currently reserved for the future macros 2.0 system.

(That doesn't prevent us from considering this, just a consideration.)

1 Like