Show rust: A syntax-diagram generator for macro_rules!()-syntax

[lukaslueg]

Hi,

allow me to shamelessly crosspost from reddit: As I found the situation about documenting macro_rules()-syntax not very satisfying, I wrote a library to automatically generate syntax (“railroad”) diagrams directly from code. The obvious target for this is docs.rs, where it might do some good. I’m now looking for comments if rust thinks this thing is useful and worth the effort.

A live demo, compiled to WASM, can be found here.

There are more examples of the macros from nom, stdlib, syn and various others.

[kennytm]

Wow thanks for the great tool!

Minor nit picks:

1. The background color of pat (blue) makes the text unreadable.

   

   macro_rules! a {
   (
   $item:item
   $block:block
   $stmt:stmt
   $pat:pat
   $expr:expr
   $ty:ty
   $ident:ident
   $path:path
   $meta:meta
   $lifetime:lifetime
   ) => {};
   }
   

2. :literal and :vis are not supported

[DanielKeep]

Macro-internal rules omitted

*claps* I have to admit, I’m rather pleased to see that. Very nice work.

Also, I think the actual obvious target for this is rustdoc itself.

[lukaslueg]

Fixed :literal and :vis, thanks for pointing this out.

[shingtaklam1324]

Apart from Rustdoc, I feel like some sort of IDE integration could also be useful, and it could speed up the development of macros significantly

[obust]

This is a great tool you have here lukaslueg !

For documentation purposes, I agree one obvious target is rustdoc, which can be discussed on the Discord channel https://discordapp.com/channels/442252698964721669/459149231702278154

For development purposes, having it part of the rust playground could be nice https://play.rust-lang.org/ Otherwise, https://lukaslueg.github.io/macro_railroad_wasm_demo/ can be polished into a full fledge website for Rust Macro development, akin to https://regexr.com/ for regexes. I see more potential there than an IDE integration for example.

[lukaslueg]

The discord-channel you linked is private or 404 !?

[obust]

Hm, maybe you have to log in once before. Here is a public invite https://discordapp.com/invite/rust-lang Then go to the #docs channel.

[lukaslueg]

I consider this whole thing a tech-demo right now. My paramount concern is being correct™ and staying that way in the long run when it comes to parsing: There may be situations where the user errs in his interpretation of the diagram, yet there must never be a situation where the diagram is plain wrong by itself. We already have issues here.

The current parser was provided by @dtolnay and it’s beyond my depth whether I should continue using syn or I should start using libsyntax (which may provide cool benefits).

[lukaslueg]

The grammar only mentions sep_token in passing and we currently fail to parse tokens like this and others (e.g. 'a) correctly. Is there a formal description? I can’t for the life of me figure out libsyntax and do want a correct parser (which is not to permissive). If anyone wants to jump in here or here, be my guest

[ehuss]

The grammar only mentions sep_token in passing and we currently fail to parse tokens like this and others (e.g. 'a ) correctly. Is there a formal description?

I recently updated the tokens documentation, which should be a fairly accurate description of all the different token types. There are some complexities with token splitting during parsing, but I think that only happens after macro expansion. The reference is currently the most up-to-date specification, but it is unfortunately incomplete, particularly for macros.

In the case of sep_token, that is just any arbitrary token (AFAIK).

[jjpe]

You’re correct and this is an awesome project, but generating a syntax diagram for macro’s is the least of it. I’d like to see the equivalent of generating class diagrams (I suppose we’d call them struct diagrams, or type diagrams if we want to include enums, tuples etc) and perhaps even sequence diagrams where possible.

That would enable Rustaceans to just start programming their ideas (which gives you literally infinitely more feedback than manually creating such diagrams would ever be capable of giving you, at least in terms of whether the design can even actually work in practice) and then just generate the other artifacts that make some people really happy.

As for where all this should happen, it could either be included by default in the docs generated by cargo doc, or it could be a separate command e.g. cargo generate-diagram --all, cargo generate-diagram --syntax, cargo generate-diagram --struct etc.

[ivanceras]

Hi, I saw your railroad diagram project and its pretty cool. I was toying on making the equivalent ascii diagram of the raiload output and come up with this.

       ┌------┐   .-.  ┌---┐
o______| elem |__( ; )_| n |_____o
   |   └------┘   `-'  └---┘  |
   | ╭------>-----╮           |
   | |    ┌---┐   |           |
   ◟_◞____| x |___◟___________◞
   |   |  └---┘  |            |
   |   |   .-.   |            |
   |   `--( , )--'            ^
   |       `-'                |
   |   ╭-------->--------╮    |
   |   |  ┌---┐   .-.    |    |
   ◟___◞__| x |__( , )___◟____◞
        | └---┘   `-'  |
        `-------<------'

                                   .------------>---------------.
       ┌-------------┐  .-.   .-.  |  ┌------┐  .-.   ┌-----┐   |    .-.   ┌------┐
  O____| struct_name |_( : )_( | )_◞__| name |_( : )__| tpe |___◟___( | )__| body |______O
    ◝  └-------------┘  `-'   `-'   ◜ └------┘  `-'   └-----┘  ◝     `-'   └------┘  ◜
    |                               |                    .-.   |                     |
    |                               `------------<------( , )--'                     |
    |                                                    `-'                         |
    `--------------------------------------------------------------------------------'

I’m planning on supporting this ascii diagram to render nicely in svgbob soon.

It would be a portable intermediate output, so we can paste this on the source code. How,ever svgbob still has still have a long to have support for detecting shapes and coloring it.

I would be very much pleased if the rustdoc team has interest in integrating this diagramming capabilities in the documentations.

[lukaslueg]

FWIW, I’ve found time to fix the optimizer getting itself stuck while folding common syntax-patterns. Beasts like nom’s method now fold beautifully.

[lukaslueg]

The general opinion seems to be positive about this. Is anyone willing to lay out what would be required of the crate itself for a possible inclusion in rustdoc? That’d be a goal to work towards.

[QuietMisdreavus]

Technically speaking, the biggest hurdle to pulling this into rustdoc proper is swapping syn for libsyntax. We get plain TokenStreams for macro definitions, but we don’t keep them for very long - instead we pluck out the Spans for just the matcher arms, convert them to strings, and discard everything else. To give you the same kind of input we’ll need to also save the TokenStream. (You can see where we grab it for local macros and for re-exported ones.)

Once you have that, we’ll want to make sure it appears on the page correctly. How good is SVG support across browsers? We have our official tiered browser support listing, but unofficially, we try to make sure our docs appear fine on browsers way older than that list. Also on a much smaller note, we’d probably want to make sure the colors switch fine when you switch themes in the docs, meaning we’ll want to make sure the colors use classes defined in the main CSS theme, even if it means updating that CSS to add the relevant classes to the color scheme.

---

However, i’m a little skittish about adding this to rustdoc. Speaking personally, while i think the diagrams are cute, i have a really hard time actually reading them. The way all the tokens get spaced out, it’s hard for me to follow what each branch is meant to represent. I don’t know how much of that is the fault of the railroad diagram and how much can be tweaked outside of it, but it just feels harder to read than if we managed to clean up and/or syntax-highlight the original macro_rules statements.

Moreover, i’m not sure how well it would fit into the style of how rustdoc displays things. Part of the design ethos in rustdoc is to format things as if they were Rust code, but macros create this gray area where they can accept literally anything if you write the right matcher for it. So i can see where having something other than the macro_rules! statement is useful, but i’m not sure the railroad diagrams are the answer. At least, not something to apply to every macro ever.

(I may be more open to adding it if it were behind a CLI flag, or were a kind of #[doc(railroad)] attribute you could add to macro definitions. This would also help us keep it unstable during the initial integration.)

[eminence]

The official Rust book (and anything built with mdbook) has a mechanism to let example code be run via an external play-pen service. This is done via an unobtrusive “play” button in the corner of each example.

What do you think about something similar for macros? There should be a small “render” button in rustdoc output that would send the macro to an external service for rendering (either displayed inline, or open a new browser tab).

[jsgf]

I think it’s a big improvement over what we currently have. Its output for some of my macros is a little dependent on how I implemented the grammar rather than how I’d think about the grammar, but I find it much more readable than trying to interpret the raw macro definition.

I certainly don’t think it should be displayed inline by default, but some kind of pop-up/down would be nice I think (esp if it could be opt-in via annotation).

Though I suspect I’d probably use this at least as much to debug a macro that I’m trying to write - I find it very easy to overlook typos and the existing tooling doesn’t help much in understanding what grammar the macro actually accepts.

[jjpe]

but it just feels harder to read than if we managed to clean up and/or syntax-highlight the original macro_rules statements.

Seems like a false dichotomy to me. Why not have both? Either one could be hidden behind e.g. a [+] button similarly to trait method entries are today.

[lukaslueg]

Allow me to shamelessly (again!) cross-post from Reddit. I’ve created a browser add-on for docs.rs which executes in Firefox and Chrome; code here, there is also a pre-built package for Chrome in the WebStore. If one is interested in this whole thing, we can try stuff out that way.

Oh and as far as I can see this is the first WebExtension written in Rust