Procedural Macros book

[alexreg]

With the proliferation of books on various topics within Rust (as well as larger 3rd party libs), I was thinking the community would really benefit from a book on the now reasonably mature feature of procedural macros. I’ve gotten positive feedback to the idea when I’ve brought it up, but I’d be curious to hear what the proc macro guys and docs team think of it. It would be a bit of effort to get this off the ground, but well worth it in the end, I feel. More and more crates have an associated *_derive crate these days, and even more that don’t currently have one could probably benefit from one.

CC @alexcrichton @dtolnay @steveklabnik

[steveklabnik]

I think such a thing would be great to have, personally.

[dtolnay]

I would be interested in writing down everything that I know about procedural macros. Books are a lot of work and this one would take potentially a couple full-time person-months to do properly, so someone may need to be willing to pay for the book to exist.

[JeanMertz]

someone may need to be willing to pay for the book to exist

I would be willing to pay to own a copy of the book. I know that’s not what you meant, but perhaps if enough people show interest, some pre-order system might help get us there (though I realise if we want to pay someone several months of salary, that requires a group larger than the one lurking on IRLO).

Another possibility is a Kickstarter-like set-up if someone with enough knowledge and a “trustworthy status” (like you two) would be invested in this.

Aside from the topic of finding the time/money to invest in this, I would definitely be very interested in such a book. I’ve asked @dtolnay for their help on some macro related questions directly on the syn repo before – which I realise is not really the right way to do this, but two months ago, the documentation was even more sparse than it is right now – and have asked proc-macro questions in #beginners on Discord over the last couple of months as well (incidentally, I just asked another question, since I was about to pick up some more proc-macro development).

[djc]

A procedural macros book authored by @dtolnay sounds like a good idea, and I also think a crowd funding approach would be interesting to try at least. Possibly (if @dtolnay is willing) with a (stretch) goal to release it into the public domain or with an open source license once it’s paid for?

[mark-i-m]

I worry about stretching the docs team too thin. It’s already really hard to find people who are willing/have time to write docs, and there are a lot of areas where docs needs some love IMHO (e.g. RBE, cookbook)…

Perhaps a short community-maintained guide might be a more manageable short-term goal?

[alexreg]

To be honest, when I wrote this, I had both a community-maintained guide and a proper book in mind. I wasn’t sure what the better approach would be. Either would very beneficial at this point, since we lack both. @dtolnay If we just went for the first, maybe you could help kick this off and “guide” the process?

[dtolnay]

As nice as it would be, the community isn’t going to write a book on “everything that @dtolnay and @alexcrichton know about procedural macros” because practically none of this has ever been written down.

I have found that there is a huge difference between what people think will be hard about procedural macros before writing any procedural macros, vs what actually is hard about procedural macros. It is easy to write a book about the former but I think doing so would be misguided.

As a basic concrete example, the ghost crate illustrates the distinction I am making. From a naive perspective this is a very simple macro – it parses some simple input using syn in a simple way and generates some simple output using quote in a simple way. How all of this happens is pretty uninteresting and someone could pick it up from an hour in the Syn docs and example code. Unfortunately I suspect this is what most people believe belongs in a procedural macro book. It belongs in syn and quote docs.

The resource missing is everything that does not belong in the syn and quote docs. This largely centers on patterns for what code to generate. This is where you find practically all of the relevant domain expertise for writing actual macros and is knowledge that transcends any specific how libraries like syn and quote.

[alexreg]

Yes, I know what you mean… perhaps the more time-efficient way for you is to write very informal notes on the topic, and others turn it into a more formal guide/book?

[dtolnay]

Bluntly but sincerely: subject to the constraint of the amount of volunteer time that I am able to dedicate, I don’t foresee a guide or book being produced that is worth reading.

[oooutlk]

I think a collection of existing code in the form of “proc-macros by example” would help.

[vorner]

I’ve went through implementing a procedural derive recently and yes, I can say that while working with the syn crate is certainly tiresome (mainly due to the low-level nature and the size of both the crate and the amount of possibilities from the actual rust syntax), that can be gotten through just by the API documentation and application of enough time.

What I was missing was some of the best practices ‒ both in form „How do I handle all kinds of attributes“ and „watch out for this trap, if this happens…“ things. I’m pretty sure the thing I wrote (structdoc-derive) works in the usual cases, but I suspect it will break terribly with arcane error messages under some corner cases.

I don’t know your actual time constraints (though judging by mine, I can imagine it’s „I have negative amount of time I can volunteer in addition to what I already do“). But it would be nice to find a way out of the situation „There’s only one person that is able to write good proc macros“.

So I’d be interested in few things:

• Is there someone else who is good at writing procedural macros? Authors of other derives might to at least some level.
• Would there be some way to transfer the knowledge to someone else that wouldn’t cost you that much time? Like, instead of writing the next proc macro you would need to, to take on an „apprentice“ ‒ that one would do the actual work for you, but you’d guide them and do review. That could make the amount of work you need to do about it potentially smaller or about the same, but another person could learn it.
• Not going to a full book, but some kind of FAQ or a blog post by someone who did write some proc macros and you only checking it’s correct and makes sense?

[alexreg]

This sounds reasonable to me, yes. Who else is good is at writing proc macros? Maybe we should just search through the _derive crates.

[Lokathor]

I wrote a proc macro and it wasn’t that hard, it was more just a long slow slog.

The second version of things, and the third, got easier and easier.

I think it’s entirely too soon for a book about proc-macros. Here’s an alternate plan that’s far easier to get off the ground: a proc-macro example repository where people can PR to add working, complete proc-macros (single file or all in a directory) that are well commented about what’s going on, including the expected input formats, expected outputs, and so on.

Among the rust folks I normally talk to (rust community discord, ~250 people), I know one other person who does proc-macro work. They’re “hard” because no one knows anything yet. We just need to show proc-macros off more and the rest can soon enough solve itself I think.

[dtolnay]

I think it’s entirely too soon for a book about proc-macros.

I agree with this, but only because a book about proc macros isn’t what is missing. I alluded to this above. The book that needs to be written (and is ready to be written now) is primarily a book about Rust not a book about macros – there are language features and patterns that would never be documented in something like TRPL because you don’t find them much in handwritten code, but are extremely useful for macros.

It isn’t a book about proc macros because it may not even teach you how to write proc macros beyond referring to existing resources, but in order to develop powerful and robust macros you would need to know this information and there doesn’t currently exist any resource for ramping up on this stuff.

[Lokathor]

Could you speak more about what you think the rust book would contain? That post and the one you linked to are a little light on the details of what you’d put in this general rust book.

[samsieber]

Having written a slightly complex macro (https://crates.io/crates/rubber_duck), here are a couple things I found useful that I more or less stumbled upon:

• functions, structs and macros occupy different name spaces (structs, traits and modules occupy the same namespace - the “type” namespace). That means you can have one of each in the same module and with a single import, you import them all! That means you can have other things in the background that the user doesn’t have to worry about, and it’ll just work with the paths being used. Though I found this to be most useful for a proc macro that generates things to be used by a generic macro, so ymmv.
• Hygiene tricks. I don’t have anything specific here, but I certainly struggled with this one. I suspect that the approach rubber ducks uses is suboptimal. I basically just tried different things until it worked.
• Again with the proc-macro to produce stuff consumed by other macros - type safe builder were awesome for enforcing invariants at compile time.

I think those things barely scratch the surface, and probably aren’t actually what dtonaly is talking about, but is probably closer to what he’s talking about.