[Pre-RFC] Allow procedural macros to be placed in the same crate (*package) as app

tools and infrastructure cargo
1

Hey! I had an idea about making proc macros easier to use, and kept thinking about it for the past few days. Here is the rfc:

EDIT: I wasn't aware of the distinction between crate and package, so anytime you read "crate", know that I meant "package".

Summary

Have a new folder in a cargo project, called proc-macro. This would be like the tests directory in that it is alongside the source code. This would eliminate the need to create an extra crate for proc macros.

Motivation

A common thing to ask about proc macros when one is first learning them is: "Why on earth does it have to be in a separate crate?!" Of course, we eventually get to know that the reason is that proc macros are basically compiler plugins, meaning that they have to be compiled first, before the main code is compiled. So in summary, one needs to be compiled before the other.

It doesn't have to be this way though, because we already have this mechanism of compiling orders – the example that come to mind is the tests directory. It relies on the src directory being built first, and likewise we could introduce a proc-macro directory that would be compiled before src.

The motivation of having this new directory comes down to just convenience. This may sound crude at first, but convenience is a key part of any feature in software. It is known in UX design that every feature has an interaction cost: how much effort do I need to put in to use the feature? For example, a feature with low interaction cost, with text editor support, is renaming a variable. Just press F2 and type in a new name. What this provides is incredibly useful – without it, having a subpar variable/function name needed a high interaction cost, especially if it is used across multiple files, and as a result, we are discouraged to change variable names to make it better, when we have new retrospect. With a lower interaction cost, the renaming operation is greatly promoted, and leads to better code.

This proposal aims smooth out the user experience when it comes to creating new proc macro, and achieve a similar effect to the F2 operation. It is important to emphasise that proc macros can dramatically simplify code, especially derive macros, but they a lot of the times aren't used because of all the extra hoops one has to get through. This would make proc macros (more of) "yet another feature", rather than a daunting one.

An objection to this one might raise is "How much harder is typing in cargo new than mkdir proc-macro?" But we should consider if we would still use as much integration tests if the tests directory if it is required to be in a seperate crate. The answer is most likely less. This is because (1) having a new crate requires ceremony, like putting in a new dependency in cargo.toml, and (2) requires adding to the project structure. A tiny bit in lowering the interaction cost, even from 2 steps to 1, can greatly improve the user experience.

In summary (TL;DR), the effort one needs to put in to use a feature is extremely important. Proc macros currently has a higher ceiling, needing one to create a whole new crate in order to use it, and lowering the ceiling, even just a little bit, could massively improve user experience. This proposal can lower it.

Explanation

Currently, we create a new proc macro as so:

  1. Create a new crate
  2. In its cargo.toml, specify that it is a proc macro crate
  3. In the main project, add the crate as a dependency
  4. Implement the proc macro in the new crate

After this change, we create a new proc macro like this:

  1. Create a new directory called proc-macro alongside your src directory
  2. Implement the proc macro in a new file in proc-macro.

To use the proc macro, simply import it via crate::proc_macro.

use crate::proc_macro::my_file::my_macro;

Or, if the file happens to be mod.rs, you can access it directly after the proc_macro bit.

Proc Macro Libraries

Crates like syn, quote, and proc-macro2, would be included under [dev-dependecies] in the cargo.toml. (Perhaps we should put it in build dependencies? or a new dependency section for proc macros.)

How it would work in the implementation

Cargo would have to compile the proc-macro directory first, as a proc macro type (of course). Then, in compiling the main code, crate::proc_macro::file_name::my_macro would resolve the module to the file /proc-macro/file_name.rs. Alternatively, if the user uses mod.rs, it would be resolved from crate::proc_macro::my_macro. This would finally be passed into rustc.

Drawbacks

  1. The proc macro directory cannot use functions from src. (but that was not possible before anyways)

Rationale and alternatives

Have proc macro files marked #![proc_macro_file] to signal to cargo to compile it first.

Since it would compile first, proc macro files cannot import functions in the main code. The problem is having it side-by-side to the rest of your code makes it seem like you could just import it, when you cannot. Having it as a seperate directory makes clear of this.

Eliminate the need for new proc macro files/folders entirely, have the compiler work out where the proc macros are and separate them.

This would suffer from the same issue as the last alternative, plus being harder to implement.

Introspection

Harder to implement, with less payoff.

Prior art

  1. Zig comptime: metaprogramming code can sit directly next to application code.
  2. Declarative macros: can sit side by side as well, but is less powerful.
  3. Lisp macros: same as last two, except more powerful.
  4. tests directory, and build.rs: compiled at a different time as the main code.
  5. Makefiles, or other build systems: they allow for more customisability for when code is built.

Unresolved questions

  1. Should proc macro dependencies be listed under [dev-dependencies], [build-dependencies], or a new [proc-macro-dependencies] section?
  2. Should we import like crate::proc_macro::file::macro, or via a new keyword, like crate_macros::file::macro? The latter would avoid name collisions, but might be more confusing.

Future possibilities

As described in the motivation section, this proposal is aimed to make the process of creating proc macros easier. So a natural extension of this is to remove the need of third-party crates like syn and proc-macro2. There is already an effort to implement quote, so they might be a possibility.

8 Likes
2

Eventually, I would love to see something like this happen. The (Pre-)RFC could use a section explaining how this compares to and differs from previous suggestions, and how it solves the problems that were brought up about those previous proposals. My Google-foo was not with me today, I was only able to find Pre-RFC: Same-crate proc macros but I know that there have been several others.

3

Same package macros would be cool. Same crate macros are really complicated and don't seem worth it at all, given that same package proc macros can offer most of the benefits.

1 Like
4

Note that as @Noratrieb implied, you should probably clarify whether you mean the same crate or the same package. Integration tests, benchmarks, examples etc are separate crates within the same package, and that's almost certainly how same-package proc macros should work as well, just with the dependency arrow flipped.

5

I see. I wasn't even aware that there was a difference between packages and crates. That's my bad; I meant packages and not crates.

Just to clarify, is a crate just a translation unit?

6

Yea... I initally tried to look for previous proposals but couldn't find any. I looked for it for a bit now, but still barely anything. :confused:

7

I would really like to see this also because it could enable some sort of $crate for proc-macros.

1 Like
8

This would actually be handy if a file could be compiled both for the proc macro and for the downstream crate, for doing things like sharing data structure implementations between the two (for parsing, etc.).

That said, I think the Nested Cargo packages RFC is the more general solution to this problem when it comes to publishing, but doesn't address the need for a separate crate. I usually just nest the crate in the downstream crate (e.g. under macros/Cargo.toml alongside my downstream crate's src/ directory).

9

I'm not sure if I understand you correctly. I would think you can just export a data structure in the proc-macro part and reexport it in the main library.

10

Unless I'm out of date or mistaken, currently only proc macros themselves (functions annotated with #[proc_macro]) can be exported from a proc macro crate.

11

Oh, I see. I'll add this to the future possibilities section, although I think this would be harder to implement.

12

Yep, exactly. One crate == one rustc invocation. People often use the term “crate” in casual language when strictly speaking they mean a package (or “Cargo project”). It’s not entirely incorrect because packages have a single “main crate”, either bin or lib, and the rest are more in a supporting role.

13

For anyone that wants to see, I have posted it as an actual rfc at RFC: Procedural macros in same package as app by ora-0 · Pull Request #3826 · rust-lang/rfcs · GitHub.

It has been revised after the feedback.