# Pre-expanding proc macros **URL:** \ **Category:** tools and infrastructure\ **Created:** [October 12, 2022, 8:03pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550 "2022-10-12T20:03:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\ **Post date:** [October 12, 2022, 8:03pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/1 "2022-10-12T20:03:36Z")
Re @matklad[avoiding serde in projects](https://internals.rust-lang.org/t/a-list-of-commands-to-pre-configure-the-project/17492/8): I wonder if crates could expand proc macros back to source code, off-line, before being published. This way a crate could use any number of proc macros without exposing that fact to its users. It wouldn't need `syn`/`quote`/etc. for every compilation downstream, only once before the crate is published. In the JavaScript world such strategy has been used for CoffeScript and ES6 - they could be compiled down to ES5 and published as a pure-ES5 package without dependency on any translation machinery. ---
**Author:** ![jhpratt](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/jhpratt/32/11640_2.png) [@jhpratt](https://internals.rust-lang.org/u/jhpratt)\ **Post date:** [October 12, 2022, 8:11pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/2 "2022-10-12T20:11:44Z")
I would love this. I have deliberately avoided syn due to compile time, and it definitely makes writing macros more tedious. Having a background in JavaScript, I can confirm that it absolutely works there. One concern, though, is conditional compilation. ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 12, 2022, 8:19pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/3 "2022-10-12T20:19:57Z")
This doesn’t even need any kind of special support form the compiler or Cargo. The only thing needed is for the proc-macro to also be published as a library operating on a proc-macro2 token-stream. The “driver” code to std::fs::read\_to\_string .rs files, parse the with syn, and find the structs to apply derives to doesn’t seem too hard. The consumer library then can include a test which checks freshness of the generated code and updates it if it isn’t fresh. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [October 12, 2022, 9:36pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/4 "2022-10-12T21:36:40Z")
On [T-cargo](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/publish.2Ers), we did some high level brainstorming for build.rs and proc macro generated results to be published with a package. ---
**Author:** ![nacaclanga](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nacaclanga/32/7461_2.png) [@nacaclanga](https://internals.rust-lang.org/u/nacaclanga)\ **Post date:** [October 13, 2022, 8:07am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/5 "2022-10-13T08:07:49Z")
The way I understand you proposal is that you basically suggest to run cargo-expand on the code and the publish the output. I am not sure if that works well, given that macros distinglish between the macro and the use scope for hygene reasons. Also dublication can be avoided, if the compiler has special infrastructure for this likely rather common case. My personal choice would be some human readable text format for macro-fragment files and when rustc is passed a macro-fragment dir, via a new compile switch, it tries to find a suitable macro fragment there, before invoking the actual macro-code generation. ---
**Author:** ![dhardy](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/dhardy/32/2399_2.png) [@dhardy](https://internals.rust-lang.org/u/dhardy)\ **Post date:** [October 13, 2022, 8:45am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/6 "2022-10-13T08:45:00Z")
> **[GitHub - dtolnay/watt: Runtime for executing procedural macros as WebAssembly](https://github.com/dtolnay/watt)** > > Runtime for executing procedural macros as WebAssembly - GitHub - dtolnay/watt: Runtime for executing procedural macros as WebAssembly ---
**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\ **Post date:** [October 14, 2022, 4:10am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/7 "2022-10-14T04:10:53Z")
> [@matklad](#): > > The only thing needed is for the proc-macro to also be published as a library operating on a proc-macro2 token-stream. To be clear, this isn't completely sufficient. The macro also needs to be written such that it does not rely on anything other than call-site hygiene. We don't have access to def-site hygiene yet, but we _do_ have stable access to [mixed-site hygiene](https://doc.rust-lang.org/proc_macro/struct.Span.html#method.mixed_site). Additionally, it can be the case that the hygiene of passed in tokens matters. I've used watt before, and it's awesome tech, but quite cumbersome to use. I've toyed with attempts at making it easier to use -- without committing a binary wasm blob into a git repo -- without breaking path dependencies, but never quite managed to push a solution over the finish line. The main thing preventing a builtin wasm abi and runner for proc macros is aiui just developer time. Though if you're interested in working on wasm+proc-macro, please do ping @eddyb (and maybe me? 🥺👉👈 I'm interested too) as they've done a decent amount of design thinking around it. Bonus points if you can make proc\_macro usable outside the proc-macro server at the same time ---
**Author:** ![carbotaniuman](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/carbotaniuman/32/6981_2.png) [@carbotaniuman](https://internals.rust-lang.org/u/carbotaniuman)\ **Post date:** [October 14, 2022, 6:25am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/8 "2022-10-14T06:25:04Z")
Just an fyi, dtolnay put out a call for help/comments on Twitter a few days ago, but the exact post has since eluded me. ---
**Author:** ![eddyb](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/eddyb/32/8411_2.png) [@eddyb](https://internals.rust-lang.org/u/eddyb)\ **Post date:** [October 14, 2022, 8:21am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/9 "2022-10-14T08:21:12Z")
> [@CAD97](#): > > Bonus points if you can make proc\_macro usable outside the proc-macro server at the same time You'll want to ping @mystor for that (and the wasm stuff ofc), she's done most of the legwork lately to which I'm more of a bystander. ---
**Author:** ![djc](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/djc/32/1592_2.png) [@djc](https://internals.rust-lang.org/u/djc)\ **Post date:** [October 14, 2022, 3:02pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/10 "2022-10-14T15:02:12Z")
> [@carbotaniuman](#): > > Just an fyi, dtolnay put out a call for help/comments on Twitter a few days ago, but the exact post has since eluded me. > @dtolnay @sunfishcode ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 14, 2022, 4:05pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/11 "2022-10-14T16:05:56Z")
Note that it will need a fallback wasm interpreter as wasmtime only supports x86\_64, aarch64, s390x and riscv64gc right now. ---
**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\ **Post date:** [October 14, 2022, 6:01pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/12 "2022-10-14T18:01:16Z")
Not necessarily -- the fallback can just be the native compilation used today. ---
**Author:** ![bjorn3](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/bjorn3/32/2736_2.png) [@bjorn3](https://internals.rust-lang.org/u/bjorn3)\ **Post date:** [October 14, 2022, 6:21pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/13 "2022-10-14T18:21:16Z")
That would require a special case in every build system to check if rustc supports wasm proc macros on the current target and if not switch to compiling native dylibs. It would also mean that we can't remove the `-Zdual-proc-macro` hack that is necessary for cross compiling rustc, but breaks cross compiling tools linked against rustc itself. ---
**Author:** ![crlf0710](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/crlf0710/32/1987_2.png) [@crlf0710](https://internals.rust-lang.org/u/crlf0710)\ **Post date:** [October 20, 2022, 11:22am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/14 "2022-10-20T11:22:04Z")
[Just posted another topic on this today...](https://internals.rust-lang.org/t/idea-pre-expanded-crates-and-expansion-dependencies/17595) Sorry i didn't found this topic because of the title! I think the major point is the pre-expansion of the usage crate, and whether the proc macro crate itself is pre-compiled is less important. pasting the major points from that post: > I'm thinking maybe it's possible to add a new kind of dependency in Cargo.toml, maybe call it [expansion-dependencies], that acts just as normal dependencies. But instead, before uploading to [crates.io](http://crates.io), the proc macros defined in these crates got expanded and inlining the expansion results into the original source code. Only the expanded source code is uploaded to [crates.io](http://crates.io), and the dependency got automatically removed. > [@Idea: Pre-expanded crates and \`expansion-dependencies\`](https://internals.rust-lang.org/t/idea-pre-expanded-crates-and-expansion-dependencies/17595/2): > > It’s very common for proc-macros to rely on an exact version of their runtime crate by using non-public semver exempt details from them. Pre-expanding would then extend this exact version requirement to the published crate, likely causing many unresolveable dependency trees. > a proc macro crate can declare itself as able to be pre-expanded or not. For crates like `thiserror` and `displaydoc` , i think they're perfectly fittable for such pre-expansion. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [October 20, 2022, 12:04pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/15 "2022-10-20T12:04:46Z")
> But instead, before uploading to [crates.io](http://crates.io), In the [above linked zulip thread](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/publish.2Ers), we were discussing the `build.rs` version of this to be a `publish.rs` with `publish-dependencies`. This naming opens the door for both `build.rs` and proc macro handling at publish time. I have seen hesitance in adding more types of dependencies as it would be disruptive to the ecosystem to handle them. I could see these falling under `dev-dependencies` as this is for development but maybe this justifies a new dependency table. Depending on how we handle this, a potential pitfall is if we merge "expansion dependencies" in with the regular "dependencies" during the build. This would cause features to be activated that wouldn't be activated for the published version which would make it harder to identifier or reproduce problems locally. We'd probably have to do an expansion pass locally as well. Something I don't think I've seen addressed yet is how feasible the proc-macro side of this is within the compiler / cargo. While we have "cargo expand", I'm assuming we can't use it 100% the same way, e.g. we would only be expanding some macros. ---
**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\ **Post date:** [October 21, 2022, 7:57am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/16 "2022-10-21T07:57:10Z")
I do think we can just use dev-dependencies for this rather than a new kind of dependency. ---
**Author:** ![Nemo157](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nemo157/32/11585_2.png) [@Nemo157](https://internals.rust-lang.org/u/Nemo157)\ **Post date:** [October 21, 2022, 8:07am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/17 "2022-10-21T08:07:18Z")
That seems expensive in situations such as when used as a `git` dependency. You would start having to build all dev-dependencies for all your unpublished dependencies. ---
**Author:** ![mathstuf](https://avatars.discourse-cdn.com/v4/letter/m/958977/32.png) [@mathstuf](https://internals.rust-lang.org/u/mathstuf)\ **Post date:** [October 21, 2022, 11:02am UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/18 "2022-10-21T11:02:00Z")
> [@epage](#): > > Depending on how we handle this, a potential pitfall is if we merge "expansion dependencies" in with the regular "dependencies" during the build. This would cause features to be activated that wouldn't be activated for the published version which would make it harder to identifier or reproduce problems locally. We'd probably have to do an expansion pass locally as well. What about a field on regular deps to indicate they belong in this (virtual) table? I would worry about publishing untested source code if these tables get desynced (and the publish-generated code doesn't match what anything else is actually using). ---
**Author:** ![Kixunil](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kixunil/32/4110_2.png) [@Kixunil](https://internals.rust-lang.org/u/Kixunil)\ **Post date:** [October 24, 2022, 6:50pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/19 "2022-10-24T18:50:20Z")
Shouldn't it be _build_ dependencies? The macro would run on host machine, not target machine. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [October 25, 2022, 12:13pm UTC](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550/20 "2022-10-25T12:13:59Z")
Good point about host dependencies. The challenge with build dependencies is knowing which build dependencies we can strip and which we have to leave, since not everything will be able to be pre-expanded, even within a single crate. [Next page](https://internals.rust-lang.org/t/pre-expanding-proc-macros/17550.md?page=2)