Caching of Proc-Macro expansions

libs
21

I'm not sure about that. I think the general opinion is that the current way to declare these information in build.rs files is not great, so that might be a possibility to design a better interface and use that in the long run in both places.

1 Like
22

That isn't the correct logic. We can reuse the cache if other parts of the input file changes. It only needs to be rerun if the specific token stream passed to the macro changed. And there may be multiple independent invocations of the the same macro in the same source file.

23

I talked with @epage a bit. To summarize:

  • There are actually two concerns being addressed here:
    • Triggering rebuilds in more cases (informing the compiler about impure dependencies) and
    • Rerunning proc macros in fewer cases (informing the runner that the set of declared dependencies is complete).
  • Modifying the RFC draft from addressing both cases to instead only addressing the latter (by declaring proc macros as being pure) is a reduction in user-impacting scope.
  • As a bonus, communicating this statically (i.e. via package manifest) instead of procedurally means that Cargo can also utilize that same information for already funded improvements to caching.
    • Specifically, putting a build in a multiple-workspace shared cache requires a higher confidence level than is currently used for triggering crate rebuilds. The sccache strategy is fundamentally flawed due to the possibility of undeclared environmental dependencies.
    • The vast majority of proc macros are fully pure and only depend on the input TokenStream, so the incremental step of confirming that makes for significant potential improvement to both cargo and rustc's caching strategies, even without handling impure dependencies better.
    • Additionally, it is meaningfully more convenient for planning shared/distributed cache usage to know statically what crates we're allowed to request from the cache (e.g. consider network caches) than discovering at build time that actually, no, we're not allowed to put this build into the cache after all.
    • The ideal end state for weird build dependencies like sqlx's isn't to maintain the status quo rebuild logic, it's to teach cargo/rustc to comprehend the idea of the needed dependency, or at least allow declaring the presence of untrackable dependencies (warning the developer that rebuilds might not occur despite being necessary).
  • Cooperation cross-team hasn't been ideal. Since both cargo and rustc want access to the fact of "this proc macro has no undeclared dependencies," it is better to have the developer state this in a way available to both interested parties, presenting the experience as a cohesive whole.
    • This point is perhaps the most important. Rustc could start caching proc macro expansions for the purpose of incremental entirely transparently to Cargo, but Cargo and Rustc are supposed to be one thing, not working around perceived development issues with the other.
    • Only getting the list of known dependencies along the existing channels is not sufficient for Cargo; Cargo also needs to know that the set of dependencies is known to be fully complete.
  • TL;DR: promising "I accurately report all of the dependencies of my exposed proc_macros" is a distinct problem from reporting additional dependencies (and a full solution for the 90% case).

Separately from that, I have a few further notes to add:

  • The buildrs rerun-if model of removing default dependencies when requesting a specific rerun condition isn't great.
    • It makes emitting such a directive non-monotonic, thus emitting them automatically from a build helper (e.g. cc or bindgen) is a risky proposition.
    • You might argue proc_macros are better off here since rustc rebuild logic basically treats them as if they don't have any default dependencies, but this is overly optimistic.
  • "Do not cache" is actually far stronger of a request than a proc_macro should have the right to request, since that would mean preventing caching the cargo build result, requiring recompile of all downstream crates for every cargo run.
    • What sqlx wants to say is actually something more like "I have untracked dependencies; ensure the proc_macro gets rerun whenever you would otherwise decide to rebuild the crate, since I can't guarantee the result is the same."
    • Or just "rerun always" is a better way of putting it than "do not cache."
  • There's no good reason to tie proc_macros to listing their dependencies as a complete list in a single call.
    • This prevents compositional structure reporting dependencies from where they occur instead of tracking them to report them in a batch.
    • The perfect future ideal solution of wasi sandbox tracking would require the compiler to track and collect dependencies in this style anyway, so requiring batching manual reporting doesn't simplify anything.
  • Cargo will probably prevent cross-workspace cache sharing if any proc_macros that ever aren't pure are available to the build (for practical build planning reasons) so splitting up proc_macro crates to mark only some proc_macros as pure only to import both crates anyway is actually counterproductive, so putting the flag in crate manifest shouldn't prompt further crate splitting.

And my recommendation:

  • Add a manifest setting for proc_macro crates to say that they fully declare their dependencies.
  • Add a proc_macro API to declare an unspecified dependency, strongly defining when that is guaranteed to result in a rebuild (i.e. never, but we do promise to rerun the macro if a rebuild of the consumer crate happens for any reason).
  • In edition 2027, make this the default behavior of proc_macro crates.
    • If rerun-if semantics had been available to derive macros from day 0, you could've made a strong argument that this was always the intended semantics.
    • This not being the case is that many proc_macros do rely on the unspecified reality of getting run on every compile.
  • At some point, use this information to improve caching, both rustc's incremental and cargo's.
5 Likes
24

Thanks @CAD97 for the summary!

Something I wanted to call out from our conversation is a re-framing of the "is this pure" Cargo.toml field (which for proc-macro expansion would be translated into a rustc flag.

Note: while I'm bringing up proposed Cargo designs, it isn't to increase the scope of proc-macro expansion but to inform the design for more leverage / cohesion as we develop these features mostly independent of each other. This will lead back to proc-macro expansion.

With Cargo's proposed caching scheme, the stakes for cache poisoning are increased. If we reuse a build when we shouldn't in a workspace, a user can run cargo clean and move on. However, in a cross-workspace scenario, the cost for a theoretical cargo clean --global is much higher. Our future plan for this caching system is to allow reading and writing to remote caches, so now an improper build doesn't just affect your system but all other developers using the same cache.

The most likely reason for the cache to be poisoned is a build input that was unknown to Cargo. In addition to what are the build inputs, fundamentally Cargo needs to know whether it knows everything.

By knowing a build unit is pure (whether a lib, build script, or proc-macro), Cargo can a priori determine what isn't at risk for poisoning the cache and plan where each build units intermediate artifacts should be stored / can be found.

If or when we get to dynamically determining whether a package can be cached, it won't be so much that we need to know whether its build process is pure but whether all of the inputs are exhaustive. In decoupling "build input reporting" from "proc-macro expansion caching", I think that that is how we should frame "proc-macro expansion caching". This way we aren't treating "report build inputs to rustc" as mutually exclusive with "can rustc cache the proc-macro expansion. We can report on exhaustiveness in addition to the inputs. As I said, I suspect knowing whether something is pure/exhaustively reported statically will make things easier for Cargo but maybe one day both build scripts and proc-macros can dynamically report whether they are exhaustive.

1 Like
25

Thanks @CAD97 for the summary!

With the two existing functions/build.rs like functionality you only can cover two out of this three variants.

Although it is a hack, you can implement "force rebuild every time" today with proc_macro::tracked_path::path("does_not_exist");, although we might change that behavior in the future. So I think all 3 cases are already implemented and supported. We could possibly add an explicit API for the "always force" rebuild. However, I'm sympathetic to the concern that forcing a rebuild every time is a bit extreme.

I'm still struggling to understand what people are asking for here, and why cargo would need to have more information than it already has. I'm not clear why cargo would need to have some declarative information in relation to a shared global cache. Today, if you use tracked_path and tracked_env, that information gets stored in the fingerprint of the crate calling the proc-macro. Why would that be a problem in a shared cache? Can someone help me understand the concerns here?

26

If we were to assume that all proc macros always properly register all of their dependencies, then cargo doesn't need any more information, and the existing rebuild/fingerprint logic would be sufficient.

However, cargo does not want to have to make this assumption, as proc macros today regularly have untracked dependencies on the build environment. Even rustc is hesitant to silently skip rerunning proc macros without an assurance that all dependencies are being tracked. But for cargo's plans the stakes are raised even further by caching which is larger in scope (and thus more impactful to reset the cache).

To reiterate: the problem for a cache is any untracked dependencies. The larger in scope a cache is, the more likely it is that an untracked dependency will exist and the artifact reused from the cache despite the (untracked) dependency changing, and the larger the impact of getting a caching decision wrong and having to purge the cache as a result.

Knowing whether a given crate build will be able to be cached ahead of time is not strictly required, but it is a meaningful simplification in planning the build.

2 Likes
27

A relevant question is what about build systems other than cargo (such as meson, bazil or buck). In mixed language projects it is common (at least initially) to have them instead. Which approach will be least painful for them?

28

Thanks for summarizing these information. I still believe that we are not trying to solve the same problem, as the proposed Cargo.toml level solution is in my opinion just not sufficient. I would like to define a few partially hypothetical proc-macros to demonstrate how well each of the proposed solutions might work with them.

  • There is serde::Serialize, which takes the tokenstream input and constructs only based on this a trait implementation (or any other rust code). These kind of macros is pure, in that sense that they are only dependent the input. As long as the input does not change the output can be cached.
  • Then there is a hypothetical my_crate::env!("SOMETHING") which is just a re-implementation of the std::env! macro. It will evaluate to the value of the environment variable as &'static str. If the variable is not set that results in a compile time error. It needs to be rerun as soon as the environment variable changes.
  • Additionally there is a my_crate::include_str!("../somefile.txt"), which loads the content of ../somefile.txt at compile time and replaces the macro call with a &'static str with the content of the file.
  • Finally there is the sqlx::query! macro which connects to a database at compile time. This kind of macro should be cached from a compiler point of view.

As far as I can summarize the current discussion I can see three possible solutions:

  • Allow the proc-macro to declare all necessary caching information at runtime as suggested by the Pre-RFC. (The exact function design is not relevant here)
  • Allow to set a caching strategy via the proc-macros Cargo.toml (at crate level?)
  • Allow to define the necessary information as part of the #[proc_macro_*] attribute that defines the proc macro.

I think we can all agree that each of this three solutions is able to cover the needs of the first and the last example proc-macro. For the serde::Serialize macro you would just declare that it is pure/cachable and for the sqlx::query! macro you would declare the opposite (impure/uncachable). It's then up to the implementation to decide what impure translates to in terms of how often and when to rerun the macro. (i.e. if it's reasonable to only rerun it if the crate is recompiled or if it should be possible to always force a recompilation of the crate via that mechanism.)

In my opinion the interesting cases are the second and third example proc-macro. For both cases the actual dependency of the proc-macro (the environment variable or the file path) are only provided to the proc-macro at run-time. I cannot see a declarative approach (like chosen by the Cargo.toml level solution or by the #[proc_macro_*] attribute solution) can even access this information. Or to word it differently what would you write into your Cargo.toml file to declare that a certain proc-macro should be recompiled if the a environment variable with a name defined by the consumer code changes? If you disagree about that point I would like to see your solution to the problem.

There are certainly other approaches here to approach the problem. For one we could just say: This kind of functionality is heavily discouraged to be used in proc-macros. I would argue that at least the file reading is somewhat common, just see the various bundling macros that essentially embed a whole directory into the compiled binary. (For example diesel_migrations::embed_migration!, the include_dir! crate, sqlx::migratio!, etc). The other solution would be to go with a two stepped solution: The declarative solution is a three state: Pure, Impure, DependendOnSomething (again, don't mind the names). If the last variant is declared rustc/cargo would expect that at least one of the existing proc_macro::tracked_path::path or proc_macro::tracked_env::env functions is called to declare additional dependencies. I personally feel that this has the downside that it splits up these similar information into different places, the dependency is declared somewhere in your code, while the pure/impure declaration is in your Cargo.toml file.

Finally I personally dislike the Cargo.toml level solution as it either severely restrictive by requiring to declare all crates in a certain crate as pure/impure or it would require essentially duplicating all the #[proc_macro_*] attributes from the source code. The first variant is bad as the handling of proc-macros crates is already painful for authors of complex crates due to various deficits. The second variant requires to duplicate information which is error prone as they tend to go out of sync. I obviously cannot speak for any team (as I'm not part of any), so if the cargo team decides that's the way to go they are free to go it. I just wont spend any more time working on this then as it doesn't align with my goals for the outlined rather fundamental limitations.

If a declarative approach is preferred I think I would suggest with the providing the three state (Pure, Impure, DependendOnSomething ) as part of the #[proc_macro_*] attribute. In that way you have it at proc-macro level and you can query it from the compiler/cargo/any other build system after compiling/before running the proc macro. Finally it is at least somewhat near to the side where you declare additional dependencies via the proc_macro::tracked_* functions.

1 Like
29

Do we have a sense of how common it is for a proc-macro to depend on environmental factors that have a significant impact? My assumption is that the vast majority of proc-macros only depend on their token inputs. The ones that depend on undeclared environment factors seems like it should be extremely small, and once the tracked APIs are stabilized, that set should be even smaller.

I'm still not quite clear why cargo would need to have this in a declarative form. If there was something emitted in the dep-info file that says "do not ever cache" or something similar, could it not control the caching behavior just the same?

30

A proc_macro version of include_str! or env! is still "pure" in the sense of what would be indicated by the Cargo.toml setting. What is being said by the flag in the manifest is not that the proc macros are only dependent on the token stream argument. Instead, the communicated property is the lack of any hidden untracked dependencies.

The pest macro reads a grammar file to control the code that it expands to. Today, it emits an include_str! in the code to get the crate to rebuild when the grammar file changes. In the prospective future, it would set the flag that indicates it lacks untracked dependencies and register the file dependency using the tracked_path API or similar.

Saying a macro is trusted to be soundly cacheable does not preclude tracking further dependencies. The two directions (rebuild less/more) are independent.

I don't have any data to back up the claim, but this matches my experience and expectations, as well as the ones that @epage communicated. That external data changing doesn't trigger a rebuild without workarounds (e.g. utilizing include_str!) discourages unnecessarily doing so.

Most use cases that need to do file manipulation get pushed towards using a buildrs step. And after the case of emulating $crate, the majority of attempts to use file access are fundamentally broken concepts (trying to communicate between macro invocations). sqlx is the only case I'm aware of that uses anything other than filesystem or environment access.

I'm not the most fond of the current naming[1] for the tracked_path/tracked_env API, but the design of the API surface is fine and, assuming it fully works, deserves a stabilization push in some form[2].

As far as I understand, it's not strictly necessary; the decision to enter a build into the cache can of course be made after doing the build. However, knowing up front makes planning the build process much more straightforward.

Just as an example, consider a networked distributed build cache a la sccache. Knowing that a package will never be in the shared (robust) cache means that you can skip asking for the package from the cache; you already know it isn't there. Additionally, knowing that a package will get cached means that if one client gets started on building a package, the next client to request the package can decide to wait on receiving access to the soon-available cached build and work on building other packages while waiting instead.

  1. Personally I'd just name them rerun_if[_env]_changed mirroring the naming of the buildrs directives, and expose them at the crate root. The current single-item module and "stutter" in the qualified item path just feels awkward to me. ↩︎

  2. Although I don't know how proc_macro2 is supposed to shim the functionality on older rustc. You can use include_str! and env!, but there's no opportunity for injecting them into the output, let alone in a way known to be grammatically valid. ↩︎

1 Like
31

I would like to see a design that can support for example include_dir - Rust

  • Clearly we need to assume untracked dependencies by default (for backward compatibility reasons).
  • Macros such as include_dir need to be able to declare non-trivial dependencies (rerun if any file is changed or added to a specific directory).
  • Sqlx also has non-trivial environmental selection between database or schema file as I understand it.

As such it seems a runtime API is needed (in the style of build scripts), the question then is "how do we tell rustc / cargo that the dependency information is complete":

  • One option is that as soon as you use the new APIs you promise that you declare everything. Issue: what about the pure case (serde etc) that have no dependencies?
  • You could have a function/attribute to promise complete dependency info.
  • You could have an attribute in Cargo.toml to promise complete dependency info.

It seems to me that what the two "sides" here are discussing is between these last two points, and that you are talking past each other.

1 Like
32

It’s not being actively used for anything as far as I know, but I published a proc macro a few years ago that expands to a fresh UUID on every invocation, mostly as a hack to enable some negative type reasoning on stable. The output of that should obviously never be cached for any reason, even within a single compilation unit.

If there had been a legitimate way for proc macro invocations to communicate with each other, I would probably have gone down that route. That being unavailable, I opted to use stochastic methods instead to avoid collisions.

1 Like
33

Please don't do that! It completely breaks reproducability of the generated executable.

34

FYI there's another similar crate, gensym, which generates random UUIDs to pass to other macros and is used a lot more. Most notably, it is used to generate functions with non-conflicting names in wasm-init and godot-ffi, which AFAIK have no other alternative.

5 Likes
35

Yes, but only because rust doesn't provide suitable alternatives. You can't be surprised when users find inventive solutions for lack of features they need in the language, espeically when there is little progress on fixing those deficiencies.

Such development takes time and is complicated, that is fine. But I don't think the language devs then also get to say "don't use the workaround". Not until the workaround isn't needed any more.

2 Likes
36

And there's another crate which is used a disappointing amount (mostly because ahash pulls it in, one reason to avoid using it); const-random.

2 Likes
37

I think there's still a communication barrier here.

The Cargo.toml flag is not going to be a list of environment variables or file paths. It's going to be something like this:

[proc-macro]
all-inputs-tracked = true

which declares that tracked_path and tracked_env are called at runtime with complete coverage.

1 Like
38

I believe ahash only used const-random in no-std configs.

39

Overall I find it really frustrating how the discussion goes. I agree that there is still a communication barrier here. For me personally it feels like that the participants (primarily @epage ) that argue for a Cargo.toml level approach do not see the full extend of the underlying problem. After all the initial draft of the Pre-RFC contained the following example:

This RFC proposes to provide this information by an explicit function call so that a proc-macro could conditionally opt-in to different caching strategies. Consider sqlx query! macro, that can connect to a database to perform checks for your SQL query. There is an alternative mode that only depends on local file information, enabled by the SQLX_OFFLINE environment variable. This macro could register itself as cacheable as long as the variable is set (by calling including SQLX_OFFLINE in the dependent_env_vars list for the Cache variant) and as long as the relevant offline files did not change (by including the relevant directory into the dependent_path list for the Cache variant). If the environment variable is unset the caching preconditions are invalided and the proc-macro is recompiled. It then connects to the database and can mark itself as DoNotCache as it relies on uncheckable extern state in this case.

I still fail to see:

  • How any declarative approach could handle this complex usecase
  • Any discussion about whether or not something like that should be supported.

Additionally for me personally it's not clear if @epage tries to push their personal opinion or if that's the official position of the Cargo team on this topic. Even for the case that this is the official opinion of the Cargo team I still would expect at least some input from the equally affected compiler team and possibly from the rust-analyzer team as both also need to handle proc macros and might have completely different requirements.

Finally I still do not see a response from @epage where they outlined why the approach proposed in the Pre-RFC wouldn't work for cargo given that cargo already needs to track essentially the same information for std-lib built-in macros. And for the case that a declarative approach is preferred (as hinted by @futile ) I still do not see why a restrictive Cargo.toml level approach would be preferable to declaring this information as part of the #[proc_macro_*] attributes. (My arguments are here (last paragraph))

That written, let my try to address some specific responses (I will just go bottom to top, as that's easier to write):

My previous post already outlines this possibility and it also outlines potential problems with this approach. I honestly do not see how that would address the problems outlined there.

I find this claim rather disappointing. I already listed a few example macros for this above that, so it wouldn't have been that hard to check how popular they are. As that was not done here are explicit number based on the download of the crates

For comparison: serde_derive lists 15M Downloads per month. So yes, this environmental factors dependent macros are used less than pure proc-macros, but I wouldn't go as far as claiming that the "vast majority" of proc-macros only depend on their input tokens if there are crates with 10% of the download number of serde_derive (the most downloaded proc-macros) that do depend on other factors. Additionally: These are only cases that I'm aware of. There are likely cases in other popular crates that I just don't know off. Additionally we must also consider that crate authors currently might decide to use different approaches (build.rs files) to achieve a similar functionality due to proc-macro limitations.

As for the "emulate it via include_str! argument: That's unfortunately often not possible. The listed macros do all load a whole directory, so include_str!() doesn't help there as it wouldn't pick up new files in the directory. (And as far as I understood the tracking issue of the proc_macro::tracked_path API it does currently only allow to register a file path and not a directory path, so it still doesn't support this use-case).

2 Likes
40

Additionally for me personally it's not clear if @epage tries to push their personal opinion or if that's the official position of the Cargo team on this topic. Even for the case that this is the official opinion of the Cargo team I still would expect at least some input from the equally affected compiler team and possibly from the rust-analyzer team as both also need to handle proc macros and might have completely different requirements.

Ed is speaking for himself. His opinion matters greatly as he is knowledgeable and a respected member of the cargo team, but The Cargo Team has not come to any official opinion. We tried to be clear and use "on behalf of the Cargo Team" or similar language when speaking for the group. In this case there was also indication that consensus had not been reached because @ehuss stepped in to express different opinions. Eric is also a knowledgeable and respected member of the cargo team.

I agree that the other teams should weigh in before any official decision is made. When an RFC is posted please make sure they're in the list of assigned teams. This will require their opinions be heard before anything is officially approved. They may be willing to post on a Pre-RFC thread if they're opinion is requested. There is no obligation that they weigh in to any given thread here on internals.

2 Likes