Let proc macros pass information to the compiler

1

This is a use case we came across in Michael-F-Bryan/include_dir#31, a proc macro for including the entire contents of a directory in your binary (think of it as an extension to include_bytes!()). Basically, if you add an asset to the embedded assets directory, but the file containing the include_dir!() invocation isn’t touched, we won’t re-run the proc macro and see the latest changes.

What are people’s thoughts on giving proc macros a mechanism for giving rustc information about incremental recompilation or to emit warnings? This would be the proc macro analog of cargo:rerun-if-changed=... and cargo:warning=... used in build scripts today.

If other people believe something like this would be useful, I’d be happy to start writing up an RFC.

cc: @ExpHP

2

A similar thing would be useful for pest. Currently, it recommends using include_str to create an unused const str with the grammar contents to introduce a data dependency.

1 Like
3

With my IDE-writer hat on, I am literally horrified by the fact that proc-macros are allowed to do IO at all.

I would much prefer a two-stage solution where a build script does all the IO and writes something like a data.bin file, while a proc-macro does something like include_bytes and work from that.

2 Likes
4

While I see where you're coming from, I find derives that can read files to generate code from much easier to work with compared to having to maintain a separate build script that generates code; if only because either it generates into the source dir which then confuses the distinction between actual source code and generated code or in the target dir which makes it hard to include in your library.

Plus, of course, given language stability I don't think we can retreat from allowing I/O in procedural macros now.

In any case, yes, as the author of Askama I've also definitely wanted this, mostly for the rerun-if-changed effect, though I've also wondered how this would eventually factored into incremental compilation. That is, it would be swell if incremental compilation wouldn't have to recompile all the templates if anything changed. So allowing macros to give the compiler information about what "external" inputs are being used so the compiler can use that information seems like a useful thing to have.

(graphql-client is another example of a crate that reads multiple files from disk for its derive-based procedural macros.)

5

I can understand that. Procedural macros which have side effects that change depending on the environment would be a massive pain to deal with. It also introduces a fair amount of non-determinism and is just asking for "but it worked on my machine" issues.

That said, if you're a game/web developer and want to bundle all your static assets in the executable, having an include_dir!() proc macro is ideal, it's why we have include_bytes!() and include_str!() in the standard library after all. We tried using build scripts in a previous iteration, but it's a massive ergonomics issue and I don't think you'd actually gain anything from an IDE perspective (instead of the proc macro doing IO, you just push the problem to a build script and associated include!()).

Another side of this is providing procedural macros with the ability to emit warnings or tell rustc about other information (e.g. cargo:rustc-link-lib=static=foo). In include_dir!() we want the ability to say "hey, you're trying to include a 200MB file, are you sure you want to do that?". I don't think proc macros have anything more fine grained feedback than a simple success/fail.

6

Quite the opposite, there's a huge difference between how proc macros and build scripts interact with IDEs. The main difference is that build script is executed explicitly on user's request for full compilation, while proc-macros are executed on every change to the source code. Currently, RLS does execute build scripts, but I personally consider this to be a design bug in RLS.

It's illustrative to think about this in terms of LALRPOP, which has huge compile times. If you are editing a grammar.lalrpop, and lalrpop works via build script, no worries, just compile the code once you are done with a grammar. If lalrpop is a proc-macro, then every change to grammar immediately results in calling into lalrpop.

To be clear, I don't have an opinion about which is more important, IDE experience, or convenience of using proc-macros. And I feel like IDE authors will have to just deal with IO-doing proc macros. What I am fearing though is such macros becoming very common in practice, which will place upper bound on what IDEs could do effectively.

8 Likes
7

In today's world, I would think this is a slam-dunk in favor of making things efficient for IDE's. The modern expectation is that a language will have an efficient and helpful IDE to aid in productivity. Without that, Rust cannot be competitive.

2 Likes
8

Two thoughts:

I'd like to pull attention to this thing mentioned by @Michael-F-Bryan:

Another side of this is providing procedural macros with the ability to emit warnings or tell rustc about other information (e.g. cargo:rustc-link-lib=static=foo ). In include_dir!() we want the ability to say “ hey, you’re trying to include a 200MB file, are you sure you want to do that? ”.

Warnings for proc_macros are a great idea that I don't recall ever seeing any discussion about. An RFC to add some sort of diagnostics API to the proc_macro crate might be very nice.

I find the IDE-based arguments here fairly compelling, and for the particular case of include_dir! it makes me wonder if perhaps it is best to go back to being a build.rs helper.

I wonder, what particular advantages were conferred to include_dir by becoming a procedural macro? Perhaps some facets of cargo build scripts could be improved to accomplish similar objectives? (I know this question is inviting discussion that is somewhat off-topic; if necessary, a separate thread could be started)

9

There is already a diagnostics API for procedural macros in nightly.

#![feature(proc_macro_diagnostic)]

use proc_macro::TokenStream;

#[proc_macro]
pub fn my_macro(input: TokenStream) -> TokenStream {
    let first = input.into_iter().next().unwrap();
    first.span().warning("you spelled this wrong").emit();
    TokenStream::new()
}

warning: you spelled this wrong
 --> src/main.rs:4:11
  |
4 | my_macro!(x);
  |           ^
5 Likes
10

I don't know about your world, but it seems to me that you're advocating for the IDE's convenience rather than the user's. I would argue that procedural macros have a superior experience for Rust develpoers exactly because they require less work to move forward -- if that means the IDE developers have to work a bit harder, that seems the right side of a trade-off for me. (Of course we should still try to make this as straightforward as possible for IDE developers, easy to understand for Rust users, and possible to optimize for procedural macro authors.)

1 Like
11

Proc macros are just normal Rust programs: they take a TokenStream as an input and they return a TokenStream as an output. Particularly, they can do anything that any compiled Rust program executed by the current user can do: they can allocate memory, spawn threads, write and read files, open network connections, download and upload anything from / to the internet of hate, use platform intrinsics (e.g. SIMD), etc. and all of that can influence the generated TokenStream.

I think that from an IDE perspective this should actually be very liberating: there is no way for an IDE to tell what a proc macro does so there is no reason to try. Sure an IDE could hardcode what derive(Copy) and the other built in derives do, but trying to go beyond that is probably not worth the effort.

12

So, if the IDE wants to allow auto-completion (or something similar) and the expansion of said macro results in definitions / declarations that can be auto-completed to, and the expansion of the macro is running a potentially arbitrarily long computation, how must the IDE handle this? This is something that needs to be carefully considered; otherwise, as proc-macros proliferate, any IDE assistance could become effectively useless which would be BAD(tm) for the Rust ecosystem IMHO(R).

3 Likes
13

Like all IDEs handle auto-completion: by guessing.

Modern IDEs for modern languages (e.g. C#) ask the compiler-build-system-oracle for "what can the user want to write here" essentially making it somebody else's problem, with the difference that this somebody has better chances of actually answering correctly. In the particular case of proc macros, the compiler would expand them (that is, compile them to machine code and execute them), cache the expansion, and every now and then update that. When an user does the query, the compiler would just load from the cache, which is often a pretty good guess.

If you don't have such an oracle, then I personally would just accept that and call it a day. You guess from the information you have, you accept you don't have information from proc macro expansion, and live with it. At most, one could hardcode support for the built in derives.

I am sure that others will try to do better than that and try to e.g. run rustc to output the macro-expanded code and try to link that back to the source code (which is extremely brittle) or try to re-implement rustc in Java so that they can compile and run proc macros... [0]

I personally don't think any of those efforts are worth it. Perfect auto-completion does not exist, not even in C#, there is only better and worse auto-completion. I can live with auto-completion that lacks info about what's expanded from proc macros, and if I couldn't live without it, the "hacks" that one could employ to obtain it wouldn't probably cut it for me either and my time would be best spent helping build the RLS which is a reasonable solution to that problem.

[0] EDIT: more realistically, an IDE would understand all Rust proc macro ABIs, find the proc macro crate, compile it using cargo, and then try to call it as an executable to expand Rust code, having logic for going from TokenStream to source code and maybe back, reusing rustc libraries, without having to actually build a full compiler, re-implementing LLVM, etc.

Also, enough people have expressed interest into executing proc macros inside a "VM" to be able to limit what they can do (file I/O, network I/O, what they can access, etc.). I could imagine that enough interested people willing to put in the work could actually end up stabilizing something like this that IDEs might also be able to use to expand rust source code.

14

This I believe is where things can break down when Proc-Macros are permitted to perform arbitrarily long computations that use I/O and Network resources (or anything for that matter). If not kept withing reasonable limits, this can make any sort of "On-the-Fly" compilation non-performant to extent that the IDE assistance becomes useless. I just think that the usages and capabilities of proc-macros should consider this and if something can be done with a one-time build.rs script as opposed to a macro, that should be considered.

15

This would be a breaking change because custom derive can do this, is stable, and has been stable for a long time - widely used libraries like diesel rely on being able to fetch a data-base schema from a data-base and generate code from that (IIRC diesel also allows using a build.rs but that doesn't change that we can't break stable code that's working correctly).

If not kept withing reasonable limits, this can make any sort of “On-the-Fly” compilation non-performant to extent that the IDE assistance becomes useless.

If you are modifying code that requires expensive/long running proc macros to continuously be expanded (e.g. typing in the fields of a struct where e.g. serde-derive and similar are used), then I don't think auto-completion should even try to produce suggestions that account for the results of those proc macros (chances are that it can't do this anyways if the struct-code being modified is not in a compilable state). A compiler with incremental compilation and a cache could detect where the changes are happening, and either return cached results from previous invocations, results that ignore the code expanded by the proc macros, or return that current auto-completion results are not available.

Otherwise, the macros just need to be expanded once, which has to happen anyways for compiling the code, and the result of that expansion can be cached and re-used.

1 Like
16

Heh, I feel this is so wrong that my further reply would be rather emotional, off-topic and unstructured, but I hope it still is interesting :slight_smile:

First of all, I really regret using the term IDE. It stands for "Integrated Development Environment", and for some people IDE's are just that: GUI programs that integrate a bunch of different tools under a single UX (in contrast to text editor + shell). For some other people, myself included, IDE means rather "Intelligent Development Environment", a tool which understands the semantics of code, and makes suggestions based on that (in contrast to a text editor, which uses regular expressions to "understand" the code).

I'll try to refrain from "IDE" word from now on, and will use a more clunky but more precise "Code Analyzer". For the purposes of the proc-macros discussion, it doesn't really matter where Code Analyzer lives. Some options are

  • A component in a GUI IDE program, this how most of JetBrains' IDEs work, including IntelliJ Rust.
  • An RPC server, this is how various LSP servers work, and how JetBrains' Rider works (it's a C# IDE with Java/Swing GUI of IntelliJ, and a code analyzer written in C# (internal one, not Roslyn) which talks to GUI via protobuf)
  • As a library, which you could use directly from IDE, wrap into an RPC server or do whatever else you want. This I think is the best way to do it, Dart works like this, and I try to implement the same architecture in libsyntax2 (which I feel I should rename to rust-analyzer?).
  • As a command-line --ide-mode switch to the compiler. This never actually works, despite the fact that it's very pleasant to think "we have a compiler, so let's just plug this into IDE and, boom, done!" :smiley:

My second point, which is rather strong (reminder: this is emotion-driven reply :slight_smile: ), is that if one haven't worked on a largish Java code base in IntelliJ, one has no idea what Code Analyzer support even means. In particular, VS Code + TypeScript is not a good Code Analyzer experience. I don't know about C#, but at least when I used it long time ago (2012-ish), Visual Studio was completely unusable without ReSharper, and my friend who does bits of C# today claims that Roslyn-based backed is much better than VS from long ago, but is still very far behind Rider.

I occasionally see claims that "small talk had IDE environment which is miles ahead of what we have today". I haven't use small talk ( :frowning: :frowning: :frowning: should really do one more ray tracer), but I don't believe that this could be true, and I think this indicates that some people have really fallen behind the state of the art in IDEs. Again, I might be wrong here, because I haven't use small-talk, but I trust Fowler on this one: PostIntelliJ.

With this in mind, I believe that "Perfect auto-completion does not exist" claim is technically correct, but is absolutely wrong. Both technically and actually correct claim would be:

Defect rates in compilers and Code Analyzers are comparable

I surely had cases where Code Analyzer for Java/Kotlin failed to provide completion that was valid. However, I also had cases where Code Analyzer worked just fine, but compiler died due to things like an edge case in type inference, which is fixed in a minor update I don't have, or a stack overflow on a large expressions on windows.

Good Code Analyzers don't guess the meaning of valid code: they run the same analysis as the compiler, with comparable number of bugs. What they do guess are intentions of the user (how to order completion variants) and what semantics to assign to incomplete and broken code to be most useful (like, for an if expr with differently typed branches, report an error but pick the then branch type as type of if and chug along).

When an user does the query, the compiler would just load from the cache, which is often a pretty good guess.

See my example with LALRPOP for why "just" doing such things might result in horrible user-experience.

12 Likes
17

@matklad I agree with everything you said.

You make the claim that:

Defect rates in compilers and Code Analyzers are comparable

but below you precise that this only holds for Good Code Analyzers:

Good Code Analyzers don’t guess the meaning of valid code:

which I think is important, particularly, if you put the bar for the IDE/Code Analyzer experience at the level of Java:

if one haven’t worked on a largish Java code base in IntelliJ, one has no idea what Code Analyzer support even means.

From an IDE POV, Java is a really nice language to work with. OTOH, stable Rust has declarative macros, infinitely powerful proc macros in the form of custom derive, ... Unstable Rust has const fn which already can drive type inference, and which if many people get their way, will be able to allocate memory, be usable in where clauses driving which trait implementations are available, etc.

Setting the bar at the same level for both Java and Rust is a good goal to have, but because Rust is not Java, it would appear to me that the amount of work to create a Good Code Analyzer for Rust would be much much higher than for Java.

I fully agree with you that a Good Code Analyzer for Rust would need to support all of that. I just don't think that it will be feasible for a Code Analyzer to become Good without re-using parts of the compiler, not only libsyntax, but at least miri. Also, for supporting proc macros, one needs to compile them (and their dependencies), and execute them, so at some point the Code Analyzer is going to have to directly call the compiler and build system to achieve that.

As a library, which you could use directly from IDE, wrap into an RPC server or do whatever else you want.

I thought that this was already how the RLS currently was implemented? That is, that it called libraries like libsyntax ? I also agree that this is the best approach.

Also, I might misunderstood @gbutler wrong, but I don't think that IDEs / Code Analyzers can do much today to become Good Code Analyzers. Shouldn't they offer no auto-completion at all? I don't think so, I think it is ok for auto completion to not be perfect until a Good Code Analyzer for Rust exists, what alternative do we have?

18

So, there's this thing called const fn :wink:

A proc macro which is const fn can't possibly do any sort of IO -- so a code analyzer should take advantage of this to avoid guessing and instead knowing that executing the proc macro won't brick the system.

Oh yes; another next-level example is interactive theorem provers and how cool Idris / Agda is wrt. case splitting, auto completing, normalizing, giving you goals, and the types of bindings which you have in scope. This is arguably more advanced than what you get with Java IDEs.

This might be an interesting read: Extensible Type-Directed Editing.

19

There's something fundamentally wrong with requirement to run compiler to get navigation/completion/etc.

Large C++ (and Rust, if you include rustc) projects (which need a Code Analyzer most, compared to small projects) I worked with often had their own bespoke build systems that required preliminary configuration, required options to pass, generated code from DSLs, lacked the "just build" command, were possibly too slow to invoke them automatically. And without that build system you cannot run the build or invoke compiler.

In this case tools using heuristics like racer, VSCode C++ plugins, ctags, whatever, more or less work and give acceptable results.
At the same time RLS that needs to run the build just plain gives up on rustc codebase.

2 Likes
20

It’s true that Rust is a more complicated language than Java, but it doesn’t have too horrible parts. Stuff like complicated type-inference, proc-macros and const fn are not fundamentally problematic: there’s “just” more stuff to implement, and more requirements for clever on-demandifying and caching. In particular, compile-time code-gen itself is not problematic, as long as it is a pure function with relatively isolated inputs.

I’d list other things in Rust as fundamentally hard for Code Analyzers:

  • macros 1.0, because they require traversing all files in the order of mod foo; declarations,
  • global nature of trait search: when resolving x.foo() you can easily say where the type of x is, and where the trait for foo method is, but finding the impl can be very hard, because it even can be hidden inside some local function.
  • proc macros doing IO: IO makes caching and on-demand harder, it makes overall performance slower, and it’s architecturally difficult to fit into Code Analyzer, because it shouldn’t do any IO by itself.

That said, there are features of the Rust language, which makes it easy to write a Code Analyzer

  • Rust is the best existing language to write a Code Analyzer in
  • rustc already has pretty-cool tech for on-deman analysis
  • and, most importantly, Rust has a project model which is the perfect fit for Code Analyzers.

The last point is worth elaborating (which will also be a response to @petrochenkov comment).

There’s a build system and a project model, and these are different things. Build system is a sequence of arcane shell invocations which somehow produce project artifacts. A project model is a logical model of dependencies between various source files. For each language, there are many different build systems (each sufficiently large project has a unique one), but only a single project model.

For Java, the model is classpath: the set of base directories with sources and compiled jars. C++, with it’s preprocessor, does not really have a project model better than “a sequence of invocations of clag++, with all flags, cwd and env vars”.

And Rust, Rust does not have a single global namespace of symbols, it has crates, and this is awesome, because it is a very precisely defined project model: a directed acyclic graph of anonymous crates, where edges are marked by extern crate names and cfg flags. It is much better than a classpath, because you know precisely when two crates are independent, so you can be much more aggresive with caching and indexing.

Now, the design bug in RLS is that it doesn’t work with project model, it works with a specific build system, and it makes it slow to incorporate changes and brittle. In contrast, IntelliJ Rust works with project model, and, while it’s not exactly fast, it can work with rust-lang/rust code base, it can do completions, it can infer types: https://www.youtube.com/watch?v=8xoTR0d6R_0&feature=youtu.be

As another point for “you can have a horrendous build system and precise Code Analyzer”, consider IntelliJ itself. It is a huge project, and I think it is build with every tool you can use to build Java. I definitely saw Ant, Maven, Gant, Gradle and JPS in various parts of it. However, IntelliJ is developed in IntelliJ and, naturally, all code insight features just work: IDEA knows little about it’s build system, it only has a project model which is a set of XML files describing classpaths of various modules. Now, syncing those XML files and build-system is a pain. However in Rust a similar task would be much simpler: it has a stronger project model and it has a mostly one build system (Cargo). For the “rustc in IntelliJ” video, I haven’t done anything special, I’ve just opened rust/src folder in IDEA, which then asked cargo metadata about project info.

EDIT: @Michael-F-Bryan sorry for derailing a thread, guess I’ll head to 2019 Strategy for Rustc and the RLS :slight_smile:

9 Likes