[Pre-RFC] Filter out unit tests BEFORE compiling the crate for faster edit-compile-test loops


#1

Background

I have a library that currently has 309 tests (and growing), and (try to) use a edit-compile-test loop while developing, so I constantly call commands like cargo test ^mat::view::row::test:: (from my text editor), which usually end up executing less than 10 tests.

The problem with this approach is that cargo test compiles ALL the tests (but that’s not cargo’s fault), and then executes only the tests that match the regex filter. So my edit-compile-test loop time is around 20+ seconds. Most (90%+) of that time is spent compiling the tests that I don’t want to execute.

Here’s some timing information:

  • cargo build (fresh dependencies, no tests are compiled) 0.77s user 0.06s system 99% cpu 0.829 total
  • cargo test ^$ (fresh dev-dependencies, all test are compiled, no test is executed)
time: 10.912 s  type checking
time: 2.223 s   translation
time: 4.590 s   LLVM passes
19.80s user 0.30s system 104% cpu 19.210 total
  • cargo test ^mat::view::row::test (only execute tests) 0.23s user 0.04s system 99% cpu 0.267 total

If I could get rustc to only compile the tests that I want to execute, then my edit-compile-test time would (probably) be in the order of 2 to 4 seconds. (and I’d be happy!)

Summary

Add a --filter-tests=${regex} flag to rustc that will filter out unit tests from the crate file before compiling it.

Motivation

Faster edit-compile-test loops by making the compiler do less work, rather than making the compiler faster.

(Non) Detailed design

(I don’t have enough knowledge about the compiler internals, to propose a more detailed design)

  • Add a --filter-tests=${regex} flag to rustc, that only works when the --test flag is also present.
  • While parsing, rustc will remove all the #[test] fn() -> () items (the unit tests) that don’t match the ${regex} from the crate file. (This step should run after the module files have been merged with the crate file)
  • Note that all the #[cfg(test)] items (modules, auxiliary functions/structs, etc) are kept. (That way, the remaining unit tests shouldn’t break)
  • The produced binary will still support extra filtering via an argument, like it does today.
  • Bonus: Make cargo test internally use this flag when it receives a regex filter.

Drawbacks

One more compiler flag.

Alternatives

  • Don’t do this, continue having edit-compile-test loops that become extremely slow as the number of unit tests increase.
  • Hope that, in the future, parallel compilation of unit tests will provide some relief to this situation. (Compiling 300+ tests with 8 threads, will still be way slower than compiling 10 tests with one thread)

UPDATE I tried the new parallel codegen feature, but it didn’t help that much in my case, because the LLVM passes phase takes “only” 4 seconds. With 4 threads, that time went down to 2.5 seconds, so my total compiling time went from 20 seconds to 18.5 seconds.


  • Hope that someday we’ll get incremental compilation that will be smart enough to recompile only the modified/affected unit tests.

Unresolved questions

Should the filtering consider #[bench] items?


I don’t have the skills to implement this or provide a more detailed design, the goal of posting this here is:

  • Get someone more familiar with compiler internals to confirm whether this is feasible or not
  • See if there are more people that are affected by these slow edit-compile-test loops (or maybe they have worked around this issue, and I’d love to hear how they did so)
  • See if the community in general thinks this is a good/bad idea

#2

Is this for a Debug build or a Release build ?

I would understand 20s for a Release build, but the speed matters little since it’s not what you would use for an edit-compile-test loop.

However for a Debug build that’s quite jarring. I’ve noticed similar issues with my own builds, to the point that my C++ code actually tends to compile faster than Rust’s (with a dependency on the preprocessor/template heavy Boost), and it is slightly disconcerting, but I assumed I was merely not correctly calling rustc (not deactivating enough stuff).


#3

It’s in debug mode. (FWIW, compiling all the tests in “release” mode takes 41 seconds, whereas compiling the library without tests in release mode takes 700 miliseconds)

Compile time is an area that still needs, it’s getting some and will get more improvements, but what I propose here makes edit-compile-test loops faster by making the compiler do less work rather making the compiler faster.