Background
I have a library that currently has 309 tests (and growing), and (try to) use a edit-compile-test loop while developing, so I constantly call commands like cargo test ^mat::view::row::test:: (from my text editor), which usually end up executing less than 10 tests.
The problem with this approach is that cargo test compiles ALL the tests (but that’s not cargo’s fault), and then executes only the tests that match the regex filter. So my edit-compile-test loop time is around 20+ seconds. Most (90%+) of that time is spent compiling the tests that I don’t want to execute.
Here’s some timing information:
-
cargo build(fresh dependencies, no tests are compiled)0.77s user 0.06s system 99% cpu 0.829 total -
cargo test ^$(fresh dev-dependencies, all test are compiled, no test is executed)
time: 10.912 s type checking
time: 2.223 s translation
time: 4.590 s LLVM passes
19.80s user 0.30s system 104% cpu 19.210 total
-
cargo test ^mat::view::row::test(only execute tests)0.23s user 0.04s system 99% cpu 0.267 total
If I could get rustc to only compile the tests that I want to execute, then my edit-compile-test time would (probably) be in the order of 2 to 4 seconds. (and I’d be happy!)
Summary
Add a --filter-tests=${regex} flag to rustc that will filter out unit tests from the crate file before compiling it.
Motivation
Faster edit-compile-test loops by making the compiler do less work, rather than making the compiler faster.
(Non) Detailed design
(I don’t have enough knowledge about the compiler internals, to propose a more detailed design)
- Add a
--filter-tests=${regex}flag torustc, that only works when the--testflag is also present. - While parsing,
rustcwill remove all the#[test] fn() -> ()items (the unit tests) that don’t match the${regex}from the crate file. (This step should run after the module files have been merged with the crate file) - Note that all the
#[cfg(test)]items (modules, auxiliary functions/structs, etc) are kept. (That way, the remaining unit tests shouldn’t break) - The produced binary will still support extra filtering via an argument, like it does today.
- Bonus: Make
cargo testinternally use this flag when it receives a regex filter.
Drawbacks
One more compiler flag.
Alternatives
- Don’t do this, continue having edit-compile-test loops that become extremely slow as the number of unit tests increase.
- Hope that, in the future, parallel compilation of unit tests will provide some relief to this situation. (Compiling 300+ tests with 8 threads, will still be way slower than compiling 10 tests with one thread)
UPDATE I tried the new parallel codegen feature, but it didn’t help that much in my case, because the LLVM passes phase takes “only” 4 seconds. With 4 threads, that time went down to 2.5 seconds, so my total compiling time went from 20 seconds to 18.5 seconds.
- Hope that someday we’ll get incremental compilation that will be smart enough to recompile only the modified/affected unit tests.
Unresolved questions
Should the filtering consider #[bench] items?
I don’t have the skills to implement this or provide a more detailed design, the goal of posting this here is:
- Get someone more familiar with compiler internals to confirm whether this is feasible or not
- See if there are more people that are affected by these slow edit-compile-test loops (or maybe they have worked around this issue, and I’d love to hear how they did so)
- See if the community in general thinks this is a good/bad idea