I've been thinking a little about how one could build an incremental test runner for Rust (i.e. one that determines precisely which tests have been affected by any updates since the last test run, in order to re-run only those tests—if this happens automatically in the background as one types, with the results conveniently displayed inline, it can be a transformative experience for the developer: I've seen it dramatically improve productivity in other languages, for example with Wallaby.js or Visual Studio Live Unit Testing).
This thought process began some months back with a short chat on Zulip, and then more recently with a Rust Analyzer GitHub issue. However, because I neither want to cut across any existing efforts in this space nor start proposing completely unacceptable changes, I'm raising this here to check whether this is a project I can/should pursue—and, if so, to sound out whether there are any team members who'd be happy to oversee my work?
I've been experimenting a bit so far, and propose building something along the following lines (obviously gated behind an unstable flag for now):
Incremental compilation query identifies all
InstanceDefs that have changed since previous compilation;
The MIR of each is visited to determine precisely which
BasicBlocks have changed (e.g. compare stable hashes of
TerminatorKind—but omitting from the hash any
BasicBlocks in the terminator, instead recursing on them; thus ignoring changes to
SourceInfoor block indexing);
Building upon the awesome work of both @bjorn3 on the Cranelift backend and @richkadel on source-based code coverage, create a test runner to launch tests in cg_clif JIT mode with a custom profiling runtime that receives the instrumented counter incrementation calls: the coverage map generated by the cg_clif backend would map counters not only to
CodeRegions but also to MIR
These instrumented test runs enable construction of maps between tests and
BasicBlocks (in each direction) and from
CodeRegions to tests;
On the next compilation, the set of changed
BasicBlocks (from step 2 above) can be used as keys in the map (from step 4 above) to yield the set of tests to be re-run;
Re-running a test clears its association with all its previously associated
BasicBlocks (hence the need for bidirectional mapping in step 4 above); and
Using the final map generated in step 4 above, the results of all covering tests can then be reported by source code region.
A further useful addition would be to obtain the stack trace of failing tests in order to highlight code regions that lie on the failure path.
I'd like to confirm that the above is both a viable and sensible approach, as despite my best research I fully appreciate that I don't have a perfect understanding of all the various pieces involved. Is there something unworkable with this proposal, whether it be glaringly obvious or exceedingly subtle?
One thing I'm particularly unsure of is whether the
StableHashes discussed above are already stored in the incremental compilation cache (I guess not, because they are not produced by queries) and, if not, can they be? If not, comparing a previous compilation's
BasicBlock will require additional context from its compilation: is that available? (In the very distant future, this problem might be avoided by running a hot-swapping JIT compiler/test-runner as a long-running background process; such process could keep context from the previous compilation around in-memory).
I can also see that the proposed approach would require changes to some existing internal data structures, but I think mostly around coverage map generation (e.g. so that the list of
BasicBlocks covered by each counter can be passed to the backend, in addition to their
CodeRegions). Will this require deeper thought/approval, or does the fact that source-based code coverage is still an unstable feature make this easier?
Finally, I don't know whether this sort of project amounts to something that will require an FCP issue, a MCP or even an RFC? Would definitely appreciate pointers on the process I should follow!