Incremental test runner

[eggyal]

I've been thinking a little about how one could build an incremental test runner for Rust (i.e. one that determines precisely which tests have been affected by any updates since the last test run, in order to re-run only those tests—if this happens automatically in the background as one types, with the results conveniently displayed inline, it can be a transformative experience for the developer: I've seen it dramatically improve productivity in other languages, for example with Wallaby.js or Visual Studio Live Unit Testing).

This thought process began some months back with a short chat on Zulip, and then more recently with a Rust Analyzer GitHub issue. However, because I neither want to cut across any existing efforts in this space nor start proposing completely unacceptable changes, I'm raising this here to check whether this is a project I can/should pursue—and, if so, to sound out whether there are any team members who'd be happy to oversee my work?

I've been experimenting a bit so far, and propose building something along the following lines (obviously gated behind an unstable flag for now):

1. Incremental compilation query identifies all InstanceDefs that have changed since previous compilation;

2. The MIR of each is visited to determine precisely which BasicBlocks have changed (e.g. compare stable hashes of [StatementKind] and TerminatorKind—but omitting from the hash any BasicBlocks in the terminator, instead recursing on them; thus ignoring changes to SourceInfo or block indexing);

3. Building upon the awesome work of both @bjorn3 on the Cranelift backend and @richkadel on source-based code coverage, create a test runner to launch tests in cg_clif JIT mode with a custom profiling runtime that receives the instrumented counter incrementation calls: the coverage map generated by the cg_clif backend would map counters not only to CodeRegions but also to MIR BasicBlocks;

4. These instrumented test runs enable construction of maps between tests and BasicBlocks (in each direction) and from CodeRegions to tests;

5. On the next compilation, the set of changed BasicBlocks (from step 2 above) can be used as keys in the map (from step 4 above) to yield the set of tests to be re-run;

6. Re-running a test clears its association with all its previously associated BasicBlocks (hence the need for bidirectional mapping in step 4 above); and

7. Using the final map generated in step 4 above, the results of all covering tests can then be reported by source code region.

A further useful addition would be to obtain the stack trace of failing tests in order to highlight code regions that lie on the failure path.

I'd like to confirm that the above is both a viable and sensible approach, as despite my best research I fully appreciate that I don't have a perfect understanding of all the various pieces involved. Is there something unworkable with this proposal, whether it be glaringly obvious or exceedingly subtle?

One thing I'm particularly unsure of is whether the StableHashes discussed above are already stored in the incremental compilation cache (I guess not, because they are not produced by queries) and, if not, can they be? If not, comparing a previous compilation's BasicBlock will require additional context from its compilation: is that available? (In the very distant future, this problem might be avoided by running a hot-swapping JIT compiler/test-runner as a long-running background process; such process could keep context from the previous compilation around in-memory).

I can also see that the proposed approach would require changes to some existing internal data structures, but I think mostly around coverage map generation (e.g. so that the list of BasicBlocks covered by each counter can be passed to the backend, in addition to their CodeRegions). Will this require deeper thought/approval, or does the fact that source-based code coverage is still an unstable feature make this easier?

Finally, I don't know whether this sort of project amounts to something that will require an FCP issue, a MCP or even an RFC? Would definitely appreciate pointers on the process I should follow!