I’d like to make a slightly more concrete proposal, which I think addresses the issues raised thus far.
Creating a test runner (/harness)
A test runner is any type that implements the following trait:
#[non_exhaustive]
enum TestEvent {
Success { name: String },
Failure { name: String, error: String },
}
trait TestRunner: Default {
type TestComponent;
fn run(self, mpsc::Sender<TestEvent>);
fn register(&mut self, Self::TestComponent);
}
In the case of the current test runner, that implementation may look like:
pub struct DefaultTestRunner(Vec<(String, Box<FnOnce() -> ()>)>);
impl TestRunner for DefaultTestRunner {
type TestComponent = (String, Box<FnOnce() -> ()>);
fn run(self, res: mpsc::Sender<TestEvent>) {
for (test_nm, test_fn) in self.0 {
test_fn(); // catch panics and send TestEvent::Failure
res.send(TestEvent::Success(test_nm));
}
}
fn register(&mut self, test: Self::TestComponent) {
self.0.push(test);
}
}
It may also inclue a number of procedural macros (probably attributes), such as:
fn test_decorator(ecx: &mut ExtCtxt, sp: Span, meta_item: &MetaItem, annotated: Annotatable) -> Vec<Annotatable> {
// NOTE: this macro does not currently exist:
register_test_component!(/* ... */);
let mut items = Vec::new();
items.push(annotated);
items
}
#[plugin_registrar]
pub fn plugin_registrar(reg: &mut Registry) {
reg.register_syntax_extension(
Symbol::intern("test"),
SyntaxExtension::MultiModifier(Box::new(test_decorator))
);
}
The register_test_component macro is the magical thing that would have to be added. It doesn’t technically have to be a macro, but whatever. Specifically, what it does is accept an expression generated at compile time, which it will evaluate in the test main() at runtime and then pass to the chosen test runner’s register method.
Note also that you can register components that aren’t necessarily just a single test. They can be whatever set of types the test runner cares about (suite, block, test-generating function, any of the above, etc.).
Choosing a test runner
To choose a test runner, the user would add, say, test_runner_foo to their dev-dependencies, and then this to their src/lib.rs (or whatever other crate they want to use that test runner in):
#[macro_use]
#[use_test_runner(DefaultTestRunner)]
extern crate test_runner_foo;
Note that this would select the indicated test runner for the entire crate. Since integration tests (files in tests/*.rs) are basically their own crates (they extern crate the crate-under-test (CUT)), they can choose whether or not they want to use the same test runner as the CUT.
What happens at compile time?
If a crate is compiled with cfg(test), the following main is generated:
fn main() {
let mut runner = crate_test_runner!()::default();
// for every expression registered at compile time with register_test_component!:
for expr in registered_test_components!() {
runner.register(expr);
}
let (tx, rx) = mpsc::channel();
let runner = thread::spawn(move || runner.run(tx));
for event in rx {
match event {
/* initially, just mimic current test output
eventually we'll want the ability to forward
to a customizeable formatter. */
}
}
runner.join();
}
What about benchmarks?
A benchmark harness could be implemented the exact same way. You’d add another annotation (#[bench]), make TestComponent an enum of Test and Benchmark, collect them separately, and choose in run which ones to run depending on whether benchmarks or tests were requested.
