Pre-RFC: Dynamic tests


#1

Summary

Add ability to run generated test closures with #[test].

Motivation

We may want to test the same function on multiple test data, while still getting colored result output and automatic parallelism.

An example of this is compiletest inside rustc. It achieves the goal by passing the test Vec on it’s own. While it works well, we need to rewrite the test harness completely and use our own main() for this.

Guide-level explanation

To use dynamic tests, a #[test(dynamic)] attribute should be used, and the fn’s signature needs to be altered:

#![feature(conservative_impl_trait)]
#![feature(test)]
extern crate test;
use test::TestDesc;

#[test(dynamic)]
fn generate_tests() -> impl Iterator<Item=(TestDesc, Box<FnBox<()>>)> {
    // ... generate tests ...
}

#[bench(dynamic)] is also available similarly.

Reference-level explanation

An iterator is used so we can lazy-generate the test cases in future if we refactor. Currently, the test harness requires all testcases to be put in a Vec.

Alternatives, unresolved questions

  • To make this available on stable, there are two obstacles:

    • We need to stabilize impl Trait or Box the Iterator.
    • We need to stabilize libtest or use a ugly tuple (sounds unacceptable).

    Meanwhile, we can just make it nightly-only just like bench does.

  • Is using impl Trait a good idea?

  • Should we use TestDescAndFn? This also implies that we can generate all kind of tests (including benches) using the syntax.


#2

I think you’ll be able to implement that as part of


#3

For what it’s worth, this is already possible using Servo’s fork of libtest as a workaround to run dynamically generated tests even on stable https://crates.io/crates/rustc-test. I’m successfully using it on several projects myself. (Alternatively, if you want to run tests only on nightly, you can use libtest directly, but its API is rather unstable).


#4

This however doesn’t resolve the complexity of replacing the test harness with a custom entrypoint, which is one goal I like to achieve with this RFC.