So this came up in the context of Dynamic set of tests · Issue #39 · la10736/rstest · GitHub which refers back to Pre-RFC: Dynamic tests
Right now, generating a dynamic set of tests in Rust requires switching to a custom test runner, for example datatest - Rust
Never mind that it requires nightly - any other proposal would likewise always live in nightly for a while - but it seems a pity one has to do this just to generate dynamic tests.
It seems there's an "easy" way to add support for the notion of dynamic tests without committing to a specific mechanism for generating them. That is, something like datatest to provide a specific way to generate the dynamic tests is great, but there's no reason this couldn't work with the default test runner.
To see this, consider the following test file:
#[test]
fn test_it_works() {
let result = 2 + 2;
assert_eq!(result, 4);
}
Right now it is expanded to (tweaked a bit so I could compile it as a standalone binary):
#![feature(test)]
#![feature(core_panic)]
#![feature(rustc_attrs)]
extern crate core;
extern crate test;
#[rustc_test_marker = "maybe_it_works"]
pub const __MAYBE_IT_WORKS: test::TestDescAndFn = test::TestDescAndFn {
desc: test::TestDesc {
name: test::StaticTestName("maybe_it_works"),
ignore: false,
ignore_message: ::core::option::Option::None,
compile_fail: false,
no_run: false,
should_panic: test::ShouldPanic::No,
test_type: test::TestType::IntegrationTest,
},
testfn: test::StaticTestFn(|| test::assert_test_result(maybe_it_works())),
};
fn maybe_it_works() {
let result = 2 + 2;
assert_eq!(result, 4);
}
pub fn main() -> () {
extern crate test;
// An array with hard-wired entries, one per `#[test]`.
test::test_main_static(&[&__MAYBE_IT_WORKS])
}
Now, suppose that the code generated by #[test] used a global vector of TestDescAndFn instead of a hard-wired array. This would allow an API for "insert test into the vector" to generate dynamic tests in whatever way one desires (e.g., using something like datatest or any other method). Add a way to register a #[generate_tests] function, and with a small tweak to the way things are done today we get:
#![feature(test)]
#![feature(core_panic)]
#![feature(rustc_attrs)]
extern crate core;
extern crate test;
// Result of expanding `#[test]` are identical to today:
#[rustc_test_marker = "maybe_it_works"]
pub const __MAYBE_IT_WORKS: test::TestDescAndFn = test::TestDescAndFn {
desc: test::TestDesc {
name: test::StaticTestName("maybe_it_works"),
ignore: false,
ignore_message: ::core::option::Option::None,
compile_fail: false,
no_run: false,
should_panic: test::ShouldPanic::No,
test_type: test::TestType::IntegrationTest,
},
testfn: test::StaticTestFn(|| test::assert_test_result(maybe_it_works())),
};
fn maybe_it_works() {
let result = 2 + 2;
assert_eq!(result, 4);
}
pub fn main() -> () {
extern crate test;
// Generated main program is modified:
// 1. Use a vector instead of an array for the tests.
let mut tests: std::vec::Vec<test::TestDescAndFn> = vec![];
// 2. Generate a push for each static test (annotated by `#[test]`).
// Can still use a simple array if there are no registered generators.
tests.push(__MAYBE_IT_WORKS);
// 3. Invoke any registered test generator (annotated by `#[generate_tests]`, see below).
maybe_it_works_for_some_values(&mut tests);
// 4. Need to call `test_main` instead of `test_main_static`, of course.
test::test_main(&vec![], tests, None);
}
// A function marked as `#[generate_tests]` is not modified, just invoked:
fn maybe_it_works_for_some_values(tests: &mut std::vec::Vec<test::TestDescAndFn>) {
for x in 0..10 {
// Provide a better API here, something like `add_test!(...)`
// with the same flags as today's `#[test]`.
tests.push(
test::TestDescAndFn {
desc: test::TestDesc {
name: test::DynTestName(format!("maybe_it_works_for_{x}")),
ignore: false,
ignore_message: ::core::option::Option::None,
compile_fail: false,
no_run: false,
should_panic: test::ShouldPanic::No,
test_type: test::TestType::IntegrationTest,
},
testfn: test::DynTestFn(Box::new(move || test::assert_test_result(maybe_it_works_for(x)))),
}
);
}
}
// The actual dynamic test (not annotated by `#[test]`).
fn maybe_it_works_for(x: i32) {
let result = x + x;
assert_eq!(result, 2 * x);
}
That is, add #[generate_tests] and add_test! (or whatever names are chosen for them), making dynamic test generation orthogonal to the choice of a test runner.
Some points:
-
If there are no dyamic tests, the generated code would be exactly the same as today - that is, dynamic tests would be a "zero cost abstraction" - only pay for it if/when you actually use it.
-
Does not change any existing types or function interfaces. So zero impact on existing code such as custom harnesses. It would make redundant any work done in them for generating dynamic tests, but they would still continue to work.
-
Either (A) expose a lot of types (
TestDescFnand everything it uses) as in the example above, or, (B) use a global variable, and hide the details in aadd_test!(...)macro using the global variable and taking similar flags to#[test].
Overall it seems like a small change (other than the inevitable bikeshedding for deciding on an exact API).
Is this something that would be favorably looked upon - e.g., worth creating an RFC for, pull requests, etc.? I'm not sure what the proper procedure for such things is.