I haven’t been able to find anything like this either on IRLO or among existing RFCs. Please, direct me to the proposal if it already exists.
The problem: cargo test / libtest currently implement coarse-grained test filtering. As far as I understand, these are the options:
- You can include/exclude all the tests (not) marked with
#[ignore]. - You can exclude all the tests marked with
#[should_panic]. - You can filter by name (by substring or full string matching).
There’re cases where it may be desirable to run only some tests. For example, there may be different reasons why the test is ignored by default. Consider this code:
#[test]
#[ignored]
fn im_ignored_because_im_long() { ... }
#[test]
#[ignored]
fn im_ignored_because_I_fail_in_runtime_outside_of_CI() { ... }
When developing locally, you may want to run test that are ignored because they take a lot of time, but not those which are guaranteed to fail. To do this currently, you need to either rely on by-name filtering or weed out unneeded tests with #[cfg]s (which requires recompilation).
Proposal: Add support for test groups in libtest and cargo test. Syntax is bikesheddable, but it should look something like this:
#[test(long)]
#[ignore]
fn im_ignored_because_im_long() { ... }
#[test(ci_only)]
#[ignore]
fn im_ignored_because_I_fail_in_runtime_outside_of_CI() { ... }
and then
cargo test -- --ignored --group=long
or
cargo test -- --enable-group=long
Using the same mechanism, you can disable certain tests if you want. For example, you may want to run long tests by default, but disable them for quick iteration with --disable-group=long.
An alternative I considered: do groups only for ignored tests, like this:
#[test]
#[ignore = "long"]
fn im_ignored_because_im_long() { ... }
and then
cargo t -- --ignored=long
This still solves the “groups of ignored tests” problem, but is kinda less flexible.
What do you think? I could write a formal RFC sometime soon.