Skippable Tests

language design
1

I have often been annoyed by not being able to skip tests at runtime.

Turns out I'm not the only one, there was a pre-RFC and related discussion a few years ago: Pre-rfc: Skippable tests

I wonder if @mathstuf can comment on why this effort died out, and if anyone is interested in resurrecting it.

2

I think skippable test would be very nice and I feel like it would be even better if you could group test like and put #[test(test_group)] instead of #[cfg(test)]:

fn add(a: usize, b: usize) -> usize { a + b }
fn minus(a: usize, b: usize) -> isize { a - b }

#[test(add_test, unit_test)]
mod add_test { /* … */ }

#[test(minus_test, unit_test)]
mod minus_test { /* … */ }

#[test(random_test)]
mod random_test { /* … */ }

so you can run:

  • cargo test --skip add_test
  • cargo test --only minus_test
  • cargo test --only add_test minus_test
  • cargo test --skip unit_test
1 Like
3

I have an implementation that works for #[test] using the "call a function before calling the test" mechanism. This doesn't compose well as if you have 2 reasons to skip, you need a new target function to combine them. But maybe multiple #[skip_if] attributes makes sense? Other than that, the bigger questions of how to actually land it were left unresolved:

  • how to actually indicate "this skipped"?
  • how to do it for tests/*.rs?

The latter definitely needs a magic return code to indicate this. If that exists, then we can "just" do the same for in-crate #[test]. But both of these lose the "why was it skipped" logic.

I definitely like the "magic return code" for composability, but the lack of insight as to why it was skipped is unfortunate. It also means that:

  • libtest either needs to expose a constant for it (but is basically perma-unstable)
  • everyone hard-codes it
  • std provides something for it (which feels weird)

IIRC, I stalled out a bit on being told that the "new" libtest custom framework supposed to be covering this, but I've not seen (nor pushed) progress on that front.

4

I think @epage has been making plans for libtest improvements?

5

The rough plan (informal agreement with libs-api over a year ago, no conversations in testing-devex yet) is to freeze libtest roughly as it is, maybe removing functionality where possible (unstable features, features without strict compatibility, etc) and instead put the focus on custom test harnesses.

I documented my thoughts and the libs-api discussion on my blog and the various efforts are being tracked in the testing-devex repo

4 Likes
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.