Pre-RFC: Lightweight test fixtures

#1

Hello

I know there’s a discussion about allowing custom test frameworks to get integrated. But most of the time the default one is just good enough solution ‒ it feels friendly, lightweight and not opinionated.

However, I often find myself writing code like this:

struct TestData {
  /* Something goes in here */
}
impl Default for TestData {
  fn default() -> Self {
    TestData {
      /* Something goes in here */
    }
  }
}

#[test]
fn do_test() {
  let test_data = TestData::default();
  /* do stuff with the data */
}

/* More tests with the TestData */

Basically, this is poor man’s test fixture ‒ it has a setup (the Default) and optionally teardown (Drop). And I had this idea how to make it more comfortable. If a data type implements Default, it could be passed as a parameter to the test function. (more different fixtures could be allowed) The test harness would create the fixture and pass it in. self would be just a special case, so one could write:

#[test]
fn do_test(test_data: &TestData) {
   ...
}

or even:

impl TestData {
    #[test]
    fn do_test(&self) {
        ...
    }
}

To me, it feels quite natural extension of what exists already. It is probably only convenience. Now, my questions are:

  • Does this idea pose some serious problem I overlooked?
  • With the coming (hopefully) of allowing custom test frameworks, is the built-in „sealed“, or would it be possible to extend it?
  • Does anyone but me see a value in such a thing?
  • How hard would it be to implement something like this (provided enough people find it interesting, I write a real RFC about it, it goes through…)? I’d like to know if it would be worth the effort, or if it is just too much work for little gain.
#2

A good way to prove this out would be by implementing it in a procedural macro. You can take this input:

#[fixture_test]
fn do_test(test_data: &TestData) {
    /* ... */
}

and expand it into:

#[test]
fn do_test() {
    let test_data = Default::default();
    run(&test_data);

    fn run(test_data: &TestData) {
        /* ... */
    }
}
#3

Good idea. I wanted an excuse to play with procedural macros anyway :slight_smile:

Still, having opinions from other people if they like it or if they see an actual problem that’d prevent it from working before I spend the time doing it might be valuable.

#4

I used the same pattern and lot and see something that make it more ergonomica would be great.

#5

I definitely find myself with shared setup in various places. I often wind up factoring it into a macro or some such, but it does feel like we could probably do better somehow.

#6

I don’t know if this needs to be in the standard testing suite…

This however gave me an interesting idea… A DefaultStrategy may be interesting for inclusion into https://github.com/altsysrq/proptest and may act as a sort of fixture. Or even simpler:

fn gen_test_data() -> LazyJustFn<(bool, usize)> {
    LazyJustFn::new(|| {
        (false, 42)
    }
}
proptest! {
    #[test]
    fn mytest(ref test_data in gen_test_data()) {
        ...
    }
}

// or possibly:
fn gen_test_data() -> (bool, usize) {
    (false, 42)
}
proptest! {
    #[test]
    fn mytest(ref test_data in gen_test_data) {
        ...
    }
}

I’ll experiment with this today =)

#7

I haven’t used it, but galvanic-test supports fixtures.

Simplified example from the readme:

test_suite! {
    // fixtures are arguments to the tests
    fixture bogus_number() -> i32 { ... }
    test a_test_using_a_fixture(bogus_number) { ... }

    // fixtures with arguments must receive the required values
    fixture input_file(file_name: String, content: String) -> File {...}
    test another_test_using_fixtures(input_file(String::from("my_file"), String::from("The stored number is: 42"))) {
        ...
    }
}
#8

Yes, I tried galvanic-test. It seems reasonably good, but:

  • It is quite young. There are many rough edges, I was fighting a bit.
  • The syntax with macros is not optimal, it feels non-obvious both when writing and reading.
  • It’s good match when you need to go heavy, but not something you want to use if you want to drop just few small tests into your library. More like when you’ve spent the time and wrote a full plan on testing strategy, etc.
#9

Do you recall what the rough edges/fights you had were? Was it mostly error messages from macros, or did things not work in the way that you expected?

#10

It’s some time since I used it. There was some problem with running the same test with multiple inputs while passing the ownership of the values, or something. It was nothing I couldn’t solve, it just felt there should be a simpler way to do that. It might be so that the problem got solved since and if not, it will probably improve over time.

#11

I am very much in favor of this RFC! While supporting custom test frameworks is important, I think it is no less important that the build-in tests are easy to use, and this RFC neatly solves an important usability problem.

I also believe that this is not a poor man’s test fixture: I believe this is the correct approach to test fixtures, which is vastly superior to common setUp/tearDown pattern.

I am also not sure that procedural macros are the right approach to implementation here: I feel there’s a lot of value for this as an out of the box feature of the language.

Questions:

  1. how does this handle parametrized fixtures? What if you don’t want Default, and instead would like to supply some parameters? Perhaps the right answer would be: “create the fixture manually in the function’s body”.

  2. why do_test accepts &TestData and not TestData?

#12

I understood the suggestion for a proc macro as a way to prototype it first and if it looks good enough, go with the RFC and all that.

The basic approach doesn’t. I was thinking about that ‒ it could be made to work, for example by providing a different trait that could iterate through the tests and have wildcard implementation for Default types, for ease of use. But I tried to keep it conservative at first.

I was thinking that all three forms (&, &mut and owned) would be allowed ‒ mostly because I didn’t see a reason to prefer a specific one.

#13

One problem in using Default is that you get one fixture per type.

I’d like to suggest an alternate approach if we want to do this in libtest.

trait FixtureSet {
    fn fixture_set() -> impl Iterator<Item = Self>;
}

impl FixtureSet for TestData {
    fn fixture_set() -> impl Iterator<Item = Self> {
        // make an iterator of one or more test fixtures to run..
    }
}

// Now we write a test:

#[test]
fn do_test(test_data: TestData) {
   ...
}

// and it desugars to:

#[test]
fn do_test() {
    <TestData as FixtureSet>::fixture_set().foreach(test_data);

    fn run(test_data: TestData) {
        /* ... */
    }
}
#14

I too find galvanic limiting. It sounds trivial, but the most annoying part is that a test fixture consists of a single object of a novel class. This proposal has the same problem. I find the JUnit style to be much easier to use. IMHO Rust doesn’t need a new kind of lightweight unit test framework; Cargo’s builtin #[test] feature is good enough. What Rust really needs is a heavy-weight framework, ala Googletest or CXXTest.

#15

That’s why I don’t propose a new one, but extending the existing one.

I’m not sure if this is clear from the description, but it would (naturally) allow passing multiple fixtures to the test, eg:

#[test]
fn test(fixture1: Fixture1, fixture2: Fixture2) {

}
#16

If we take the iterator approach, then you would get the cartesian product of the iterators provided when you have multiple arguments. I think that is a flexible approach that allows you to specify a large number of tests without much work.

#17

To tie this back the custom test framework discussion for a second: custom test frameworks will enable the default test runner that ships with Rust to be moved outside of rust-lang/rust into its own project. This means that we can continue to expand on it and develop it, hopefully at a higher rate than what the current libtest has seen. I think there may be an argument for keeping the default tester somewhat simple, but setup/teardown/fixtures in some form is probably common enough that it’s a good idea to include. Whether we expand libtest in its current form, or wait until it’s been extern’d, is unclear, but in theory the externing process should be unproblematic, so libtest could continue to evolve in its current form until we’re ready to switch.

#18

I tried to implement it as a proc macro. I’m not sure I’m happy with the code (it would need some more work, at least). Furthermore, I don’t think I’m able to do the test as a method (accepting the fixture as &self).

Suggestions how to proceed are welcome. Testing and bugreports too.