Past, present, and future for Rust testing

[quodlibetor]

I am a huge fan of py.test framework, which uses a better approach for test fixtures than JUnit-style setup/teardown.

I strongly agree with this. py.test “fixtures” are basically dependency injection managed by the py.test runner. They’re very similar to Rocket’s request guards, except for how they’re defined.

A direct translation of pytest’s fixtures might look like:

#[cfg(test)]
mod test {
    #[test_fixture]
    fn simple_data() -> String {
        "A fixture that returns simple data and is called each time it's requested".into()
    }

    #[test_fixture]
    fn complex_data(simple_data: String) -> String {
        simple_data.to_kebab_case()
    }

    // This uses the names of functions that the test_fixture attribute was applied to
    #[test]
    fn uses_data(simple_data: String, complex_data: String) {
        assert!(simple_data.starts_with(&"A fixture"));
        // you would never compare fixtures to each other in a test, but for our purposes
        assert_eq!(simple_data.to_kebab_case(), complex_data);
    }

    // pytest has a built-in `request` fixture that allows you to do things after tests finish,
    // this is how you would do a setup/teardown pair.
    //
    // autouse means the same thing as a #[begin] would mean in the current
    // proposal
    #[test_fixture(autouse="true")]
    fn begin(request: ::std::test::TestRequest) {
        setup_database(env::get("DB_URL"));
        request.register_finalizer(|| {
            reset_database(env::get("DB_URL"))
        })
    }
}

A more rusty solution might look more like Rocket’s request guards (based on the types of arguments rather their names), but it’d probably be nice to have something like a test_fixture macro to reduce boilerplate even in that case.

The big advantage of pytest fixtures, in my experience:

• they encourage clarity in test design where each test states what it requires
• they’re efficient: you only end up actually doing the required work to run each test, without requiring crazy refactoring to make things more efficient.
• DI is just super nice

[Centril]

Nice work summarizing things.

There seems to be a heavy focus here on unit testing, and while that is nice, discussion regarding the needs of property based testing (PBT) libraries seems lacking. Currently, the two larger libraries providing PBT functionality are the crates proptest and quickcheck. These may be helped by having a notion of “number of tests passed” (which is different from number of #[test]s passed).

I want Rust to have excellent support for PBT surpassing even Haskell and Erlang in the future.

[quodlibetor]

I realized my previous comment could be read as me not wanting any progress unless it looks like py.test. mostly I was hoping to demonstrate (without requiring folks to read through some Python libs’ docs) that there are reasonable alternatives to the xUnit standard. (And maybe to inspire some enterprising developer to port pytest to rust to see if it’s possible.)

That said, something maybe more useful to the discussion:

• I would prefer if there weren’t any required attributes (no test, nor setup/teardown)
• That would mean test-lib authors would be free to write their own collectors (e.g. test attribute macros) and runners. Which would also require that the built-in proc macro system was powerful enough to do that.
• Given that, it would be fine for the current libtest to grow more opinionated (xUnit, Catch C++, pytest, whatever)
• To bootstrap an ecosystem it might be nice for the libtest test runner to depend on itself as a library, so that any default collection strategies (test, begin/end) can be picked up and enhanced by other test libs.
• I really like the direction the original summary went in, clearly defined inputs, outputs, and a way to define your test runner as a dev-dependency, combined with proc macros that can keep state across calls seems like all you really need.
• Again, to reiterate, I would love for someone to port py.test to rust, but that isn’t intended as stop energy towards this proposal.

I’m extremely excited to see progress in this area, thanks for all the work!

[gnzlbg]

Nice write up!

One question: by setup/teardown do you also mean process setup/teardown? It is not 100% clear to me whether this is what you mean

One thought, a lot of people seem to only want to customize the output format, so maybe we can have an API for plugin in a different test or bench runners but they all produce JSON, and we have a different API for pluggin in a formatter (so the JSON formatter does nothing, but we can have a formatter to human readable text that transform JSON to the output we have today). That way, those who want a specific let’s say XML output for some CI system, or maybe even a different type of JSON for an IDE, only have to plug in a relatively small component, and that component would work with all test runners that users might want to use.

This might be overengineering, but it at least makes sense to me that “how to run test” and “how to display test results” are two different orthogonal problems.

[matthieum]

@matklad It seems to me that the py.test “features” could be used not only for DI, but also to inject different values (for parameterization) or am I mistaken?

Is it possible in py.test to get something like:

 #[test_fixture]
 fn make_length(index: usize) -> usize {
     1usize << index
 }

 #[test(make_length(0..4))]
 fn test_lengths(make_length: usize) { ... }

And have test_lengths be invoked with multiple lengths?

Could it possibly be adapted to type parameterization?

 #[test_types]
 type Unsigned = (u8, u16, u32, u64, u128);

 #[test_fixture]
 fn make_type<T: Default>() -> T { T::default() }

 #[test(make_type(Unsigned))]
 fn test_type<T>(t: T) { ... }

[jan_hudec]

What’s missing is some way to find all tests in a crate

THIS. The macro #[test] is special, but this is really the only bit that can’t be reproduced with macros 2.0. So if we fill in that bit, we can go wild prototyping better test suites as crates without having to touch the compiler.

and generate code in fn main() {…} based on that

Actually, it would be simpler to create a registry that would be accessible at runtime. That way there would be no need to define dependencies between macros and running them in multiple passes. The compiler would put the items aside, the linker would collect them and the test runner would simply iterate an array.

I would suggest something like this:

First a registry array would be defined. Something like:

#[registry]
static test_functions: &[&fn()];

and then each function would be annotated like:

#[register_to(test_functions)]
fn test_function() { … }

And the compiler and linker would conspire to simply make the test_functions be an array of all the appropriately annotated test functions. A code in main would then simply iterate that array.

Now in crate this should be simple to implement as the Rust compiler does have all of the information at some point. But

This crate-global registry mechanism

I would actually see some uses for this being cross-crate global. Even that can be done. On ELF platforms it can be done by putting the array members (references to the annotated items) to a special section and having the linker collect them. And on platforms that don’t have special sections, it can still be done using the mechanism provided for C++ static constructors, i.e. there would be a bit of code before main that would collect the pointers. It would mean some code would run before main, but it would be implementation detail hidden in the standard library, so it should still not cause the sort of trouble the fully general mechanism causes for C++.

I even think that the mechanism could be prototyped currently with macros and the cpp crate.

---

Note that this tool would also be reused for things like the output formatters. The

#[test_runner]
static RUNNER: MyTestRunner = MyTestRunner;

could similarly desugar to #[register_to(test_runners)] and then the main would look in the test_runners array and pick one by comparing something in it to a command-line option, falling back to either first one or default. If the mechanism was global (i.e. if the attribute permitted qualified name like #[register_to(::test::runners)]), merely adding the formatter crate to Cargo.toml would make the formatter available for selecting.

[SimonSapin]

Actually, it would be simpler to create a registry that would be accessible at runtime.

That requires defining some data structure for this runtime access. How much metadata should be in there? &[fn()] is not enough because you want metadata to support attributes like #[should_panic(expected = "some message")], or just the name of the function. https://github.com/rust-lang/rust/blob/1.22.1/src/libtest/lib.rs#L209-L217 is somewhat complex and has changed over time, stabilizing it would be tricky.

This is why I mentioned generating code: you can use the usual proc macro APIs to inspect attributes and other info about the crate’s items, then generate whatever data structure is appropriate for a given test harness.

[jan_hudec]

Well, an alternate harness that would want different metadata would have to process the attributes itself anyway, no?

So it shouldn’t really matter whether it does so for each item separately or at one place after all are collected. And I think separately it would be easier to implement. I can also imagine that being completely global, for which I can imagine other uses like registering extensions (one crate would define array of factories to which other crates could register—if that happens somewhere deep in the dependency chain it is better if the final user does not have to register them manually at the start of main).

Maybe later an extensible metadata format could be developed so the extended harnesses could reuse parts of the standard one, but for the first experiments I don’t think it’s big obstacle.

[jan_hudec]

Thinking about it more, it would also be possible—and probably simplest—to use kind of “inverted objects” for this. That is the #[test] macro would register the test method with its name and the #[should_panic(expected = "some message")] would register separate record to a separate registry and then the runner would match them up by the function poitner or something. That would allow implementing—and reusing—the individual attributes independently.

[killercup]

Sorry to crash this thread – but what about doc tests? Should they be able to use custom test runners as well in some capacity? (relevant tweet)

[quodlibetor]

It seems to me that the py.test “features” could be used not only for DI, but also to inject different values (for parameterization) or am I mistaken?

Yes py.test’s fixtures support parameterization. The feature in pytest has a typo/uses a different languages spelling (I’m not sure which), so the method is called “parametrize”. It’s extremely convenient, and uses a python-equivalent syntax to what you suggest.

FWIW your idea of type parameterization is one of the things I what I was thinking of when I described using something like rocket’s request guards as a possible implementation direction. If nothing else, rocket proves that the concept is already implementable on nightly.

[mindsbackyard]

I’d agree that having type parametrization for tests would be immensely useful, e.g., like Theories which combine type and value parameterization. I see a major use case for libraries exposing traits. The library authors could also expose a set of type-parameterized tests which assert invariants on the implementations of the library’s users (e.g. if I add something to a list, I should also be able to retrieve it).

Just fyi, simple test fixtures and parameterization can be emulated in stable Rust (too some point) with macros (sry, for the shameless self-plug, but I think it illustrates what could be done with the current state of affairs). Though proper support by a test harness API could be far more useful, easier to develop, and maintain.

[bheisler]

As the maintainer of Criterion.rs, I’d like to add my thumbs-up to creating a stable benchmark-harness API. I have no idea what it should look like at this point (I suspect any serious candidate will require extensions to the macro system) but I want to make a few comments based on that experience. Some of these comments will apply to test harnesses as well.

The current way to use different benchmark harnesses is unsatisfactory for a few reasons.

First, it increases the burden on the user in that they need to separately disable the standard harness for each benchmark executable before they can use a different one, and they need to either write their own main function or use a macro from the benchmark library to do the same. I’m not sure how this burden can be reduced but it’s something we should think about.

Second, it appears that it’s only possible to replace the benchmark harness for benchmarks defined in the benches/ directory - not those defined in the src/ directory. Even if there is a way to disable the standard harness for the default benchmark executable, cargo only generates one executable that contains tests and benchmarks, so doing so would presumably disable tests. I’ve already had a user raise an issue requesting support for benchmarks in src/, as the user wanted to benchmark crate-private code which wasn’t available to benchmarks in benches/ (which are considered to be part of a separate crate). Cargo should build separate executables for tests and benchmarks and make it possible to change the harness for benchmarks in src/.

A third point is that some users will probably want to use different benchmark harnesses for different benchmark executables. If nothing else, cargo bench runs the src/ test executable as well as the benchmark one. There’s only one place to provide command-line arguments to these executables, and the standard harness panics (halting the cargo bench run) when it sees an argument it doesn’t recognize. That means I can’t define additional command-line options for Criterion-based benchmark executables without breaking cargo bench.

[jonhoo]

Thanks to everyone who has chimed in already; all these suggestions are super helpful! After spending some time digesting all the posts so far, and ingesting holiday food, I now have some more refined ideas for what we should think about for libtest 2.0. I’ll try to summarize so you all have an opportunity to disagree

There are many ways to design a testing framework — see in particular relevant Catch C++ and pytest examples for particularly good ones that have been highlighted elsewhere in this thread. However, reading through this thread again, and the various issues and PRs linked in the original post, I have realized that what we really want is solid infrastructure for implementing a test harness (so that a Catch C++ or pytest clone could be implemented in, and integrated with, Rust). I believe that is what @nrc was hinting at in his first response when he said

this model does not support the division between test harness and test framework, afaict

In particular, we should be thinking about what would be needed to facilitate someone writing their own test driver, and having it integrate nicely with cargo test and the like. I’d like to echo my original post here, and say that such an integration requires two distinct components: one for running/declaring tests, and one for collecting test results. How the tests are written and organized should be up to a crate’s chosen test driver.

For example, we could/should provide a default Rust test driver (shipped with rustup), which conforms pretty much exactly to what libtest has today (i.e., #[test] annotations on FnOnce() -> (), running all tests in a flat namespace, supporting the same runtime flags as today’s libtest). Users could then write new testing frameworks with more sophisticated test declarations (e.g., test suites, BDD, PBT, etc.), arrangement (e.g., setup/teardown, fixtures, parameterization, serial execution), better runtime flags (e.g., regex test name matching), etc.

More concretely, I believe libtest 2.0 should consist of two components: a test driver, and a test output formatter. The former is chosen by the crate by declaring it in dev-dependencies, and setting it globally inside the crate using something akin to the global system allocator (except it would be crate-local). The user cannot change the test driver without modifying the source, partially because the test driver defines how tests are declared and specified. The latter is chosen by the user invoking cargo test, and should work for any test driver the crate may have chosen. The exact mechanism for choosing the formatter isn’t clear to me yet, but changing it should not require source changes, nor changes to Cargo.toml, just changes in the user’s environment. Ideally it should be possible to change with a runtime flag to the test binary (probably --format), but it would likely be acceptable for a recompile to be needed if the formatter is changed.

I believe a concrete plan for making progress would need to answer the following questions:

• For test drivers:
  • How does the crate author select a test driver? I propose something akin to selecting the global system allocator as outlined in the original post.
  • How does the chosen test driver interact with the test binary’s main? I think this can be absorbed into the trait we require the chosen test driver’s “main” struct to implement.
  • How does a test driver enable users to declare tests? This is the trickiest point in my mind. Procedural macros alone aren’t sufficient (afaict), since #[test] couldn’t be implemented solely using those. In particular, the missing piece is that (again, afaict) a custom attribute macro cannot generate identifiers that are discoverable at runtime. This is similar to why Rocket cannot automatically register routes just from request guards, but instead require them to be explicitly named when mounting (see what the route attributes generate here). @jan_hudec aired the idea of a “registry”, which could work if we carefully decide what information is included in the registry, but I’d be in favor of a slightly more flexible model where a macro has a way of declaring code that should be run in a resulting test binary’s main?
• For test result collectors:
  • How does the collector receive test results from the driver? I continue to believe that an event stream as highlighted in the original post is the way to go here. Having a crate-local (thread-safe?) on_test_event function seems like a decent option here. It could then either dispatch to the chosen formatter either with dynamic linking, some kind of lookup among multiple compiled-in implementations, or something else.
  • How does the collector receive flags and other options? This one is also a little tricky given that the command-line parameters are shared with the test driver. How do we deal with overlapping options/flags? Positional arguments allowed by both? Giving them both access to a &[&'static str] seems like punting what will certainly become an issue.

With the questions above resolved, we should be in a place where pytest, Catch C++, xUnit, and the like can all be implemented externally from libtest itself, while still integrating nicely with both the existing tooling, and orthogonally to the much-requested custom test output formats.

[vorner]

Hello

I like one think about the current situation. It is simple and trivial to use. There was Catch mentioned and its sections. Sometimes, you need to read the code many times to know which different test scenarios will run in the end (especially if you nest these sections), and with which values. Also, when you look at tests in different C++ projects, each one is different and you need to adjust.

I’m actually looking forward for the ? support in tests. And I wanted to see (and haven’t written any RFC for it yet) a quite natural extention for basic fixtures in the current implementation: a test function could take a parameter, provided this parameter would implement Default. With special case of the parameter being self, so the test could be actually a method of the harness itself.

struct Tests { ... };

impl Default for Tests {
  fn default() -> Self { ... }
}

impl Tests {
  #[test]
  fn do_stuff(&mut self) -> Result<(), Error> {
    self.it_works()?;
  }
}

The good think about it is that part is already in the pipeline and the other part feels just obvious.

Anyway, maybe this isn’t the exact place to discuss it, is it?

[Manishearth]

Dumping my thoughts on this. TLDR: I think we also need to care about use cases like cargo fuzz, and try and make all the Cargo.toml goodness for test frameworks work well as well.

The problems

Ideally, a solution for this would involve

• being able to specify a custom framework that takes #[test]
• being able to specify a custom framework that takes #[bench]
• being able to specify a custom framework that takes custom attributes, e.g. #[fuzz]

This will probably make the most sense as a proc-macro style crate.

The compiler should do all the “collect up all the attributes” work for the framework, and provide an endpoint from which the test framework can provide the output AST for the program.

Cargo currently allows you to specify [[test]] and [[bench]] entries. Custom test runners should be usable with these, and also should be able to make their own.

For example, currently cargo-fuzz does a whole bunch of hacks by creating a second nested crate for storing fuzz targets, which would ideally be obviated by this.

A rough solution

Like custom derives, test frameworks can be specified via attributes:

#[test_framework]
fn framework(x: TestHelper) -> TokenStream {

    let body = vec![];
    for func in x.iter_functions("test") {
        let name = parse(func).name;
        body.push(quote_expr!($name()));
   }
   quote_item!{
       fn main() {
           $(body);+
       }
    }
}

(I haven’t written proc macros in a while, so this is pseudocode)

We can also make this more like a regular proc macro where it takes in a TokenStream, spits out a TokenStream, and you need a library to do the collecting of attributes for you. But we may want to add a &TestContext parameter through which you can get additional info and tweak parameters (e.g one could add extra Cargo dependencies through this, which would be nice but not necessary for cargo-fuzz)

A test runner crate will have a test_runner=true key in its Cargo.toml.

Crates using a custom test runner can do so like so:

[testing.test]
cratename = version


[testing.fuzz]
cratename = version

Creating a runner called test or bench will override cargo test and cargo bench respectively. Creating a runner called something else will provide a new kind of test, which can be run with cargo test --kind foo. cargo test --kind foo will run the harness on the crate itself as well as any .rs files found under foo/.

If you wish to specify targets for this kind of test, you can do it like so:

[[testing.target]]
kind = fuzz
path = "foo.rs"
name = "foo"

The [[test]] and [[bench]] keys are sugar for [[testing.target]] kind = test and kind = bench respectively.

[bheisler]

I’ve been thinking about posting a pre-RFC on this subject myself. I have to thank @Manishearth for saving me the trouble - this is exactly what I was going to suggest, only more fleshed out. The only thing I’d add is that the test::black_box function would have to be stabilized as well for benchmarking.

This already covers pretty much everything I need for Criterion, so I’m ready to support it as-is.

With regards to the wishlist of features discussed earlier in this thread (JSON output, customizable formatters, test generation) - I’d suggest that rather than trying to design and standardize the One True Test Framework, this should be left to the Crates.io ecosystem, for all the same reasons that Rust leaves so much else to the ecosystem already. If desired, there can always be something like the HTTP crate added later to improve interoperability. For now, I’d say let’s stabilize the minimum possible interface necessary to allow community experimentation.

[vorner]

Crates using a custom test runner can do so like so:

Just a question… any plans making it possible to combine them, eg let tests/a.rs use one harness and tests/b.rs another?

[Manishearth]

Not in my proposal – in my proposal you’d use different folders for these – but this can be made to work as well if it’s something folks want.

[vorner]

I didn’t have the need, but now there’s just the one default harness, so there’s not much to choose from. But I can imagine I could want two or three different types of tests for the same crate, but would prefer to put all of them into tests, just for clarity.