#[test] and external test harnesses

[alexcrichton]

We talked about this a bit at the most recent work week (and I actually did similar work awhile ago), but we unfortunately didn’t reach many conclusions in our discussion. I’ll write down some of my thoughts at least and I know that @brson has opinions about this as well!

• #[test] is very ergonomic today, and I would like to strive as much as possible to have custom test frameworks just as ergonomic.
• #[test] is also quite limiting, and there’s quite a laundry list of features others have wanted from test frameworks that aren’t quite well supported, such as:
  • Custom arguments to the function and/or return values (e.g. #[quickcheck])
  • setup/tear down, global, module, and perhaps even hierarchically based
  • state as input to a test
  • various signatures on a test for just a different testing experience. For example maybe instead of an assert!-based failure tests could have a Result-based failure. Maybe I want to return something like Test::Pass to ignore a test at runtime.
  • The are other attributes at play here like #{bench] and #[ignore] we would also want to consider.
• The primary point of src/libsyntax/test.rs is in my opinion doing the clever tricks necessary to reexport all test functions to get to the crate namespace. That part basically makes this transformation outside the scope of a simple syntax extension.

In terms of ergonomics, my “absolute ideal” would be to add this to Cargo.toml

my-custom-test-framework = "1.0"

and be done with it. There are various ways that this could be made to work, and there are various reasons why this could be construed as “too magical”. I would think, however, that if custom test frameworks are much less ergonomic than #[test] they won’t actually end up getting any use. I personally wish to strive as close to this as possible.

I don’t really know what an API we’d want to stabilize would look like (I doubt it’s what it looks like today). I would want to make sure that it has the ability to do all the things I mentioned above (and more if possible). Essentially the test harness should have as much control over it needs to.

[DanielKeep]

This sounds like an opportunity to introduce something which could be used to build #[test]; specifically, the “collecting items” behaviour.

There are various cases where you want some kind of index or registry of things defined elsewhere. Sub commands, tests, error types… possibly even on-startup functions.

Just off the top of my head thinking:

macro_rules! test {
    (() $(pub)* fn $test_fn_name:ident $($_tail:tt)*) => {
        // I'm assuming eager expansion here ($*):
        add_to_global_list!(tests,
            ($*module_path_as_path!(), $test_fn_name, false,
            test::ShouldPanic::No));
    };
}

macro_rules! inject_test_harness {
    () => {
        mod __test {
            extern crate test;
            fn main() {
                test::test_main_static(&::os::args()[], tests)
            }
            inject_test_harness! { @tests [$*read_global_list!(tests)] }
        }
    };

    (@tests [$(($test_mod:path, $test_fn_name:ident, $test_ignore:expr, $test_panic:expr))*]) => {
        static tests: &'static [test::TestDescAndFn] = &[
            $(
                test::TestDescAndFn {
                    desc: test::TestDesc {
                        name: test::TestName::StaticTestString(
                            concat!(stringify!($test_mod), "::",
                            stringify!($test_fn_name))
                        ),
                        ignore: $test_ignore,
                        should_panic: $test_panic,
                    },
                    testfn: test::TestFn::StaticTestFn(
                        as_expr!($test_mod :: $test_fn_name)
                    ),
                },
            )*
        ];
    };
}

Ok, ignore for a moment that we don’t have eager expansion, or the ability to use macros-as-attributes: those can be worked around. We’d need module_path_as_path! or something similar to be added as a built-in (currently, module_path! returns a string literal).

add_to_global_list! and read_global_list! would be used to append to and dump a global, named list of tokens. For sanity, it should be an error to add to a list that’s already been dumped (so that things don’t get lost); so long as the read is triggered at the end of the root module, it should be fine. It’s up to the user to sort out ordering issues with the actual contents. This allows us to construct the list of tests from the attributes.

Also, it’d be nice to have a way to inject a macro invocation at the end of the crate module. --test could just be a shorthand for --extern test=.. --macro-use-extern test --inject-macro inject_test_harness. This would also allow Cargo to specify one or more custom test harnesses.

Aside from the “there’s no stable way to capture output” issue, that should be enough to divorce testing from the compiler, and buy us a hugely convenient way to collect information from across a crate at the same time.

[SimonSapin]

Oooh, very interesting. I didn’t think of making that connection, but yes, a general “collect things” mechanism would be very useful. I have something similar in Servo: CSS properties are declared in one place where they each have a name, a parsing function, an initial value, etc; and several other places generate code for each property (e.g. match on their names and dispatch to parsing functions). Currently this whole thing is a Python/Mako template… which is not pretty.

[brson]

An important requirement for me that I rarely see mentioned is that I need to be able to decouple the test running from the test generation, for example I want to be able to run tests written in any Rust test framework in a Visual Studio GUI.

In this respect I like the internal design of Rust’s test framework - it is just a list of functions to run with some metadata about what the outcome is expected to be. If other test frameworks can compile down to a list of functions that makes the interface to the test runner very simple (e.g. we don’t have to define any specific setup/teardown interface if the frameworks can bake them into the test cases they generate).

Edit: I’m also less enthusiastic than @alexcrichton about adding implicit crate dependencies based on Cargo.toml.

[brson]

A simplified version of TestDescAndFn, with just enough to describe what rustc --test generates. (So probably without dynamic tests.) Some way to override which crate/function is used instead of test::test_main_static. Maybe #[test_harness] extern crate fancy_test;? (If benchmark are supported in this mechanism, TestDescAndFn would probably have to be generic over Bencher.)

I’ve been thinking about this, and it’s similar to what I want. I’d like to have a separation between ‘test framework’ - the plugin that generates the list of tests at compile time - and ‘test runner’ - the thing that runs and reports them. Nearly all the code generation for testing currently in the compiler could be extracted into the ‘default test framework plugin’, while the ‘test’ crate becomes the ‘default test runner’.

I think that the current TestDescAndFn type can probably be simplified. The information it’s passing around about how to interpret tests can be test framework-specific, and just captured in the test functions themselves. The list passed to the test runner can just be pairs of (fn, name), where each function returns a TestResult of some kind.

For example, given the current syntax:

#[test]
fn mytest() {
}

The default test framework would expand that to something vaguely like

fn run_my_test() -> TestResult {
    let res = catch_panic(mytest).;
    if res.is_ok() { TestResult::Ok } else { TestResult::Err }
}

Or an ignored test to

fn run_my_test() -> TestResult {
    return TestResult::Ignored
}

Hopefully test frameworks can just be plugins. Test runners though are something else - they are a crate that defines a main function that accepts some custom data. We may be able to abstract this concept in a way that the std runtime entry and the test runner entry use the same mechanism.

[ssheldon]

An important requirement for me that I rarely see mentioned is that I need to be able to decouple the test running from the test generation

A big +1 to this from me; that’d make it trivial to run tests in an iOS app. My current solution is a Python script to copy all of my test code into a separate crate so I can run it on iOS… https://github.com/SSheldon/rust-objc/blob/e034af2/xtests/build.py

[samphippen]

Hi Everyone,

This is my first post to internals, so please let me know if I make any faux pas. To introduce myself, I’m Sam. For the past 3 years now, over in Ruby land, I’ve been a maintainer of the RSpec testing framework. I’ve been very excited about rust, and I’ve been working on a testing framework as a hobby project. The core part of the framework is called descriptor, and you can find the code here.

I’ve read through this thread and have a couple of thoughts that would make writing descriptor easier:

• at the moment, describe (the function to create test groups) and it (the function to create tests) have to be invoked from main, or something called from main. My understanding is that rust doesn’t have a static context, and do we can’t just have files that have bare describes in them, but if there was an annotation a user could but in front of a describe that’d be great.
• source locations of tests: at the moment, I’m having to pull source location with a macro. One problem with this approach is that if you nest block macros, you lose all line number information (see this example.
• cargo test being a way to invoke descriptor_main(), preferably with any flags passed to cargo test being passed through. It’d also be nice if I could specify a directory (like spec/, perhaps user specified) where the compiler should look for test files.

I’m not sure how achievable any of this is, and I’d be happy to talk about my test framework’s design goals, how I see it going and so on. Do let me know if you’ve got any questions

[samphippen]

Oh, one further thing I forgot to mention earlier that might be useful. All my tests are dispatched in their own thread, which is pretty interesting I think, and might affect the design choices here a little.

[alexcrichton]

Thanks for the reply @samphippen! The split @brson is talking about between a test framework and a test runner seems like it’d fit quite nicely into what you’re talking about. Specifically the descriptor crate would in theory largely be a test framework, which in this case would manifest itself as a plugin. As a plugin you could have any kind of syntax you want almost and you’d also have information naturally to info like file/line numbers.

The descriptor_main would probably be handled by your own test runner, but the split means that some could use the descriptor syntax for defining tests but something else for running them all (maybe to generate machine-readable output or something like that).

The stabilization plan for plugins is pretty long term at this point, but it provides us quite a nice vector for being extra flexible in how test frameworks are implemented!

[jimmycuadra]

Bump to rekindle discussion cause testing in Rust is really painful right now without setup and teardown.

We can get around per-test setup/teardown by wrapping the test bodies in a function that takes a closure, but is there any way to do a setup/teardown once for the entire test suite? I can achieve a global setup phase by abusing lazy_static, but AFAICT the destructor on the value created doesn’t get run at the end of the program, so I’m not sure how to achieve a global teardown phase right now.

/cc @wycats

[SimonSapin]

@jimmycuadra Anything added into Rust proper is gonna take time to be designed, developed, and stabilized. Assuming someone does start doing all that work.

In the meantime, I recommend doing as I describe at the start of this thread:

Cargo.toml

[[test]]
name = "foo"
harness = false

[dependencies]
rustc-test = "0.1"

tests/foo.rs

extern crate test;
fn main() {
    let tests = collect_tests();
    setup();
    test::test_main(&std::env::args().collect::<Vec<_>>(), tests);
    teardown();
}

To go further, you can fork https://github.com/SimonSapin/rustc-test, add your own functionality to the harness, and use your fork instead of rustc-test in [dependencies].

[matklad]

I would like to link this Cargo issue her: https://github.com/rust-lang/cargo/issues/1924, it seems relevant.

[jimmycuadra]

Just want to double check: Right now setting harness = false in Cargo.toml prevents Cargo from passing --test to rustc, which disables both the test runner and the libsyntax code that collects the tests and generates the module which calls the runner. AFAICT there is currently no way to disable just the test runner part and not the libsyntax part.

Edit: Nevermind, I see now that --test just does the code generation, and the generated code uses the test crate which contains the runner, but that crate doesn’t need to be “disabled” in any way. But it is true that the libsyntax stuff is all or nothing, so you can’t use its test collecting capabilities but write your own main function to kick off the runner.

Given that we’re talking about a larger overhaul of the testing system, I wonder if there would be any value in an RFC to add two separate flags to control both of those things separately, i.e. --test expands to --test-main and --test-collect, but you could control the behavior using only one of the latter two arguments if you wanted.

[Ericson2314]

A few thoughts. [Just skimmed the thread so sorry if I am repeating things.]

Most importantly, I’d like to see everything in rustc related to tests turned into library (compiler plugin + runtime, probably). Because our current test framework is so ergonomic (yay!!), I think that is a superb, maybe even unparalleled, goal with which to stress the compiler plugin framework.

With that in place, there are a few fun things I’d like to add on top. Besides what is listed, it would also be nice to either run 1 process per test, or more efficiently continue with a new process where testing was left off after test failure. Finally, putting everything together, I’d like to ergonomic write unit tests for OS dev (or more generally, cross compiled + emulator, e.g. this would be good for Servo on Android). System software traditionally has been more poorly tested than most, and I’d like to not only smash that problem, but smash it with today’s stellar ergonomics.

[dhardy]

Would it make sense to allow libraries to auto-initialise themselves for tests? I’m just thinking of a simple way to get a logger initialised; extern crate env_logger; vs having to write env_logger::init().unwrap(); in every test or create some kind of set-up function just to initialise the logger.

It might make more sense if this is configurable (for test harnesses or more generally), e.g.

#[init]
extern crate env_logger;

[Ericson2314]

That is like C++'s “life before main” and very very scary IMO. Most initialization is a smell and indeed if we had needs-provides we wouldn’t need it for log.

[dhardy]

Yes, life-before-main. But (1) libraries can document what they do and (2) it can be disabled simply by not including the #[init] line (in this case manual initialisation would be needed).

Alternatively, I suppose a test-driver could work with a user-provided main function which does the init then calls some “special” function to run the tests.

[Ericson2314]

Yeah, something like your alternative is good because it solves the problem—without making initialization too easy or implicit :).

[naicode]

Wrt. to IDE integration, I think it is a good idea if test harnesses pass there “results” (inclusive possible stacktraces, output errors etc.) to some standard interface, which reports them by default to the command line, but can be configured to report in some more “automatic consumer” friendly stile for IDE’s or alternate test reporters.

What do I mean with alternate test reporters?

• while alternate reporters might be specified through Cargo.toml and might be integrated into the test binary, it could also be possible to “just” tell rust, cargo or the test binary to report through a machine readable format over stderr/stdout. A alternate reporter could “just” consume this stream/pipe. (edit: rewrote this bullet point)
• they have a way to consume the test output and differentiate between panic’s error messages, [maybe also normal stderr/out output?], stack traces and test results [ok,fail,ignore] (note 1)
  • edit: e.g. do not send a text over the stderr/stdout but e.g. stream JSON like objects
• example for alternative reporters:
  • a reporter used in some CI system displaying errors on a web site
  • a reporter for having/testing some additional features in the reporter (note 2)
  • a reporter used by a IDE to display results “integrated” with the IDE
  • a reporter making thinks easier for blind developers (edit: added this)

note 1: this should be possible, because only one test is run per thread concurrently, so the test harness should be aware about which information are related to which test, including the currently hidden “normal” output.

note 2: for example I really would like it, if the test detects errors reported by ‘assert_eq!’ and then uses a string based diff to show exactly where the difference starts/ends (through yes, this only works if the Debug output differs) (edit: there is a RFC for this example)

EDIT: removed misplaced though about implementation details wrt. reporting multiple test in parallel (or how to stream JSON like data as multiple parts)

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.