#[test] and external test harnesses

[Ericson2314]

Yeah, something like your alternative is good because it solves the problem—without making initialization too easy or implicit :).

[naicode]

Wrt. to IDE integration, I think it is a good idea if test harnesses pass there “results” (inclusive possible stacktraces, output errors etc.) to some standard interface, which reports them by default to the command line, but can be configured to report in some more “automatic consumer” friendly stile for IDE’s or alternate test reporters.

What do I mean with alternate test reporters?

• while alternate reporters might be specified through Cargo.toml and might be integrated into the test binary, it could also be possible to “just” tell rust, cargo or the test binary to report through a machine readable format over stderr/stdout. A alternate reporter could “just” consume this stream/pipe. (edit: rewrote this bullet point)
• they have a way to consume the test output and differentiate between panic’s error messages, [maybe also normal stderr/out output?], stack traces and test results [ok,fail,ignore] (note 1)
  • edit: e.g. do not send a text over the stderr/stdout but e.g. stream JSON like objects
• example for alternative reporters:
  • a reporter used in some CI system displaying errors on a web site
  • a reporter for having/testing some additional features in the reporter (note 2)
  • a reporter used by a IDE to display results “integrated” with the IDE
  • a reporter making thinks easier for blind developers (edit: added this)

note 1: this should be possible, because only one test is run per thread concurrently, so the test harness should be aware about which information are related to which test, including the currently hidden “normal” output.

note 2: for example I really would like it, if the test detects errors reported by ‘assert_eq!’ and then uses a string based diff to show exactly where the difference starts/ends (through yes, this only works if the Debug output differs) (edit: there is a RFC for this example)

EDIT: removed misplaced though about implementation details wrt. reporting multiple test in parallel (or how to stream JSON like data as multiple parts)