# Alternate libtest output format **URL:** \ **Category:** Uncategorized\ **Created:** [October 27, 2017, 3:14pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121 "2017-10-27T15:14:06Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Gilnaa](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gilnaa/32/2942_2.png) [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)\ **Post date:** [October 27, 2017, 3:14pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/1 "2017-10-27T15:14:06Z")
I would like to implement alternative output formats to Rust’s testing tools. (e.g. JSON) I’ve looked at the code and the it looks pretty straight forward. What is the usual process for these kind of things? Should I write an RFC, or just send a PR with implementation? _Edit_: What I plan to do is this: -q should print nothing, not even a single character. Maybe this is a bad idea since people may depend on this for parsing. –format should support at least these three formats: “pretty” the default. “terse” previously -q (should be present even if q is unchanged) “json” json ---
**Author:** ![ehuss](https://avatars.discourse-cdn.com/v4/letter/e/9de0a6/32.png) [@ehuss](https://internals.rust-lang.org/u/ehuss)\ **Post date:** [October 29, 2017, 6:29pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/2 "2017-10-29T18:29:19Z")
I could see this being useful for integrating with an editor. Would it be possible to get the file/line number of each test? Parsing the error messages is very unreliable. ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 30, 2017, 2:34pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/3 "2017-10-30T14:34:34Z")
It would be super useful to have a better test reporter for IntelliJ Rust! The main problem I see here is though is that ideally we should use some standard test reporting protocol, but unfortunately there isn’t one good enough as far as I know. There’s [http://testanything.org/](http://testanything.org/), which is rather widespread, but it does not specify what to do with tests stdout/stderr, and it completely breaks down if you want to run tests in parallel, print to stdout from tests and receive test output in a streaming fashion. IIRC, current libtest does not handle this either, and I am not sure if this problem is even solvable at all! ---
**Author:** ![Gilnaa](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gilnaa/32/2942_2.png) [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)\ **Post date:** [October 30, 2017, 6:16pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/4 "2017-10-30T18:16:47Z")
> [@ehuss](#): > > Would it be possible to get the file/line number of each test? It depends on if the current libtest has access to this information. Wouldn't a test name be enough? > [@matklad](#): > > it does not specify what to do with tests stdout/stderr, Well, assuming the stdout/stderr is textual, it can be just added as another field to the JSON output. Non-printable characters can be escaped, I guess. ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 30, 2017, 6:29pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/5 "2017-10-30T18:29:15Z")
> [@Gilnaa](#): > > Well, assuming the stdout/stderr is textual, it can be just added as another field to the JSON output. Non-printable characters can be escaped, I guess. And this should support streaming as well: if a long running tests prints status updates once in a while, they should be available as they appear, not after the test has ended. ---
**Author:** ![Gilnaa](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gilnaa/32/2942_2.png) [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)\ **Post date:** [October 30, 2017, 6:36pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/6 "2017-10-30T18:36:34Z")
> [@matklad](#): > > And this should support streaming as well: if a long running tests prints status updates once in a while, they should be available as they appear, not after the test has ended. I'm not sure if I understand how this works with the nature of json. Is it something that is implemented today? AFAICT it's best that a test - long running or not - will only be printed as soon as it finishes, otherwise a test might span multiple distinct json objects. ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 30, 2017, 6:41pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/7 "2017-10-30T18:41:35Z")
> [@Gilnaa](#): > > I’m not sure if I understand how this works with the nature of json. Is it something that is implemented today? Yep! You print a \n delimited JSON messages without internal \n's. That is, output is a stream of messages, each message is a single-line JSON. ---
**Author:** ![Gilnaa](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gilnaa/32/2942_2.png) [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)\ **Post date:** [October 30, 2017, 6:46pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/8 "2017-10-30T18:46:35Z")
Not sure how that answers my question. My thought was exactly one JSON object per test result, and your comment suggests otherwise. Anyway, I think it’s best for the whole output to be parseable as a single file, so “\n” is probably less ideal than “,” ``` [{"test_name": "bla", "status": "passed", "stdout": "..."}, {"test_name": "bla2", "status": "ignored", "stdout": "..."}] ``` ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 30, 2017, 6:51pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/9 "2017-10-30T18:51:41Z")
> [@Gilnaa](#): > > My thought was exactly one JSON object per test result, and your comment suggests otherwise. Hm, I think it's important to give at least separate messages for `test started` / `test ended` events. That is, at least _some_ streaming is important, otherwise you'll have to wait until the whole tests suite ends without any feedback, which might take a while! ---
**Author:** ![Gilnaa](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gilnaa/32/2942_2.png) [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)\ **Post date:** [October 30, 2017, 6:54pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/10 "2017-10-30T18:54:08Z")
I now see what you mean, but you _do_ get test result as they are ready, are event starts really necessary? (I’m not opposed, just interested) ---
**Author:** ![matklad](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/matklad/32/12266_2.png) [@matklad](https://internals.rust-lang.org/u/matklad)\ **Post date:** [October 30, 2017, 7:06pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/11 "2017-10-30T19:06:37Z")
I think yes: if test `foo` gets stuck in an infinite loop, it's helpful to see `started test foo`. > but you do get test result as they are ready, It's important that machine readable output is easy to split into discrete chunks, that's why I suggested JSON per line: it seems to be easier to parse bit by bit. ---
**Author:** ![Gilnaa](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gilnaa/32/2942_2.png) [@Gilnaa](https://internals.rust-lang.org/u/Gilnaa)\ **Post date:** [October 30, 2017, 7:16pm UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/12 "2017-10-30T19:16:33Z")
I see, that makes since. As far as the separator; if we keep it a comma, you would be able to use a [stream parser](https://docs.serde.rs/serde_json/struct.StreamDeserializer.html) (which is a thing and it’s pretty great). Edit: actually the stream parser doesn’t work as I remembered, after looking at old code, I actually had to manually remove commas from the stream ---
**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\ **Post date:** [March 25, 2019, 8:29am UTC](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121/13 "2019-03-25T08:29:09Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.