Alternate libtest output format

[Gilnaa]

I would like to implement alternative output formats to Rust’s testing tools. (e.g. JSON)

I’ve looked at the code and the it looks pretty straight forward.

What is the usual process for these kind of things? Should I write an RFC, or just send a PR with implementation?

Edit: What I plan to do is this:

-q should print nothing, not even a single character. Maybe this is a bad idea since people may depend on this for parsing.

–format should support at least these three formats: “pretty” the default. “terse” previously -q (should be present even if q is unchanged) “json” json

[ehuss]

I could see this being useful for integrating with an editor. Would it be possible to get the file/line number of each test? Parsing the error messages is very unreliable.

[matklad]

It would be super useful to have a better test reporter for IntelliJ Rust! The main problem I see here is though is that ideally we should use some standard test reporting protocol, but unfortunately there isn’t one good enough as far as I know.

There’s http://testanything.org/, which is rather widespread, but it does not specify what to do with tests stdout/stderr, and it completely breaks down if you want to run tests in parallel, print to stdout from tests and receive test output in a streaming fashion. IIRC, current libtest does not handle this either, and I am not sure if this problem is even solvable at all!

[Gilnaa]

Would it be possible to get the file/line number of each test?

It depends on if the current libtest has access to this information. Wouldn’t a test name be enough?

it does not specify what to do with tests stdout/stderr,

Well, assuming the stdout/stderr is textual, it can be just added as another field to the JSON output. Non-printable characters can be escaped, I guess.

[matklad]

Well, assuming the stdout/stderr is textual, it can be just added as another field to the JSON output. Non-printable characters can be escaped, I guess.

And this should support streaming as well: if a long running tests prints status updates once in a while, they should be available as they appear, not after the test has ended.

[Gilnaa]

And this should support streaming as well: if a long running tests prints status updates once in a while, they should be available as they appear, not after the test has ended.

I’m not sure if I understand how this works with the nature of json. Is it something that is implemented today?

AFAICT it’s best that a test - long running or not - will only be printed as soon as it finishes, otherwise a test might span multiple distinct json objects.

[matklad]

I’m not sure if I understand how this works with the nature of json. Is it something that is implemented today?

Yep! You print a \n delimited JSON messages without internal \n’s. That is, output is a stream of messages, each message is a single-line JSON.

[Gilnaa]

Not sure how that answers my question. My thought was exactly one JSON object per test result, and your comment suggests otherwise.

Anyway, I think it’s best for the whole output to be parseable as a single file, so “\n” is probably less ideal than “,”

[{"test_name": "bla", "status": "passed", "stdout": "..."},
{"test_name": "bla2", "status": "ignored", "stdout": "..."}]

[matklad]

My thought was exactly one JSON object per test result, and your comment suggests otherwise.

Hm, I think it’s important to give at least separate messages for test started / test ended events. That is, at least some streaming is important, otherwise you’ll have to wait until the whole tests suite ends without any feedback, which might take a while!

[Gilnaa]

I now see what you mean, but you do get test result as they are ready, are event starts really necessary? (I’m not opposed, just interested)

[matklad]

I think yes: if test foo gets stuck in an infinite loop, it’s helpful to see started test foo.

but you do get test result as they are ready,

It’s important that machine readable output is easy to split into discrete chunks, that’s why I suggested JSON per line: it seems to be easier to parse bit by bit.

[Gilnaa]

I see, that makes since.

As far as the separator; if we keep it a comma, you would be able to use a stream parser (which is a thing and it’s pretty great).

Edit: actually the stream parser doesn’t work as I remembered, after looking at old code, I actually had to manually remove commas from the stream

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.