Libtest: provide position in JSON output

tools and infrastructure
1

since the libtest JSON output is not yet stabilised (Tracking issue for libtest JSON output · Issue #49359 · rust-lang/rust · GitHub) there should still be a chance to get this in:

it'd be great if the JSON output would contain the position information (file, line, character or better yet: span) for at least the test function being executed and in the best case also for assertions.

this could then be used by other tools which e.g. annotate pull requests on GitHub (e.g. i'm using nextest in combination with their JUnit output, followed by action-junit-report to publish the results). currently it does not contain the position information, accordingly no results are being printed on the actual files in the diff.

example: instead of:

{ "type": "test", "event": "started", "name": "tests::it_works" }
{ "type": "test", "name": "tests::it_works", "event": "failed", "stdout": "\nthread 'tests::it_works' panicked at src\\lib.rs:12:9:\nassertion `left == right` failed\n  left: 4\n right: 5\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n" }

i'd like to see:

{ "type": "test", "event": "started", "name": "tests::it_works", "file": "src/lib.rs", "line": 10, "position": 8 }
{ "type": "test", "name": "tests::it_works", "event": "failed", "stdout": "\nthread 'tests::it_works' panicked at src\\lib.rs:12:9:\nassertion `left == right` failed\n  left: 4\n right: 5\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n", "line": 12, "position": 9 }

or, even better:

{ "type": "test", "event": "started", "name": "tests::it_works", "file": "src/lib.rs", "line_start": 10, "position_start": 8, "line_end": 10, "position_end": 16 }
{ "type": "test", "name": "tests::it_works", "event": "failed", "stdout": "\nthread 'tests::it_works' panicked at src\\lib.rs:12:9:\nassertion `left == right` failed\n  left: 4\n right: 5\nnote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n", "line_start": 12, "position_start": 9, "line_end": 12, "position_end": 30 }

i'm aware of the fact that this request is - at least for the part about reporting this information on the failure message - is probably not fully fleshed out as it might be that it contains multiple failures (e.g. when using expect_that from googletest) or possibly no line where it failed (because it instead returned an Error). i'm posting this to get the discussion started because i was surprised that i didn't see an existing issue/post about this yet.

see also: Feature request: add file & line information to JUnit report · nextest-rs/nextest · Discussion #2420 · GitHub

1 Like
2

When we laid out the considerations for designing json output, we included

Test location (for IDEs)

My assumption would be this would be the start and end of the test within the "discovery" message. Most test failure methods don't have a way to report back to the framework where the failure occurred and so I'm unsure how worth it it would be to design that into the schema.

3

ah, i missed that it was mentioned there (but not since?) - thanks for pointing that out!

i'd hope that once the possibility for reporting is there (in a backwards-compatible manner) then the test failure methods could start using it?

come to think of it: it'd probably be great if things like expect_that could report an individual failure so that a failing test then has multiple failures attached if multiple expect_that failed while executing that test - with each failure being reported separately and with its own location.

4

For test failures involving a panic, the framework could extract the location via panic hook. In most cases that would pinpoint the failing assertion, which would be quite valuable.

Backtrace information could be output as well, though that comes with the complications of defining what to output (the same reason why backtraces can currently only be printed).