# Path for stabilizing libtest's json output? **URL:** \ **Category:** tools and infrastructure\ **Created:** [January 9, 2024, 8:03pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163 "2024-01-09T20:03:45Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [January 9, 2024, 8:03pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/1 "2024-01-09T20:03:45Z")
I proposed to [T-testing-devex](https://www.rust-lang.org/governance/teams/dev-tools#Testing%20DevEx%20team) that our first order of business should be to stabilize `--format json`. Think of this as a Pre-eRFC. Unsure what route the more official proposal will take for being approved. **What I'm looking for is input on what we should consider when vetting the format to see if it'll be sufficient.** See the list below that I've collected so far. Rust 1.70 fixed libtest so that `--format json` required being used with a nightly which highlighted [how important it is to people](https://www.reddit.com/r/rust/comments/13xqhbm/announcing_rust_1700/jmji422/). `--format json` could also help us improve `cargo test`, including - [Wanting to run test binaries in parallel](https://github.com/rust-lang/cargo/issues/5609), like `cargo nextest` - [Lack of summary across all binaries](https://github.com/rust-lang/cargo/issues/4324) - [Noisy test output](https://github.com/rust-lang/cargo/issues/2832) (see also [#5089](https://github.com/rust-lang/cargo/issues/5089)) - [Confusing command-line interactions](https://github.com/rust-lang/cargo/issues/1983) (see also [#8903](https://github.com/rust-lang/cargo/issues/8903), [#10392](https://github.com/rust-lang/cargo/issues/10392)) - [Poor messaging when a filter doesn't match](https://github.com/rust-lang/cargo/issues/6151) - [Smarter test execution order](https://github.com/rust-lang/cargo/issues/6266) (see also [#8685](https://github.com/rust-lang/cargo/issues/8685), [#10673](https://github.com/rust-lang/cargo/issues/10673)) - [JUnit output is incorrect when running multiple test binaries](https://github.com/rust-lang/rust/issues/85563) - [Lack of failure when test binaries exit unexpectedly](https://github.com/rust-lang/rust/issues/87323) Most of that involves shifting responsibilities from the test harness to the test runner which has the side effects of: - Allowing more powerful experiments with custom test runners (e.g. `cargo nextest`) as they'll have more information to operate on - Lowering the barrier for custom test harnesses (like `libtest-mimic`) as UI responsibilities are shifted to the test runner (`cargo test`) ### Proposed Plan While having a plan for evolution takes some burden off of the format, we should still do some due diligence in ensuring the format works well for our intended uses. My rough idea for a plan is 1. Create an experimental test harness that uses a `serde` structure for passing information from its core to different `--format` modes, emulating what `libtest` and `cargo`s relationship will be like on a smaller scale for faster iteration 2. Transition libtest to this proposed interface 3. Add experimental support for cargo to interact with test binaries through json 4. Create a stabilization report for json for T-libs-api and a cargo RFC for custom test harnesses to opt into this new protocol Potential considerations when running the experiment to vet the format: - Plan for future evolution - Ability to implement different format modes on top - Both test running and `--list` mode - Ability to run test harnesses in parallel - [Tests with multiple failures](https://docs.rs/googletest/0.10.0/googletest/prelude/macro.expect_that.html) - Bench support - Static and dynamic [parameterized tests / test fixtures](https://crates.io/crates/rstest) - Static and [dynamic test skipping](https://doc.crates.io/contrib/tests/writing.html#cargo_test-attribute) - [Test markers](https://docs.pytest.org/en/7.4.x/example/markers.html#mark-examples) - doctests - Test location (for IDEs) - Collect metrics related to tests - Elapsed time - Temp dir sizes - RNG seed **Warning:** This doesn't mean they'll all be supported in the initial stabilization just that we feel confident the format will support them) If you are interested in helping this stabilize the json output, contributing to the experimental harness (when we get there) will be a good area for first-time contributors to the Rust project as most of the code will be new, small, and with no stability / correctness pressure of an existing user base. ### Misc Comments made on libtests format - [Format is complex](https://github.com/rust-lang/rust/issues/49359#issuecomment-467994590) (see also [1](https://github.com/rust-lang/rust/issues/49359#issuecomment-1531369119)) - [Benches need love](https://github.com/rust-lang/rust/issues/49359#issuecomment-467994590) - [Type field is overloaded](https://github.com/rust-lang/rust/issues/49359#issuecomment-467994590) - [Suite/child relationship is missing](https://github.com/rust-lang/rust/issues/49359) - [Suite lacks a name](https://github.com/rust-lang/rust/issues/49359#issuecomment-699691296) - [Format is underspecified](https://github.com/rust-lang/rust/issues/49359#issuecomment-706566635) - ~~[Lacks ignored reason](https://github.com/rust-lang/rust/issues/49359#issuecomment-715877950)~~ ([resolved?](https://github.com/rust-lang/rust/issues/49359#issuecomment-1531369119)) - [Lack of `rendered` field](https://github.com/rust-lang/rust/issues/49359#issuecomment-1531369119) Existing formats - junit - [subunit](https://github.com/testing-cabal/subunit) - [TAP](https://testanything.org/) See also - [Tracking issue for libtest JSON output · Issue #49359 · rust-lang/rust · GitHub](https://github.com/rust-lang/rust/issues/49359) - [Iterating on Testing in Rust](https://epage.github.io/blog/2023/06/iterating-on-test/) - [Alternate libtest output format](https://internals.rust-lang.org/t/alternate-libtest-output-format/6121) - [Past, present, and future for Rust testing](https://internals.rust-lang.org/t/past-present-and-future-for-rust-testing/6354) ---
**Author:** ![endsofthreads](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/endsofthreads/32/1769_2.png) [@endsofthreads](https://internals.rust-lang.org/u/endsofthreads)\ **Post date:** [January 9, 2024, 8:49pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/2 "2024-01-09T20:49:20Z")
It's funny you opened this just now, I posted about something somewhat related in [on Zulip](https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Mechanism.20for.20Tools.20to.20opt.20into.20unstable.20features/near/411762121). I'm planning on writing a proposal outlining: - A mechanism for opting into nightly-only machine human/machine-readable formats on the stable channel that _isn't_ `RUSTC_BOOTSTRAP=1`. - A stability/evolution evolution policy for said unstable formats (such as a versioning scheme) and a mechanism for tooling authors to opt-into notifications for changes. One of the motivating cases was libtest's JSON output, with the hope that the barrier to feedback is reduced and implementers don't feel hamstrung by notions like "perfect is the enemy of good". How would you feel about something along those lines for libtest? ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [January 9, 2024, 9:07pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/3 "2024-01-09T21:07:29Z")
> [@endsofthreads](#): > > One of the motivating cases was libtest's JSON output, with the hope that the barrier to feedback is reduced and implementers don't feel hamstrung by notions like "perfect is the enemy of good". How would you feel about something along those lines for libtest? While I think it would generally be helpful to think in terms of making it easier to experiment with unstable features, I don't think it would help in our case. We have a path forward that can help us learn a lot, quickly outside of the rust-lang/rust tree, more so than I expect a process like this to allow. It might help get whats there today into people's hands but there isn't even a definition of what exists today for us to version, just inference on behavior. I don't expect any useful feedback from doing so because I honestly expect to throw the existing format out. That also means I don't expect much benefit from investing in the current format. ---
**Author:** ![endsofthreads](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/endsofthreads/32/1769_2.png) [@endsofthreads](https://internals.rust-lang.org/u/endsofthreads)\ **Post date:** [January 9, 2024, 9:33pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/4 "2024-01-09T21:33:41Z")
> [@epage](#): > > While I think it would generally be helpful to think in terms of making it easier to experiment with unstable features, I don't think it would help in our case. Ah, my bad! If you can go faster out-of-tree, then that's good to hear. I assumed that you'd want to _partially_ add/stabilize support for additional features based on this comment: > [@epage](#): > > **Warning:** This doesn't mean they'll all be supported in the initial stabilization just that we feel confident the format will support them) At least for things like test locations, benches or doctests, I (naively) imagined that it might not end up in the initial stabilization. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [January 9, 2024, 9:45pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/5 "2024-01-09T21:45:59Z")
They won't. I had assumed you were talking more generally for the stabilization process. Sounds like you are instead referring to the item "Plan for future evolution". That will be something we focus on through the effort and would appreciate input on then. ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [January 19, 2024, 7:40pm UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/6 "2024-01-19T19:40:14Z")
I've gone ahead and created [eRFC: Iterate on and stabilize libtest's programmatic output by epage · Pull Request #3558 · rust-lang/rfcs · GitHub](https://github.com/rust-lang/rfcs/pull/3558) ---
**Author:** ![infogulch](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/infogulch/32/4912_2.png) [@infogulch](https://internals.rust-lang.org/u/infogulch)\ **Post date:** [January 20, 2024, 12:29am UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/7 "2024-01-20T00:29:36Z")
Have you considered how benchmarking harnesses like [iai-callgrind](https://github.com/iai-callgrind/iai-callgrind) could work with the new apis? ---
**Author:** ![epage](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/epage/32/3171_2.png) [@epage](https://internals.rust-lang.org/u/epage)\ **Post date:** [January 20, 2024, 1:26am UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/8 "2024-01-20T01:26:56Z")
Benchmarks are listed in the RFC is one of the areas we will explore. In the next T-testing-devex meeting, we're going to be talking with one of the creators of divan about this (who also happens to be on the team). ---
**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\ **Post date:** [April 19, 2024, 1:27am UTC](https://internals.rust-lang.org/t/path-for-stabilizing-libtests-json-output/20163/9 "2024-04-19T01:27:20Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.