Path for stabilizing libtest's json output?

tools and infrastructure
1

I proposed to T-testing-devex that our first order of business should be to stabilize --format json. Think of this as a Pre-eRFC. Unsure what route the more official proposal will take for being approved. What I'm looking for is input on what we should consider when vetting the format to see if it'll be sufficient. See the list below that I've collected so far.

Rust 1.70 fixed libtest so that --format json required being used with a nightly which highlighted how important it is to people.

--format json could also help us improve cargo test, including

Most of that involves shifting responsibilities from the test harness to the test runner which has the side effects of:

  • Allowing more powerful experiments with custom test runners (e.g. cargo nextest) as they'll have more information to operate on
  • Lowering the barrier for custom test harnesses (like libtest-mimic) as UI responsibilities are shifted to the test runner (cargo test)

Proposed Plan

While having a plan for evolution takes some burden off of the format, we should still do some due diligence in ensuring the format works well for our intended uses.

My rough idea for a plan is

  1. Create an experimental test harness that uses a serde structure for passing information from its core to different --format modes, emulating what libtest and cargos relationship will be like on a smaller scale for faster iteration
  2. Transition libtest to this proposed interface
  3. Add experimental support for cargo to interact with test binaries through json
  4. Create a stabilization report for json for T-libs-api and a cargo RFC for custom test harnesses to opt into this new protocol

Potential considerations when running the experiment to vet the format:

Warning: This doesn't mean they'll all be supported in the initial stabilization just that we feel confident the format will support them)

If you are interested in helping this stabilize the json output, contributing to the experimental harness (when we get there) will be a good area for first-time contributors to the Rust project as most of the code will be new, small, and with no stability / correctness pressure of an existing user base.

Misc

Comments made on libtests format

Existing formats

See also

9 Likes
2

It's funny you opened this just now, I posted about something somewhat related in on Zulip. I'm planning on writing a proposal outlining:

  • A mechanism for opting into nightly-only machine human/machine-readable formats on the stable channel that isn't RUSTC_BOOTSTRAP=1.
  • A stability/evolution evolution policy for said unstable formats (such as a versioning scheme) and a mechanism for tooling authors to opt-into notifications for changes.

One of the motivating cases was libtest's JSON output, with the hope that the barrier to feedback is reduced and implementers don't feel hamstrung by notions like "perfect is the enemy of good". How would you feel about something along those lines for libtest?

1 Like
3

While I think it would generally be helpful to think in terms of making it easier to experiment with unstable features, I don't think it would help in our case. We have a path forward that can help us learn a lot, quickly outside of the rust-lang/rust tree, more so than I expect a process like this to allow. It might help get whats there today into people's hands but there isn't even a definition of what exists today for us to version, just inference on behavior. I don't expect any useful feedback from doing so because I honestly expect to throw the existing format out. That also means I don't expect much benefit from investing in the current format.

4

Ah, my bad! If you can go faster out-of-tree, then that's good to hear. I assumed that you'd want to partially add/stabilize support for additional features based on this comment:

At least for things like test locations, benches or doctests, I (naively) imagined that it might not end up in the initial stabilization.

5

They won't. I had assumed you were talking more generally for the stabilization process. Sounds like you are instead referring to the item "Plan for future evolution". That will be something we focus on through the effort and would appreciate input on then.

6

I've gone ahead and created eRFC: Iterate on and stabilize libtest's programmatic output by epage · Pull Request #3558 · rust-lang/rfcs · GitHub

2 Likes
7

Have you considered how benchmarking harnesses like iai-callgrind could work with the new apis?

8

Benchmarks are listed in the RFC is one of the areas we will explore. In the next T-testing-devex meeting, we're going to be talking with one of the creators of divan about this (who also happens to be on the team).

1 Like
9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.