libtest is in
rust-lang/rust/src/libtest, and it implements both the stable
#[test] system and the unstable
#[bench] functionality (along with some other things). The standard
#[test] harness relies on some libsyntax internals for interacting with other test related macros, which prevent it from being a “normal”
custom_test_framework in the near future, but fixing this isn’t really high-priority, nor would it deliver a lot of value.
There are a couple of things we could easily do that are low effort, deliver instant value, and would help putting some long awaited features on the track towards stabilization (e.g.
We could move
libtest out of
rust-lang/rust, e.g., to
rust-lang/libtest/libtest-shim (and publish
libtest-shim to crates.io), such that the "one and only
libtest" crate in
rust-lang/rust/src/libtestjust re-exports this shim. This would allow us to iterate quickly out of the
rust-lang/rust bors queue.
We could then de-entangle the test framework from the benchmarking framework into three crates:
libtest2: which uses
custom_test_frameworksto expose the testing harness. Initially, this will still implement the
libtestshims re-exported by
libtest, and will be less powerful than the
libtestcrate because it wouldn’t have access to
libsyntaxwhen used directly. Solving these issues will require improving
custom_test_frameworksand other Rust features, but that’s something worth exploring.
libbench: which uses
#[bench]macro . The idea is to make
libbencha 1:1 replacement for the nightly
#[bench]attribute, so that we can, at some point, deprecate it, and once
libbenchwill work on stable.
libtest-core(or similar): both
libtestformatting framework, parts of argument parsing, some traits, types, etc. Initially, we would re-factor them into a
libtest-corecrate, that other
custom_test_frameworkscan use. The idea is to iterate on these APIs on crates.io, and at some point allow people to plug in, e.g., their own output formats that maybe even the normal
libtestcould use. It is hard to iterate on this in tree.
Initially, all of these changes will be non-functional changes, and will mostly benefit
custom_test_frameworks on crates.io, but I hope that the experience we gain will help us land a stable
#[bench] (and stable
custom_test_frameworks in general) quicker.
This is a bit handwaved, but I wrote a quick and dirty PoC of the refactor of
libtest into a
libtest+libcore crate in this PR (https://github.com/rust-lang/rust/pull/57068), @djrenren has an implementation of #[bench] as a custom test framework and a libtest2 here (https://github.com/djrenren/rust-test-frameworks), and @alexcrichton suggested that maybe it is worth it to just split this off tree to allow us to get more experience with these APIs and iterate quicker than if we do things in tree, and need to, e.g., fill in an RFC first for some API to customize libtest output formats or allow user-defined test types to interact with libtest formatting.