A path forward towards re-usable libtest functionality, custom_test_frameworks, and a stable #[bench] macro

[gnzlbg]

Right now, libtest is in rust-lang/rust/src/libtest, and it implements both the stable #[test] system and the unstable #[bench] functionality (along with some other things). The standard #[test] harness relies on some libsyntax internals for interacting with other test related macros, which prevent it from being a “normal” custom_test_framework in the near future, but fixing this isn’t really high-priority, nor would it deliver a lot of value.

There are a couple of things we could easily do that are low effort, deliver instant value, and would help putting some long awaited features on the track towards stabilization (e.g. #[bench] !).

We could move libtest out of rust-lang/rust, e.g., to rust-lang/libtest/libtest-shim (and publish libtest-shim to crates.io), such that the "one and only libtest" crate in rust-lang/rust/src/libtestjust re-exports this shim. This would allow us to iterate quickly out of the rust-lang/rust bors queue.

We could then de-entangle the test framework from the benchmarking framework into three crates:

• libtest2: which uses custom_test_frameworks to expose the testing harness. Initially, this will still implement the libtest shims re-exported by libtest-shim into libtest, and will be less powerful than the libtest crate because it wouldn’t have access to libsyntax when used directly. Solving these issues will require improving custom_test_frameworks and other Rust features, but that’s something worth exploring.

• libbench: which uses custom_test_frameworks to implement #[bench] macro . The idea is to make libbench a 1:1 replacement for the nightly #[bench] attribute, so that we can, at some point, deprecate it, and once custom_test_frameworks are stabilized, libbench will work on stable.

• libtest-core (or similar): both libtest2 and libbench share the libtest formatting framework, parts of argument parsing, some traits, types, etc. Initially, we would re-factor them into a libtest-core crate, that other custom_test_frameworks can use. The idea is to iterate on these APIs on crates.io, and at some point allow people to plug in, e.g., their own output formats that maybe even the normal libtest could use. It is hard to iterate on this in tree.

Initially, all of these changes will be non-functional changes, and will mostly benefit custom_test_frameworks on crates.io, but I hope that the experience we gain will help us land a stable #[bench] (and stable custom_test_frameworks in general) quicker.

This is a bit handwaved, but I wrote a quick and dirty PoC of the refactor of libtest into a libtest+libcore crate in this PR (https://github.com/rust-lang/rust/pull/57068), @djrenren has an implementation of #[bench] as a custom test framework and a libtest2 here (https://github.com/djrenren/rust-test-frameworks), and @alexcrichton suggested that maybe it is worth it to just split this off tree to allow us to get more experience with these APIs and iterate quicker than if we do things in tree, and need to, e.g., fill in an RFC first for some API to customize libtest output formats or allow user-defined test types to interact with libtest formatting.

[alexcrichton]

I personally think this is a fantastic idea. The libtest crate deserves a lot more love than it’s gotten and I think this’d be a great vehicle for enabling that!

[djrenren]

I am 100% on board with this initiative, but I think we could go for a slightly different crate layout:

Currently we rely on libsyntax because the test macros can appear in non-standard order. If we modify the libsyntax implementation of #[test] to canonicalize this order then defer to a standard macro in libtest, we could just cleanly externalize libtest.

That is, if we make libsyntax convert:

#[ignore]
#[test]
fn ...

into:

#[test]
#[ignore]
fn ...

then a standard proc macro can process things.

This means the entirety of libtest (types, command line parsing, formatting) can exist as an external library though we still ship it with the compiler.

This will allow users to override the internal libtest with a crates.io implementation a la Cargo renaming:

[dependencies]
test = {package= "..."}

In this world, we can just ship a pinned version from cargo with the compiler. If users want to experiment, they just override the version used. This will also allow for easy integration of libbench because it could implement the requisite traits from libtest to be properly handled as a test case.

I think this approach is preferable because we avoid the maze of crates (especially libtest2. I already have enough nightmares about proc_macro2).

Does this seem reasonable?

[alexcrichton]

I’m not sure I completely follow the trickiness with #[test] and #[ignore] orderings, but it seems to me like we could make both of them procedural macros in the prelude by default perhaps? If both #[test] and #[ignore] were routed to the same procedural macro, it wouldn’t matter which runs first and #[ignore] would just otherwise verify that #[test] exists (removing it).

In any case I think moving towards a world where #[ignore] isn’t special and #[test_case] (the current unstable attribute I think?) is the only thing that’s special.

[djrenren]

Ah yeah that actually seems doable! My only other concern is that we’d be shipping another rust parser with the compiler which seems… not the best.

I suppose we could create two versions of the macro: one that uses libsyntax to ship with the compiler and one that uses syn?

I’m not sure what’s preferable.

[alexcrichton]

Oh I’m thinking that #[ignore] becomes a procedural macro like #[test] is today, defined in libsyntax. Custom test frameworks would likely all be based on syn, however

[alercah]

This is already easily implemented with proc macros. Proc macro attributes see future attributes, so all you need to do is make #[ignore] move itself to after #[test] if it sees #[test], and otherwise delete itself.

[petrochenkov]

I don’t understand, what’s the problem with #[ignore]? Why it has to become a macro?
It’s an inert (built-in) attribute, it should be visible to #[test] regardless of its position.

#[test] // receives `#[ignore] fn f() {}` as input 
#[ignore]
fn f() {}

#[ignore]
#[test] // still receives `#[ignore] fn f() {}` as input 
fn f() {}

[alercah]

That’s true, but there’s no particular reason to leave it like that I think. If we want to move libtest to use custom_test_frameworks, then leaving #[ignore] as a magic built-in inert attribute seems like a poor choice to me.

[petrochenkov]

no particular reason to leave it like that

Backward compatibility at least?
(I mean additional macros named ignore can be introduced, but the built-in attribute need to stay for compatibility.)

(Also, you can write #[ignore = "reason"] now, which is not allowed for macro attributes.)

[alercah]

Why can I attach #[ignore] to anything and it’s only a warning and not an error? T_T That might be worth a crater run on honestly, given that as far as I know it’s meaningless outside of tests?

And I thought that the = "reason" form was permitted for proc macros, but apparently I was wrong on that point.

[gnzlbg]

@djrenren could you open a tracking issue for this in rust-lang/rust ?