# A path forward towards re-usable libtest functionality, custom\_test\_frameworks, and a stable #\[bench\] macro

**URL:** <https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139>\
**Category:** libs\
**Created:** [January 3, 2019, 7:50pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139 "2019-01-03T19:50:00Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![gnzlbg](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gnzlbg/32/3858_2.png) [@gnzlbg](https://internals.rust-lang.org/u/gnzlbg)\
**Post date:** [January 3, 2019, 7:50pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/1 "2019-01-03T19:50:00Z")

</div>

Right now, `libtest` is in `rust-lang/rust/src/libtest`, and it implements both the stable `#[test]` system and the unstable `#[bench]` functionality (along with some other things). The standard `#[test]` harness relies on some libsyntax internals for interacting with other test related macros, which prevent it from being a “normal” `custom_test_framework` in the near future, but fixing this isn’t really high-priority, nor would it deliver a lot of value.

There are a couple of things we could easily do that are low effort, deliver instant value, and would help putting some long awaited features on the track towards stabilization (e.g. `#[bench]` !).

We could move `libtest` out of `rust-lang/rust`, e.g., to `rust-lang/libtest/libtest-shim` (and publish `libtest-shim` to [crates.io](http://crates.io)), such that the "one and only `libtest`" crate in `rust-lang/rust/src/libtest`just re-exports this shim. This would allow us to iterate quickly out of the `rust-lang/rust` bors queue.

We could then de-entangle the test framework from the benchmarking framework into three crates:

- `libtest2`: which uses `custom_test_frameworks` to expose the testing harness. Initially, this will still implement the `libtest` shims re-exported by `libtest-shim` into `libtest`, and will be less powerful than the `libtest` crate because it wouldn’t have access to `libsyntax` when used directly. Solving these issues will require improving `custom_test_frameworks` and other Rust features, but that’s something worth exploring.

- `libbench`: which uses `custom_test_frameworks` to implement `#[bench]` macro . The idea is to make `libbench` a 1:1 replacement for the nightly `#[bench]` attribute, so that we can, at some point, deprecate it, and once `custom_test_frameworks` are stabilized, `libbench` will work on stable.

- `libtest-core` (or similar): both `libtest2` and `libbench` share the `libtest` formatting framework, parts of argument parsing, some traits, types, etc. Initially, we would re-factor them into a `libtest-core` crate, that other `custom_test_frameworks` can use. The idea is to iterate on these APIs on [crates.io](http://crates.io), and at some point allow people to plug in, e.g., their own output formats that maybe even the normal `libtest` could use. It is hard to iterate on this in tree.

Initially, all of these changes will be non-functional changes, and will mostly benefit `custom_test_frameworks` on [crates.io](http://crates.io), but I hope that the experience we gain will help us land a stable `#[bench]` (and stable `custom_test_frameworks` in general) quicker.

This is a bit handwaved, but I wrote a quick and dirty PoC of the refactor of `libtest` into a `libtest+libcore` crate in this PR ([https://github.com/rust-lang/rust/pull/57068](https://github.com/rust-lang/rust/pull/57068)), @djrenren has an implementation of #[bench] as a custom test framework and a libtest2 here ([https://github.com/djrenren/rust-test-frameworks](https://github.com/djrenren/rust-test-frameworks)), and @alexcrichton suggested that maybe it is worth it to just split this off tree to allow us to get more experience with these APIs and iterate quicker than if we do things in tree, and need to, e.g., fill in an RFC first for some API to customize libtest output formats or allow user-defined test types to interact with libtest formatting.

---

<div class="post-metadata">

**Author:** ![alexcrichton](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alexcrichton/32/4501_2.png) [@alexcrichton](https://internals.rust-lang.org/u/alexcrichton)\
**Post date:** [January 3, 2019, 8:09pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/2 "2019-01-03T20:09:25Z")

</div>

I personally think this is a fantastic idea. The libtest crate deserves a lot more love than it’s gotten and I think this’d be a great vehicle for enabling that!

---

<div class="post-metadata">

**Author:** ![djrenren](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/djrenren/32/4543_2.png) [@djrenren](https://internals.rust-lang.org/u/djrenren)\
**Post date:** [January 5, 2019, 6:51pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/3 "2019-01-05T18:51:08Z")

</div>

I am 100% on board with this initiative, but I think we could go for a slightly different crate layout:

Currently we rely on `libsyntax` because the test macros can appear in non-standard order. If we modify the `libsyntax` implementation of `#[test]` to canonicalize this order then defer to a standard macro in libtest, we could just cleanly externalize `libtest`.

That is, if we make libsyntax convert:

```rust
#[ignore]
#[test]
fn ...

```

into:

```rust
#[test]
#[ignore]
fn ...

```

then a standard proc macro can process things.

This means the entirety of libtest (types, command line parsing, formatting) can exist as an external library though we still ship it with the compiler.

This will allow users to override the internal `libtest` with a `crates.io` implementation a la Cargo renaming:

```rust
[dependencies]
test = {package= "..."}

```

In this world, we can just ship a pinned version from cargo with the compiler. If users want to experiment, they just override the version used. This will also allow for easy integration of `libbench` because it could implement the requisite traits from `libtest` to be properly handled as a test case.

I think this approach is preferable because we avoid the maze of crates (especially `libtest2`. I already have enough nightmares about `proc_macro2`).

Does this seem reasonable?

---

<div class="post-metadata">

**Author:** ![alexcrichton](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alexcrichton/32/4501_2.png) [@alexcrichton](https://internals.rust-lang.org/u/alexcrichton)\
**Post date:** [January 7, 2019, 3:41pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/4 "2019-01-07T15:41:24Z")

</div>

I’m not sure I completely follow the trickiness with `#[test]` and `#[ignore]` orderings, but it seems to me like we could make both of them procedural macros in the prelude by default perhaps? If both `#[test]` and `#[ignore]` were routed to the same procedural macro, it wouldn’t matter which runs first and `#[ignore]` would just otherwise verify that `#[test]` exists (removing it).

In any case I think moving towards a world where `#[ignore]` isn’t special and `#[test_case]` (the current unstable attribute I think?) is the only thing that’s special.

---

<div class="post-metadata">

**Author:** ![djrenren](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/djrenren/32/4543_2.png) [@djrenren](https://internals.rust-lang.org/u/djrenren)\
**Post date:** [January 8, 2019, 7:16pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/5 "2019-01-08T19:16:05Z")

</div>

Ah yeah that actually seems doable! My only other concern is that we’d be shipping another rust parser with the compiler which seems… not the best.

I suppose we could create two versions of the macro: one that uses libsyntax to ship with the compiler and one that uses syn?

I’m not sure what’s preferable.

---

<div class="post-metadata">

**Author:** ![alexcrichton](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alexcrichton/32/4501_2.png) [@alexcrichton](https://internals.rust-lang.org/u/alexcrichton)\
**Post date:** [January 8, 2019, 10:04pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/6 "2019-01-08T22:04:13Z")

</div>

Oh I’m thinking that `#[ignore]` becomes a procedural macro like `#[test]` is today, defined in libsyntax. Custom test frameworks would likely all be based on `syn`, however

---

<div class="post-metadata">

**Author:** ![alercah](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alercah/32/4065_2.png) [@alercah](https://internals.rust-lang.org/u/alercah)\
**Post date:** [January 9, 2019, 3:49pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/7 "2019-01-09T15:49:54Z")

</div>

This is already easily implemented with proc macros. Proc macro attributes see future attributes, so all you need to do is make `#[ignore]` move itself to after `#[test]` if it sees `#[test]`, and otherwise delete itself.

---

<div class="post-metadata">

**Author:** ![petrochenkov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/petrochenkov/32/2784_2.png) [@petrochenkov](https://internals.rust-lang.org/u/petrochenkov)\
**Post date:** [January 9, 2019, 4:31pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/8 "2019-01-09T16:31:27Z")

</div>

I don’t understand, what’s the problem with `#[ignore]`? Why it has to become a macro?  
It’s an inert (built-in) attribute, it should be visible to `#[test]` regardless of its position.

```rust
#[test] // receives `#[ignore] fn f() {}` as input 
#[ignore]
fn f() {}

```

```rust
#[ignore]
#[test] // still receives `#[ignore] fn f() {}` as input 
fn f() {}

```

---

<div class="post-metadata">

**Author:** ![alercah](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alercah/32/4065_2.png) [@alercah](https://internals.rust-lang.org/u/alercah)\
**Post date:** [January 9, 2019, 4:33pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/9 "2019-01-09T16:33:22Z")

</div>

That’s true, but there’s no particular reason to leave it like that I think. If we want to move libtest to use custom\_test\_frameworks, then leaving `#[ignore]` as a magic built-in inert attribute seems like a poor choice to me.

---

<div class="post-metadata">

**Author:** ![petrochenkov](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/petrochenkov/32/2784_2.png) [@petrochenkov](https://internals.rust-lang.org/u/petrochenkov)\
**Post date:** [January 9, 2019, 4:35pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/10 "2019-01-09T16:35:06Z")

</div>

> [@alercah](#):
>
> no particular reason to leave it like that

Backward compatibility at least?  
(I mean additional macros named `ignore` can be introduced, but the built-in attribute need to stay for compatibility.)

(Also, you can write `#[ignore = "reason"]` now, which is not allowed for macro attributes.)

---

<div class="post-metadata">

**Author:** ![alercah](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/alercah/32/4065_2.png) [@alercah](https://internals.rust-lang.org/u/alercah)\
**Post date:** [January 9, 2019, 5:21pm UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/11 "2019-01-09T17:21:59Z")

</div>

Why can I attach `#[ignore]` to anything and it’s only a warning and not an error? T\_T That might be worth a crater run on honestly, given that as far as I know it’s meaningless outside of tests?

And I thought that the `= "reason"` form was permitted for proc macros, but apparently I was wrong on that point.

---

<div class="post-metadata">

**Author:** ![gnzlbg](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/gnzlbg/32/3858_2.png) [@gnzlbg](https://internals.rust-lang.org/u/gnzlbg)\
**Post date:** [January 17, 2019, 11:44am UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/12 "2019-01-17T11:44:21Z")

</div>

@djrenren could you open a tracking issue for this in rust-lang/rust ?

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\
**Post date:** [April 17, 2019, 11:44am UTC](https://internals.rust-lang.org/t/a-path-forward-towards-re-usable-libtest-functionality-custom-test-frameworks-and-a-stable-bench-macro/9139/13 "2019-04-17T11:44:21Z")

</div>

Closed on April 17, 2019, 11:44am UTC

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.