Hey folks. I’m wondering about the status of #[bench]. This is the feature that is preventing me from migrating to stable rust, since benchmarks are a development workflow–my code needs to be fast, and I run cargo bench a lot to find out if I’m fast yet or not. As a consequence I have to keep nightly on the development machines.
The stabilization metabug says the API has “known problems” and that we “want more pluggable framework”, but what that means in practice, I have no idea.
What is the problem exactly with the existing API?
Is there a shovel-ready solution to solve that problem floating around somewhere?
How will we recognize a solution to this problem if we see it?
Who owns this feature? Is it unowned?
One other tidbit I’ve dug up in my research–this feature was discussed in March but the conclusion of that discussion seems to be “punting” and “we should probably make a FAQ about this”. It does seem to be punted on, but if there’s a FAQ, I can’t dig it up.
Since this is the last mile to get to stable Rust, I’m at least somewhat motivated to make a go at it, if I can wrap my head around the problem and the parameters for an acceptable solution.
The status is, “needs more design work before it’s stable.” Off the top of my head, I can’t quite rememeber the exact problems that we want solved here, though, maybe someone else will chime in on that.
You can make it work with stable Rust by using Cargo feature flags to only run your benchmarks with a flag, and then only use that flag on nightly.
Multirust as suggested above. Although I'd suggest a bit different path: instead of keeping stable + nightly, you can build out of 1.0.0 tag as nightly and install it. This way you'll be sure bench numbers are real as nightlies include codegen changes. Here is a working sample:
cd rust
mkdir build
cd build
../configure --target=armv7-apple-ios,i386-apple-ios,armv7s-apple-ios,aarch64-apple-ios,x86_64-apple-ios --release-channel=nightly --disable-jemalloc
make -j4
make dist
cd dist
multirust update release-compat-ios --installer rust-docs-nightly-x86_64-apple-darwin.tar.gz,rustc-nightly-x86_64-apple-darwin.tar.gz
Note that --release-channel=nightly allows all unstable features including bench although you'll have warnings anyway.
Note also that you'll need manually copy cargo to ~/.multirust/toolchains/release-compat-ios/bin - it can be a cargo from nightly or release.
Later on you can select that toolchain by running multirust override release-compat-ios in project dir or multirust default release-compat-ios for global change.
The primary reason for not stabilizing #[bench] as-is is the requirement of extern crate test. Right now we only have one stable crate in the main distribution, std, and all other crates are not accessible. We could in theory expose only a small slice of the test crate (that which is necessary for benchmarking), but it’s still reserving the very nice name of “test”.
In my perfect world, I would envision this feature stabilized like so:
Writing a benchmark would not require importing a crate by default (like writing a unit test).
A different benchmarking or testing framework could be swapped out for the standard suite for different options and/or output.
A few ideas have been floated for dealing with the first point, but dealing with the second would require some more serious design, and you can find some more information in the RFC issue: https://github.com/rust-lang/rfcs/issues/816
While we are on the subject, html5ever has an issue about wanting back the old metrics in JSON format. So it would be great if Bencher could be extended to do that.
I’ve read through the history on this and it seems like it might be a perfect-is-the-enemy-of-the-good situation. The existence of [bench] in unstable Rust means that people have a preference for [bench] style tests. But, the need for benchmarking on stable Rust means that we can’t use [bench].
I actually think that [bench] adds very little value compared to just delegating all benchmarking to third-party benchmarking crates and doing things more manually, so I would be fine with deprecating [bench] and encouraging people to factor out its code (or use other code) to create their own benchmarking crates.
Or, if people feel like [bench] is really important, then maybe we can accept the current state of things as good enough? IMO the biggest problem with [bench] is the need for manual use of black_box, but that doesn’t even seem to be the issue blocking stabilization.
Either way, it would be good to have a decision soon so that people can move forward.