Currently benches are compiled as part of the crate they’re defined in. However, for a library this could produce “false excellence” as this would give it all the metadata for full inlining and optimization of all functions, where a user of the library can only do this for #[inline] functions (if I understand correctly). It consequently also becomes harder to evaluate the full consequences of introducing #[inline] using a library’s benchmark.
Therefore, it would be nice if there was some way to force a benchmark (or module of benchmarks) to be built and run as if it was in an external crate. Having to split out benches to a separate crate manually is a fairly onerous task that reduces maintainability.
Not quite the same thing, but cargo can use a "benches" directory, and each rust file in there is compiled as a separate crate, and can be run with "cargo bench"
I also find it weird that benches are compiled with tests, and that there is no separate rustc --bench. I think I remember @huon saying that this was because of the hacky way that they were added in the first place… (I could be wrong). Dunno if it it is RFC worthy though.
I don't think I ever said they were hacky; I may've just pointed out that they are managed by the test harness (they're "unit benchmarks", like "unit tests").