Long ago, somewhere, I picked up the habit to run benchmarks in liballoc like this:
python x.py bench --stage 0 src/liballoc
But, even leaving out the --stage 0 src/liballoc, that stopped working after PR #67281 because the compiler "cannot find function black_box in this scope".
By the way, the same happens for testing with --no-doc:
python x.py test --no-doc
Am I not using correct command lines here? Is no one else benchmarking? Has everyone been thinking like me "this seems so simple that surely someone is already fixing it, let me just hide these tests for the moment" - for the past month?
I wouldn't be too surprised that the benchmark is broken. In my experience, very few people run them. Issue #54176 suggests testing them in CI to ensure they don't get broken, which seems like a good idea to me. Running cargo test --benches or something like that should only take a few extra seconds of time (it'll run one iteration of each benchmark).
I think the change would go around here. Unfortunately due to the way target selection flags work, Cargo will need to be run twice (once with doctests, once without). An alternative would be to mark each benchmark with test=true in Cargo.toml, but that has the hassle of listing out all of the targets manually.
I didn't expect a simple explanation and really didn't expect commits that don't compile to pass through the system. It's not very obvious to notice:
./x.py check: completely ignores benchmark files (they don't even have to exist)
./x.py test: reports syntax errors in benchmark files, but not undefined symbols
./x.py test --no-doc: tries to compile and test benchmarks
./x.py bench: tries to compile and run benchmarks
Whatever the pr systems do, it doesn't include compiling or checking. However, rest assured that tidy will pull the brakes if you haven't properly formatted your non-compiling benchmark!
It's not as simple as above... Currently there seems to be some issue such that ./x.py test --no-doc compiles the benchmarks fine but ./x.py bench does not.