X.py: how to benchmark liballoc

Long ago, somewhere, I picked up the habit to run benchmarks in liballoc like this:

python x.py bench --stage 0 src/liballoc

But, even leaving out the --stage 0 src/liballoc, that stopped working after PR #67281 because the compiler "cannot find function black_box in this scope".

By the way, the same happens for testing with --no-doc:

python x.py test --no-doc

Am I not using correct command lines here? Is no one else benchmarking? Has everyone been thinking like me "this seems so simple that surely someone is already fixing it, let me just hide these tests for the moment" - for the past month?

1 Like

I wouldn't be too surprised that the benchmark is broken. In my experience, very few people run them. Issue #54176 suggests testing them in CI to ensure they don't get broken, which seems like a good idea to me. Running cargo test --benches or something like that should only take a few extra seconds of time (it'll run one iteration of each benchmark).

I think the change would go around here. Unfortunately due to the way target selection flags work, Cargo will need to be run twice (once with doctests, once without). An alternative would be to mark each benchmark with test=true in Cargo.toml, but that has the hassle of listing out all of the targets manually.

2 Likes

I didn't expect a simple explanation and really didn't expect commits that don't compile to pass through the system. It's not very obvious to notice:

  • ./x.py check: completely ignores benchmark files (they don't even have to exist)
  • ./x.py test: reports syntax errors in benchmark files, but not undefined symbols
  • ./x.py test --no-doc: tries to compile and test benchmarks
  • ./x.py bench: tries to compile and run benchmarks

Whatever the pr systems do, it doesn't include compiling or checking. However, rest assured that tidy will pull the brakes if you haven't properly formatted your non-compiling benchmark!

2 Likes

It's not as simple as above... Currently there seems to be some issue such that ./x.py test --no-doc compiles the benchmarks fine but ./x.py bench does not.

There's an alternative way to do it now. Not sure if it can be simplified more (sticking to linux notation, even though I benchmark on Windows):

  • ./x.py build src/liballoc --stage 1
  • rustup toolchain link latest-stage1 build\x86_64-unknown-linux-gnu/stage1
  • cd src/liballoc
  • cargo clean # otherwise things are not rebuilt properly
  • cargo +latest-stage1 bench --bench collectionsbenches

where latest-stage1 is some label you choose and x86_64-unknown-linux-gnu the build triplet.

Well that sounds complicated.

The issue was somehow resolved, and x.py --stage 0 bench src/liballoc works again.