# Measuring compiler performance **URL:** \ **Category:** tools and infrastructure\ **Created:** [March 16, 2017, 8:58pm UTC](https://internals.rust-lang.org/t/measuring-compiler-performance/4966 "2017-03-16T20:58:16Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![nikomatsakis](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nikomatsakis/32/5410_2.png) [@nikomatsakis](https://internals.rust-lang.org/u/nikomatsakis)\ **Post date:** [March 16, 2017, 8:58pm UTC](https://internals.rust-lang.org/t/measuring-compiler-performance/4966/1 "2017-03-16T20:58:16Z")
So, sadly, the `perf.rust-lang.org` website has been broken for quite some time (basically since rustbuild landed). @Mark_Simulacrum has done some awesome work (which they can describe) building up a new system that uses the pre-built binaries produced by travis. This offers the promise of very precise info about which PR triggered a regression. I’d like to spark a discussion on two topics: 1. What are the minimum steps we can take to get **some kind of results available again?** - I really dislike having no measurements - Is it just a matter of needing to get the old server up and going again, or what? 2. How should we structure our test suite? @Mark_Simulacrum and I have had quite a few conversations on the second point and it seems like a good idea to get broader input. My current take is that there are roughly four kinds of measurements I would like: - Compilation times that target very specific parts of the compiler and workflows - this includes incremental flows, i.e., build from scratch, apply diff, etc - Regression tests for known performance issues (kind of the same) - “Real-world” tests that correspond to frozen versions of actual crates - this includes incremental flows, i.e., build from scratch, apply diff, etc - Performance of generated code - not currently measured at all; obviously somewhat different from compilation time, but perhaps can share infrastructure Our [current set of compilation-time benchmarks](https://github.com/rust-lang-nursery/rustc-benchmarks) has grown somewhat organically and includes a smattering of the above categories. Perhaps we should carefully review them? I made a [brief stab at set of runtime benchmarks](https://github.com/nikomatsakis/rust-runtime-benchmarks) as well but that never _quite_ got off the ground. Some more suggestions for entries there would be helpful. Thoughts? ---
**Author:** ![leonardo](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/leonardo/32/1344_2.png) [@leonardo](https://internals.rust-lang.org/u/leonardo)\ **Post date:** [March 16, 2017, 9:09pm UTC](https://internals.rust-lang.org/t/measuring-compiler-performance/4966/2 "2017-03-16T21:09:29Z")
Those benchmarks also need to include the maximum RAM memory used during the compilation, and the size of the resulting binaries. 80-90% of the times I update the Nightly compiler (64 bit Windows, and I update it daily when possible) I see the binaries grow a little (some months ago the binaries shrunk hugely in one update, and we’re very far from regaining that lost binary weight, so it’s not a big problem). ---
**Author:** ![michaelwoerister](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/michaelwoerister/32/3557_2.png) [@michaelwoerister](https://internals.rust-lang.org/u/michaelwoerister)\ **Post date:** [March 17, 2017, 1:23pm UTC](https://internals.rust-lang.org/t/measuring-compiler-performance/4966/3 "2017-03-17T13:23:01Z")
- I think there is value in measuring all four of those categories. - There should be some pruning of current test cases, we have two versions of `regex` in there, for example. - The visual presentation of performance numbers could be a lot better. More clearly separating the above categories would probably go a long way towards addressing concerns about mixing them. One more thing to keep in mind is that Travis nightlies have debug assertions and LLVM assertions turned on, I think. This skews measurements a bit. ---
**Author:** ![retep998](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/retep998/32/729_2.png) [@retep998](https://internals.rust-lang.org/u/retep998)\ **Post date:** [June 12, 2017, 10:15pm UTC](https://internals.rust-lang.org/t/measuring-compiler-performance/4966/4 "2017-06-12T22:15:09Z")
I’m still holding on to the hope that `winapi` gets added to the set of crates being measured. ---
**Author:** ![nikomatsakis](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/nikomatsakis/32/5410_2.png) [@nikomatsakis](https://internals.rust-lang.org/u/nikomatsakis)\ **Post date:** [March 25, 2019, 8:28am UTC](https://internals.rust-lang.org/t/measuring-compiler-performance/4966/5 "2019-03-25T08:28:11Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.