Should PGO automatically compile hot code O3 and cold code Oz?

1

For example, by manually splitting the hot code of a simple benchmark into a library and compiling it at O3 while compiling the rest as Oz we end up with a binary that is strictly better than both Oz and O3... smaller than Oz and faster than O3!

Configuration                    Bin Size     Iterations   Primes  Time (ms)
------------------------------ ---------- --------------  ------- ----------
Oz-LTO                             793408    35926266643  7675094  300000.05
O3-LTO                             910824    51372490784  9711920  300000.00
Oz-bin-O3-lib-LTO                  791640    52860924508  9896242  300000.01

While it should be very easy for PGO to find the hot code, it does not seem to use this information. With PGO Oz/s make small binaries, but they do not reach 50B iterations. O1/2/3 Make large binaries.

Configuration                    Bin Size     Iterations   Primes  Time (ms)
------------------------------ ---------- --------------  ------- ----------
Oz-PGO-LTO                         796480    39338956236  8147490  300000.02
Os-PGO-LTO                         793240    45096789314  8913665  300000.02
O1-PGO-LTO                         961408    40714684893  8333863  300000.01
O2-PGO-LTO                         895872    55387455174 10204971  300000.02
O3-PGO-LTO                         906288    55252916198 10188636  300000.04

Manually splitting hot-code into crates has a number of issues. Firstly, refactoring is work, and programmers are lazy. More importantly the hot-code in your app may be spread amongst a number of upstream crates and forking upstream crates just to split out hot code creates a lot of technical debt. Note below that the speed attribute does not seem to be sufficient, and even adding 1 line to upstream still requires a fork, which is still suboptimal You could try to convince upstream to take your hot-code patches, but areas of their library that are hot in your application might not be hot in others.

It may be possible to do all sorts of clever things. For example, tepid code could be compiled Oz if it is run only once each time it is loaded into cache, to reduce load on the cache, while bursty tepid code could be compiled with something like O2. However a trivial 'hot code should go fast' heuristic would seem to be very useful. Perhaps a -Oa (auto) or -Op (PGO) should be used to specify this mode.

3 Likes
2

Note that anything here would basically be an LLVM change, so you might want to raise it with them. I don't think rust itself actually uses the PGO information in any meaningful way. (At most it forwards things along.) I don't think it needs to be a different opt level; PGO can just make other choices with the information it has when you use it.

2 Likes
3

I was thinking that having this on by default might cause performance regressions where the profile doesn't cover every real world use-case. Anyway, I will talk to LLVM.

4

I think I'm, personally at least, not that worried about it because it's easily fixable: if you have something where you end up with bad performance, you just need to include that scenario in the PGO training, which is what you already want when using PGO anyway.

1 Like
5

Apparently "There is already an LLVM option to do this (-mllvm -pgo-cold-func-opt=optsize, or minsize for -Oz behavior)." -- Should PGO automatically compile hot code O3 and cold code Os? - IR & Optimizations - LLVM Discussion Forums

3 Likes
6

Flakebi suggests the pgo-cold-func-opt option is implemented in clang and opt but Rust would need to wire this up into where it sets PGOOptions.

1 Like
7

I have knocked together a very rough prototype for testing purposes only. However, LLVM doesn't seem to support what we want to do. It seems that LLVM only marks 15% of our functions as cold, despite making a benchmark where the load is focused only on a couple of functions. Also minsize and attrspeed etc. don't seem to have much effect. For example as shown below, marking the hot functions with the nightly speed attribute doesn't seem to come close to compiling them with O2/O3. Similarly applying the optsize/minsize attributes seems to only save a couple of KB.

Configuration                    Bin Size     Iterations   Primes  Time (ms)
------------------------------ ---------- --------------  ------- ----------
Oz-attrspeed-LTO                   793408      354929995   368696    3000.00
Oz-bin-O2-lib-LTO                  791640      494633874   458460    3000.01
Oz-bin-O3-lib-LTO                  791640      487291591   453980    3000.00