Feedback request: Performance improvements from reducing debug info

There are many potential improvements we could make for build times. In the course of evaluating possibilities, we'd like to know how much impact debug information has on build time.

Could people provide feedback on the build time differences in their projects between the default dev profile configuration and with the following added?

[profile.dev]
debug = "line-tables-only"

Ideally, we'd love to get the performance delta for a couple of cases:

  • A clean cargo build after cargo clean.
  • After doing cargo build, using touch on your lib.rs or main.rs, and then doing cargo build again.
1 Like

I tried my Project Euler solutions with 340 binaries (for each I've solved) and 70 dependencies. This is potentially heavy as it involves lots of linking with debuginfo, but they're also pretty small.

  • With the default profile, cargo build took 23.34s; after touching lib.rs, it took 11.17s.
  • With "line-tables-only", cargo build took 20.84s; after touching lib.rs, it took 10.00s.
3 Likes

For one of my projects, a clean default dev profile gives:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 12s

A clean debug = "line-tables-only" build:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 02s

With the default profile, after touching main.rs/lib.rs:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.59s

With debug = "line-tables-only", after touching main.rs/lib.rs:

Finished `dev` profile [unoptimized + debuginfo] target(s) in 14.21s
  • default profile:
    • cargo clean then cargo build: 7.36s
    • touch main.rs then cargo build: 1.09s
  • with debug = "line-tables-only":
    • cargo clean then cargo build: 6.62s
    • touch main.rs then cargo build: 0.86s

It would perhaps also worth including times for split-debuginfo which could presumably impact link times?

1 Like

There are also other factors to consider such as the linker used and the OS

1 Like

Better linkers can make this better, but it still takes time to do the work.

I mean if commenters here posted those tidbits (os, linker, split debuginfo) it could provide more context in case there are discrepancies in their numbers.

Testing on Fedora 40 x86_64.

Default dev profile:

Post cargo clean: cargo build 385.75s user 34.70s system 599% cpu 1:10.08 total

After touching main.rs: cargo build 3.91s user 1.23s system 99% cpu 5.151 total

With debug = "line-tables-only"

Post cargo clean: cargo build 353.69s user 34.42s system 619% cpu 1:02.62 total

After touching main.rs: cargo build 2.86s user 1.02s system 99% cpu 3.881 total

1 Like

ubuntu + mold

Less chonky executable, default profile clean build followed by touch on lib.rs and one more build

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 57s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 7.16s

Same, but with line-tables-only

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 46s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.80s

More chonky executable, all 4 cases

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 22s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.68s

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 03s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 3.29s
5 Likes

Ubuntu-based with Wild as the linker

Smallish build (10MiB binary):

default cold: 22.49s
line-tables cold: 20.78s
debug=0 cold: 19.66s

default warm: 0.24s
line-tables warm: 0.20s
debug=0 warm: 0.18s

Larger build (several binaries around 450MiB each):

default cold: 4m 51s
line-tables cold: 3m 46s
debug=0 cold: 3m 16s

default warm: 4.48s
line-tables warm: 2.28s
debug=0 warm: 1.63s

I recently ran an informal survey about whether people regularly use a debugger. Of the 89 responses, 18% say they sometimes or always use a debugger while 82% say they rarely or never use a debugger. This seems consistent with what I've observed from talking to rustaceans in person.

I'm definitely supportive of any change to reduce the debug info for a default build.

I think that line number information in backtraces is less important now that it used to be and the reason is #[track_caller]. Generally the line that I care about when I get a backtrace is the line that called unwrap, expect, panic etc and these days that line is reported whether or not you have line debug info. So personally, I set debug=0, but I can understand that's likely a step too far for the default - just changing to line-tables by default would be great.

3 Likes

Does line-tables-only debug information still provide enough detail that GDB's bt command (or equivalent in your favorite debugger) can print the values of function arguments? I would really miss that if it went away by default.

1 Like

If we decide to focus the dev profile on check/test iteration time, how do we do it without negatively impacting people using a debugger?

Challenges:

  • profiles operate on the workspace level
  • dev profile uses the debug target dir
  • you cannot have a built-in profile inherit from another

Some tools we have for evolving this include:

  • We could add a build.profile to override the default profile
  • We could add workspace editions to evolve profiles
    • How do different editions at the workspace and package level mix?
    • What is the behavior like on publish/vendor?
    • How does a workspace edition interact with inheriting an edition?
    • Some things we do with editions may not be able to be warned about / migrated
  • We can split dev / debug profiles
    • debug profile name is reserved
  • We can add cargo:: prefixed profiles that are the true built-in profiles and immutable, relaxing the constraints on root level profiles

Example Solutions

Change dev profile

dev has the following changes:

[profile.dev]
-opt-level = 0
+opt-level = 1
-debug = true
+debug = "line-tables-only"
split-debuginfo = '...'  # Platform-specific.
strip = "none"
debug-assertions = true
overflow-checks = true
lto = false
panic = 'unwind'
incremental = true
codegen-units = 256
rpath = false

Benefits

  • cargo test is sped up

Downsides

  • Users have to go out of their way to setup debugging

Split the dev / debug profiles + override the default profile

Changes:

  • The current dev profile settings are mirrored over into debug.
  • dev becomes the profile shown earlier, in the dev directory
  • dev profile remains the default
  • We add build.profile to allow a user or IDE to set debug as the default

Open question:

  • Do we add a --dev flag or tell people to use --profile dev?

Benefits

  • cargo test is sped up

Downsides

  • Anyone doing a cargo build and looking in target/debug will be broken.
  • Debugging requires a manual rebuild or to change the config

Variants:

  • We split the profiles on the next edition (requires workspace editions)
  • debug becomes the default profile (build.profile), maybe changing to dev on the next edition (requires workspace editions)
    • But artifact location changes on cargo build --profile dev from target/debug to target/dev
  • We add cargo::* profiles so people can more easily make dev and debug whatever they want (e.g. dev inheriting from cargo::debug)
1 Like

No, only -Cdebuginfo=2 emits the necessary information for that.

I think this will only be a net win when tests are taking a non-trivial amount of time already. If you only got a couple of tests or are only doing say cargo run for a web server that starts up instantly, it would likely only slow down things.

My performances:

  • normal dev profile: 14.30 secs
  • with line-tables-only: 12.38 secs

I can't access touch as I am on windows :frowning:

I wonder if it would be feasible to provide automatic tuning. Imagine if cargo test would:

  • Measure both the compile time and the test run time, and remember this over multiple sessions.
  • If historical run time is high relative to compile time, run an experiment (with a note to the user that this is happening): switch to opt-level = 1.
  • Based on the results of the experiment, decide whether to keep it — and maybe suggest to the user to configure [profile.dev].

The biggest problem with this would be that it triggers a rebuild of all dependencies, which is a potentially large interruption. This would be less of a problem if the default dev profile always compiled dependencies with more optimization, so only the workspace needs to be rebuilt.

Alternatively, you could ask Cargo to run the experiment explicitly and choose the best settings, as a single command, but this has the problem that it doesn't know what kind of edits you're going to be making and thus it can't reproduce the right incremental build cost (unless it saves copies of your edited source files)

That seems like an artificial limitation to me. Is there any inherent reason such support couldn't be added?

Could cargo automatically make the distinction between outside-workspace dependencies (since those are unlikely to need recompiles) and apply a higher opt-level to those so that people don't have to set it via [profile.dev.package."*"] manually?

2 Likes