PR: Less dependency on correct mtimes

(Ignore the quality of the PR - it's WIP, focus for now on the intent)

I'm keen on speeding up the compiler by not compiling at all.

I'm working on a rustc/cargo PR ( https://github.com/rust-lang/rust/pull/75594 , https://github.com/rust-lang/cargo/pull/8623 ) to try and reduce our depedency on correct mtimes and I wanted a bit more visibility / debate on what I'm trying to do.

Problem: When mtimes work well they work really well. When they are upset by external processes cargo falls back to a full rebuild even though the content hasn't actually changed.

For the simple cases mtimes work nicely, but when dealing with more convoluted [corporate] situations mtimes have a hard time with:

  • docker-like things with stashed layers stored at rounded precision.
  • conda-build isolated environments
  • other? I'm sure the above aren't the only way to mess up mtimes if we try hard enough!

Mtimes can't always be relied upon as things get moved about and unziped. It would be great if, even in those circumstances, cargo/rustc could still realise it didn't need to rebuild.

I don't want dodgy mtimes to stop me falling into the cached pit of success.

To this end we already hash all the source files and it seem sensible to record those already computed hashes and take advantage of them when checking whether a rebuild is needed in Cargo. (Potentially this PR is already a positive improvement on the status quo).

I'd like to do the same thing with binary dependencies and the suggestion is to add SVH to the output binaries by default so that we have a consistent hash to compare against. It will take a bit of platform dependent fishing to extract it out again, but it should be considerably faster than hashing a file potentially large binary file.

Anyway as it's a bit more of an adventurous PR than anything I've done before so I thought I'd get a few more eyes and minds on it.

2 Likes

Do you have a suggestion for how to force a rebuild through cargo? That is something I had to do a few times for benchmarking purposes. I also have to do this when working on Miri with a locally built compiler, and the compiler changed -- cargo doesn't realize this on its own.

The tried-and-true method is touch src/file.rs && cargo build; it sounds like your approach will make that not work any more. (For benchmarks this is already not reliable but it works most of the time.) This is not a blocker, but it needs careful documentation.

1 Like

It sounds like there should be a --force option to support such use cases in a more robust manner.

1 Like

cargo clean && cargo build will rebuild

1 Like

I often just remove the rlib/executable that I want to have recompiled. rm target/debug/deps/libfoo-*; cargo build.

Cargo has a compile_opts.build_config.force_rebuild which is currently only used for cargo fix. I don't think it would take much to expose it if there was demand.

1 Like

Yeah that sounds great. :slight_smile: