Edition 2027: Stop the tests!

tools and infrastructure cargo
1

TL;DR: cargo test is compiling too much, for nothing.

State of the Art

I was running cargo test again today, and again I found myself sighing as cargo test was recompiling everything, for nothing.

While the test binary that is produced only contains tests for the current crate, cargo will still instruct rustc to build all the upstream crates in test mode just in case:

  1. Said crates happen to export more items in test mode, some test-specific API.
  2. Upstream crates happen to use said test-specific API.

Nifty, ain't it?

Frequency

While the above is convenient, for anyone using a test-specific API, I suspect the reality is that:

  1. Very few crates export any test-specific API, or behave in any way differently when compiled in test mode.
  2. Even fewer crates use any test-specific API from an upstream dependency.

This is certainly true for the codebases I maintain at work. Over the few hundreds of crates I maintain:

  • Maybe a handful export a test-specific API.
  • Maybe a handful use a test-specific API from an upstream dependency, and none come from 3rd-party dependencies.

Which leads me to questioning whether forwarding the "test mode" is the right choice, and whether it should not, instead, be opt-in.

Let's not!

I don't know about you, but I'd be quite happy to save up half my disk space, and a good chunk of compilation time!

And in a world where crates compiled in test mode depended on the regular version of their dependencies, it would just happen...

... but of course that'd break backward compatibility. Damned!

Automatic Detection

Detecting whether compiling a crate with or without test mode makes any difference may actually be... complicated.

If this could be done, then cargo could write some metadata next to the local copy of the source files indicating there's no need to re-compile that crate in test mode -- it's all the same. Sweet, no?

But I have great doubts:

  1. It would require compiling in test mode at least once. May not seem like a big deal, but CI without caching wouldn't benefit.
  2. It would require a reachability analysis, comparing not only the API, but also the implementation of reachable items.
  3. It would notably require ensuring that nobody smuggled a reachable trait impl in test mode.

Explicit opt-in

Unless someone has a miracle solution for the above, the best solution may be to switch to opt-in propagation on an edition boundary, such as edition 2027.

A crate in edition 2027 compiled in test mode would, by default, depend on the non test-mode versions of its dependencies.

There would then be an option in Cargo.toml to enable test mode on a per-dependency basis when compiled in test mode.

3 Likes
2

I suspect something else is going and more details on the problem would be helpful.

Cargo does not build any of your dependencies with --test / --cfg test (and in fact this is something some people wish for). Instead, only test binaries (tests/*, src/ib.rs, src/bin.rs) are being built with --test. We do sometimes build examples and binaries for integration tests which some complain about for building too much.

My best guess if all dependencies are being rebuilt for cargo test compared to cargo run, then there is a dev-dependency that is activating a feature that is causing it.

13 Likes
3

This is not true, only the final crate is compiled in test mode.

4

I've definitely run into this. A dev-dependency affects the features of something deep in the stack, and a few hundred dependencies rebuild.

3 Likes
5

Yeah, same here -- I spent quite a bit of time working around this in Miri. We ended up always building everything with --all-targets, which overall saves a lot of time.

3 Likes
6

The workspace hack should be able to solve this too I would think. See e.g. crates.io: Rust Package Registry for a tool for automatically managing that.

7

Yeah I looked into various solutions and none of them worked well for Miri, including the automatic workspace hack. (Miri is inside a workspace in the rustc repo so we can't have a workspace in the Miri repo, and cargo-hakari requires a workspace.)

(Btw your link doesn't work, it should be https://crates.io/crates/cargo-hakari)

8

That is a shame. Nested workspaces would be extremely useful.

(Thanks, fixed the link. My phone keyboard ate the last character somehow, I think.)

9

I've hit this too when using workspaces. It's not specific to tests, only that building an individual package and building the whole workspace can activate different sets of dependencies.

I try to use [workspace.dependencies] as much as possible to ensure I enable the same feature flags for the same dependencies. However, that's not always sufficient, because in some cases merely a crate not using a dependency can affect which features of transitive dependencies get enabled.

I don't know if Cargo can do anything about it, other than more precisely warning about the issue, and offering a setting. Forcing unification can avoid recompilations, but unification is undesirable in some cases, e.g. no-std libraries don't want std features enabled.

Feature unification is also theoretically already a problem for tests, because your tests are running with different features than the features which will be enabled in libraries built outside of the workspace.