C/C++ header files are awful, but the one big advantage they have over Rust is that they allow code to be compiled before or in parallel with its dependencies. In practice, we’ve noticed that its easy to get long chains of dependent crates and “bottleneck” crates (i.e. crates which take a while to compile which lots of other crates depend on). The net effect is the overall build becomes significantly serialized.
I’ve been wondering how practical it might be to introduce a new crate type that has enough of the public interface of the crate to allow dependent crates to compile against it. The assumption here is that generating such an interface crate is much faster than full compilation (akin to a check build).
Ideally the work done by the interface build could be reused to build the real crate, so that the overall serial compilation time doesn’t increase much. But for me that’s a secondary concern - I’m more interested in unlocking more parallelism.
Is this worth pursuing? Is there any work underway which addresses this problem, perhaps in a different way? Incremental builds and rustc’s internal parallelism help a bit, but they’re still serial in Amdahl’s law terms - I’d like to move more onto the parallel side.
The ideal answer would be something like “oh yes,
.rmeta files already have everything that’s needed”, but I suspect that’s not true because inlining and macros make internal implementation details blur across crate boundaries.
Other properties I’d like:
- Produces identical executable/rlib output to when not using this mechanism (i.e., like incremental builds it’s a build-time optimization which doesn’t affect the generated artifacts).
- Works in all compilation modes (ie, not just restricted to dev builds). Release build times matter too. (Though a dev-only intermediate state would be fine given the previous constraint.)
Ideally (definitely on the “nice to have” end of the scale") the interface crates would be invariant if implementation details change which don’t affect the public interface (including inlined things), so that there’s less rebuilding. Though really this is just a special case of incremental builds, and its hard to see how this would work without revamping the SVH mechanism. I guess we could have a notion of “SVH of the public interface” and use that to express crate dependencies.