Next steps for reducing overall compilation time

[crlf0710]

Quite a few people investigating the possibility of using Rust in their work are interested in using and distributing precompiled binary crate dependencies (instead of source code) inside their corporation. Please take such kind of possibility into consideration whenever possible.

[nikomatsakis]

Interesting. I am definitely interested in this, especially the “unlocking” part. At the same time, as you say, it is an interesting joint venture that extends beyond the compiler team and into (e.g.) infrastructure. It feels like we need to start on a more complete sketch of what the moving parts are – though your e-mail has a lot of great specifics. Ideally, we’d try to gather up a reasonable set of people and pull that sketch together soon, before the Rust 2018 All Hands, so that we can use the All Hands time to do more of a deep dive.

[mitsuhiko]

Having something like incredibuild would be amazing for rust projects. I think a lot of frustration about the rust compiler performance is felt by large projects and teams which also could trivially pool processing power.

[eddyb]

Finally the story for incremental in rustc doens’t integrate well with sccache -like strategies or otherwise external caches.

I think in general, external caching tools are not that useful of a model here, and only Cargo, at most, could interface with the compiler closely enough to orchestrate any sort of complex compilation “dataflow” graph.

(I also think it’s kind of sad Cargo still doesn’t compile crates.io dependencies user-wide, or at least in a configurable fashion, but that’s another story)

The reason tools like ccache exist is entirely historical, IMO, and it has to do with the often opaque architecture of C compilers, and the simplistic file-based compilation model, meaning it was much easier for a third-party program to pretend to be a C compiler while acting as a “caching proxy”.

To soundly do this for Rust, you need the compiler’s help, and considering how incremental works inside the compiler (the “query” system), you can build much more flexible systems, depending on where/how you’re willing to split things up.

For example, you could have a purely incremental online cache of many crates.io crate*version combinations (including popular choices for feature flags), with rustc sending requests for any external information it requires. This might seem slow at first, but it’s locally cacheable, and could give you really quick starts (think IDE code browsing with type information, or a REPL).

[nikomatsakis]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.