[Idea] Cargo Global Binary Cache

[haohou]

Sort of similar, but it’s really nice to have a builtin cache with Cargo. Plus Cargo already have the perfect hash for this.

I think later we can even have the feature so that we can pin some binary in the cache, so that it just like install a lib.

The centralized binary cache will reduce the target directory size as well.

For Ad-hoc projects, I would like to keep each Ad-hoc project separate, they are really different things, and just want to build them separately.

[kornel]

Shared, machine-wide guts of target subdirectories would be great! In addition to saving lots of disk space and time, it would also fix my two main pet peeves about Cargo:

• macOS wants caches in ~/Library/Caches. Large target subdirectories scattered all over the disk cause unwanted churn in backups (Time Machine) and search (Spotlight), and tools to free up disk space don’t know how to deal with them.

• In Vagrant you want to place Rust source code on the network drive to share it with the host. You also never want to have cache on that slow network drive with unreliable locking and broken hardlinks.

[kornel]

BTW, is there a room to improve sccache integration? Currently sccache hooks up as RUSTC_WRAPPER, but it doesn’t really know what Cargo is doing, so it has to guess some details. For example, could Cargo give sccache some uniquely-identifying hash of the operation instead of sccache guessing it?

[matklad]

@haohou if you want to push this to the real implementation, I think it makes sense to open a PR to Cargo with rough design, to get the feedback from the team early.

Note that this is a pretty significant change to Cargo’s “API”, so, in addition to implementation work, some design/discussion/stabilization efforts will also be required.

[haohou]

Yes, I am happy to do that.

I think it makes sense to open a PR to Cargo with rough design, to get the feedback from the team early.

Just want to make sure you mean make a formal RFC documentation and let people discuss it, rather than the implementation PR to cargo.

[matklad]

you mean make a formal RFC documentation and let people discuss it, rather than the implementation PR to cargo.

I don’t think a full RFC is needed here (though, Cargo teem might disagree), as this is not a terribly large feature. However, when preparing PR, it might be a good idea to write some-kind of documentation for this feature, which explains what it does exactly, and explicitly mention points where design can go either one way or another.

For example, how would the user configure this shared dir? Will it be a flag like --target-dir, and env var, a config option? Will cargo use a shared dir by default, or you need to opt-into this feature?

The implementation side of things is “easy” in a sense that we can always adjust it later. The “interface” part is much harder, because you can’t change them later due to backwards compat.

[Eh2406]

I love to see more work in this space. I also appreciate having more people getting involved. So thank you!

In addition to the excellent questions @matklad layed out, we would also need to describe how this cache would be different then setting a single shared target directly with the CARGO_TARGET_DIR environment variable or .config file?

[haohou]

I think it would be better to have a cache, instead of a shared target dir.

Currently I have the code change for sharing the .rlib and .d. But I think it’s another problem is, the shared target folder actually consumes huge disk space. I think it should have a cache management mechanism.

For example, if the dependency is compiled only once, cargo make the output to the package’s target. And if cargo finally realized this binary is used several times, then it will cache the compilation result. Also, we can put a limit of amount of disk space the cache can use. If the binary haven’t been use for a while, we probably can just remove it without hurting the performance and just make it a LRU cache.

So it turns out probably a file like ~/.cargo/bin-cache.conf, which defines the cache configurations.

At the same time, we probably need a manifest file tracking the status of the cached binaries. For example: the timestamp of last use, the total size of the binary, and probably flag prevents the binary from being deleted for a while (I would like call it a lease. Just think about there are multiple cargo instance is running, once one cargo decide to use a cached binary, it should have a way to pin the binary in the cache a little while. And the lease has a expire timestamp just in case cargo crashes during the build).

Any ideas on this ?

[Eh2406]

Any ideas on this ?

Meny! I have shared some of them in previous discussions including this, this, this, or the aforementioned this. The TLDR is that the CARGO_TARGET_DIR = ~/.cargo/buid_cache/ is most of the way to a good solution to the technical problems, except for the problem of cleaning it out. (There are UI problems like only sharing things that are likely to be reused, but I don’t see them as big blockers.) There are many good ideas for how a GC should work LRU by size, LRU by number, last used cutoff, if the compiler is still installed, if the toolchain is still installed… and many many more. Each one is perfect for someones use case, and unacceptable for someone else’s. It is going to take a lot of design work to find something good enough to become part of Cargo itself. Luckily most of these ideas can be experimented with as out of tree cargo-subcommands like cargo-sweep, or as cargo wrappers like sccache.

Go bild a thing, and please tell us (the Cargo team) what we can do to make things easier. The Cargo team is open to making changes to make exploring this design space possible! We are working on improving the docs of the files that cargo generates, because several of these experiments have reported back that they don’t know what the impact of cleaning each file is likely to be. We are definitely willing to consider PRs that add functionality to cargo to unlock more exploration, for example a timestamp file recording the last time cargo used that artifact.

[kornel]

Small problem with changing actual cargo target dir is that the final build results (binaries, dylibs, etc.) are also placed there. It’s convenient to have ./target/release/exe in project’s own directory, not in some shared cache dir hidden somewhere.

Basically, Cargo target dir conflates concepts of build/temp dir and products dir. These need to be separated out.

BTW, Xcode nicely separates temp, intermediaries and products as separate dirs, and I had to work around Cargo to use it from Xcode.

[haohou]

I see. I think it’s what I current have:

Put all non-primary outputs to the dir specified by CARGO_SHARED_TARGET_DIR, at the same time, if there’s required binary in that directory, just do not recompile the code.

Is this sounds a reasonable change ? If yes I can do some cleanup and publish a PR.

[haohou]

Small problem with changing actual cargo target dir is that the final build results (binaries, dylibs, etc.) are also placed there. It’s convenient to have ./target/release/exe in project’s own directory, not in some shared cache dir hidden somewhere.

What I am doing is only change the output of non-primary units.

Basically, Cargo target dir conflates concepts of build/temp dir and products dir. These need to be separated out.

Yes, maybe this is another thing should the change handles.

[Eh2406]

Small problem with changing actual cargo target dir is that the final build results (binaries, dylibs, etc.) are also placed there. It’s convenient to have ./target/release/exe in project’s own directory, not in some shared cache dir hidden somewhere.

Yes! This is the kind of UX problems the current implementation has. We would definitely need to make that better before we can change cargos default behavior.

BTW, does the unstable --output-path help with this in the meantime?

Is this sounds a reasonable change ? If yes I can do some cleanup and publish a PR.

I do not have the authority to unilaterally merge such a change, but I would be happy to look at and comment on such a PR! I think it may spark a useful conversation on how to make incremental progress.

[dcarosone]

A fair way down this thread, and no-one has mentioned security?

Rust’s safety and security depends on the compiler. Compiler output is assumed to have gone through all the borrow-checker (etc) validation and be trustworthy. We can read and validate the source code input to this process (at least in theory).

How would this scheme address the concern of untrustworthy code being uploaded to the cache?

[edit] I seem to have inferred too much from the word ‘global’ here; it seems the proposal is more per-user between projects. That’s a good idea and something I’ve wished for, too. My confusion happened because there was another similar conversation about downloading build products from crates.io recently, and I conflated the two somewhere.

I’m leaving this concern here, because it’s important to keep in mind if ever going beyond a single trust boundary.

[CAD97]

If it’s local to your machine (or even your physical network), you can (hopefully) trust anyone who would be putting a build in the cache, as they either have access to your machine or your physical network. Both are basically game over security-wise even without a binary cache for cargo.

If it’s on the internet, it’s a similar process to trusting the source code you’re downloading. Check a hash to make sure you have the correct thing, and check to see if people you trust trust that binary (or source code) using something like cargo crev.

If we assume cargo has reproducible builds, you could have a build bot that builds the crate locally and checks against the prebuilt, and if they’re the same, trusts it to be the binary resulting from building that source code.

It’s all a trust problem. For this though, I hope you can trust yourself

[haohou]

I would be happy to look at and comment on such a PR!

I just published the PR https://github.com/rust-lang/cargo/pull/6437

[josh]

I would love to see cargo sharing binary artifacts between builds by the same user on the same system by default.

[repi]

Agreed, it would be a huge step forward and improve the default and overall experience for everyone!

Both for developers that may build lots of small Rust apps and where we now rebuild common dependencies and waste diskspace for the duplication, and also for developers that work on just a few but much bigger projects where even 2-3x redundant compilation across the projects is a big hit.

[Eh2406]

A quick update on the progress. I have two PRs to improve cargo sweep. One that uses the data stored by Cargo to clean files that are for a no longer installed version of rust. The other uses the hash files checked by Cargo to accurately determine that atime of the last Cargo invocation using each file. (still relies on the file system maintaining atime.) I also have a PR to Cargo to have Cargo maintain a last used timestamp file in the folder that (my PR to) cargo-sweep is already checking. (so we can get away from atime.)

@haohou PR to Cargo requires some work on a technical problem that is related to a design problem. The technical problem is how can we be sure that artifacts will only be reused if exactly equivalent. The design problem is what exactly should go in the Cached folder. The PR as it stands choose to put only the, non primary, file dep artifacts in the Cached folder, and was working on heuristics for making sure reuse was correct. Unfortunately, Alex was unconvinced that we would find and maintain a sufficient set of heuristics without somehow reusing the existing fingerprint module.

[Eh2406]

So let me try to articulate an alternative answer to that design problem. Edit: to be clear I am not speaking for the Cargo Team.

Proposal

There are two folders Cargo uses for keeping track of and building your project and its dependencies.

1. There is one set by the environment variable CARGO_TARGET_DIR and the target-dir field in .cargo/config and defaults to /target. The exact content and structure of this folder is an implementation detail of Cargo, and can be changed in the future, but this proposal does not suggest changing it. This folder is used for artifacts that Cargo thinks will only be useful to this project. The exact rules for what goes in this folder are implementation details of Cargo, and can be changed in the future, but it will include the bin and lib and test artifacts for the project being compiled.
2. The other one is set by the environment variable CARGO_CACHE_DIR and the cache-dir field in .cargo/config and defaults to being the same as target-dir. The exact content and structure of this folder is an implementation detail of Cargo, and can be changed in the future, but this proposal suggests starting it the same as the target-dir. Cargo will always support using the same folder for both even if the default changes in the future. This folder is used for artifacts that Cargo thinks may be useful to other projects.

Why keep the same structure between the two folders?

Cargo does a good job of rebuilding/reusing only when needed. That “fingerprint” system is coupled to the folder structure. By keeping the structure we can keep the working system. Also tools built for cleaning one will work on the other.

Why does cache-dir default to target-dir?

This means that accepting this proposal will not break any existing users. Then when this is working well it is not hard to propose changing this default to /target/cacheable/ for eazy CI cashing, or to ~/.cargo/build_cache/ for good multi project cashing. The bikeshed of what is a good default and whether it is worth braking users, is left for a follow up. Edit: This also means that if we are not caching in a way best optimized for this new use, then we are not inflicting it on the community. Unless a user opts into it, or until that follow up. User opt in gives us time to debug this before we seriously suggest that follow up. See the conversation below.

What exactly goes in the `cache-dir?

I don’t know. That is why I want it to be an implementation detail we can change. I think making it “things from crates.io” as @matklad demonstrated, or “not primary” as @haohou was working on, would be a good place to start.