Breaking down crate dependencies?

tools and infrastructure cargo
1

At the moment, dependencies in Cargo.toml are coarse-grained.

In theory a crate can contain:

  • One library.
  • Multiple binaries.
  • Multiple example binaries.
  • One unit-test binaries.
  • Multiple doc-test binaries.
  • Multiple integration test binaries.
  • Multiple benchmark binaries.
  • One build-script binary.

And it's quite useful to have a single crate contain all those, as it means that one's fingertips, one gets to run cargo check, cargo clippy --all-targets, cargo test, cargo bench, etc...

Unfortunately, dependency-wise, it's a complete nightmare.

There's only 3 categories of dependencies in cargo:

  • [dependencies] means that if one binary includes heavy-weight dependencies, like, say tokio, then any crate wishing to use the library will also depend on (and bundle) tokio now. Tough luck.
  • [dev-dependencies] means that if there's a single benchmark depending on criterion, then all unit-test, doc-test, and integration test binaries will also depend (and bundle) criterion.
  • [build-dependencies] is fairly nicely insulated at least, though with multiple build scripts being worked on... a single build script depending on a heavyweight dependency will drag down all other build scripts.

The current work-around is to break down the crates. Binaries must be their own crates. Examples and integration tests are better off in their own crates. But then convenience is lost. Doubly so as running cargo in a folder which is not a crate does NOT limit cargo to running only on the crates contained in this folder, but instead lead to cargo attempting to run on the entire workspace.

Is there any work towards making the dependencies more fine-grained?

Ideally, one would be capable of specifying the dependencies for each compilation target/artifact independently, something like:

[dependencies.lib]
[dependencies.bin.my-binary-1]
[dev-dependencies.test]
[dev-dependencies.doctest]
[dev-dependencies.examples.my-example-1]
[dev-dependencies.tests.my-integration-test-1]
[build-dependencies.my-script-1]

Which would be backward compatible with using a plain unqualified [dependencies] or [dev-dependencies] or [build-dependencies] as today.

Note: in the context of multiple build scripts, it would notably allow starting to run fast-to-compile build scripts while the heavyweight dependencies of other build scripts are still being compiled.

It's not that the state of things today is unworkable. It's just... sad. One has to choose between convenience/ergonomics and compile-time performance/binary size1.

1 Especially so as apparently, per Kobolz discovery, debug information are not tree-shaked, so that including a heavyweight dependency in the binary results in said binary containing the DI of this dependency even if not its code.

6 Likes
2

Per-build-target features was discussed in This Development-cycle in Cargo: 1.77 | Inside Rust Blog which can be used similarly to per-build-target dependencies and talks about the trade offs of overloading treating the primary unit of work as a build target in a package vs a package in a workspace.

Which would be backward compatible with using a plain unqualified [dependencies] or [dev-dependencies] or [build-dependencies] as today.

That data model would not work because dependencies.<name> is your dependencies.

In TOML, the following are the same

[dependencies.regex]

[dependencies}
regex = {}
1 Like
3

This creates even more overlap between crates with multiple artifacts vs workspaces with multiple crates.

It feels like both of these concepts are converging (more subdivision inside crates, more integration across crates in a workspace). Eventually crates and workspaces will become the same thing.

1 Like
4

I can see the overlap, and yet at the same time...

Logical Unit

There's a strong argument for bundling library code with its benchmark, examples & tests.

Logically speaking, this is a single logical unit, and the benchmarks, examples, & tests are "satellites" of the library.

This single logical unit happens to be called a crate.

Work Unit

Similarly, after making changes on a library, it's quite natural to first ensure that the changes on this library are sound (in isolation) by running the tests, and potentially the benchmarks. By adjusting the examples and ensuring they still compile & work.

Only then, after the library has been proven to work according to the updated expectations, is it time to integrate the changes into the downstream dependencies within the workspace and beyond.

It's therefore natural to run a workflow equivalent to clear && cargo check && cargo clippy --all-targets && cargo test, and optionally cargo run --example and cargo bench as appropriate.

Due to past unfortunate decisions, cargo can only work at crate or workspace level, and not any intermediary (sub-folder) level, so crate it is by necessity.

Practice

The main codebase I work on is a single workspace of 250 crates, and growing. Given that I have personally written most of the code in the past 3 years, I expect larger companies have much larger workspaces.

This is 250 crates using crates as a logical unit: without artificially splitting out test crates, bench crates, binary crates, or example crates to reduce false dependencies. We are fortunate that the compilation times are not atrocious -- most single binary can be compiled within 15s from scratch on my own machine -- and therefore we have not sought to break the ergonomics to gain more performance.

But I have been tempted to, countless times, and I have sighed, countless times.

Is a crate that similar to a workspace?

I don't feel so, in my work.

They do share many things -- dependencies, perhaps one day features, etc... -- but they have very distinct roles regardless.

I would also reiterate that at the scale I mentioned, the fact that cargo can only compile either 1 crate or 250 crates (and their 500 binaries when testing) is actually just impractical.

The ability to run cargo on subsets of the workspace is really missing. I cannot recall having this issue with Bazel, for example.

Every time I run cargo in a subfolder and it starts re-compiling the entire workspace, I curse, reminded of how unintuitive and unergonomic this decision is at scale. Hindsight...

If it was possible to run sub-folders, then I could in good conscience split up the benchmarks, the binaries, the tests, the examples, etc... in separate crates in sibling folders. I could.

It would still be slightly unergonomic, death by boilerplate:

  1. Paperwork: I would need to create yet another Cargo.toml, and a src subfolder.
  2. Names: I would need to name those. Sigh.
  3. Unit tests: separating them requires, unfortunately, exposing APIs which shouldn't be. And it's not possible to conditionally define the visibility by cfg, I'd need a macro, or a forwarding function. Not great.
  4. Doc tests: they cannot be separated from the items they document, so they would remain behind...

Then again, unit-tests and doc-tests are likely to have relatively few dependencies, and if they're the only ones sharing [dev-dependencies] maybe it's not too bad.

[From the link] To that end, a thought experiment was proposed: what if we only supported one built output per package? Where would be the pain points?

I guess it really depends what is consider a build output:

  • First of all: benchmarks, examples, doc tests, integration tests, and unit-tests are satellites, and not really "packages" of their own.
  • With regard to libraries vs binaries, I could see it, though it also has a cost.

One great pattern -- especially in Rust -- is a Sans IO with a "pure" functional core wrapped into an "impure" (IO) shell. Today, the easy way to do so is to:

  1. Define a library for the functional core.
  2. Define a binary which provides the shell.

Splitting this explicitly in a lib & bin allows ensuring that I/O doesn't accidentally sneak in in the library, as well as allows "black box" benchmarking & integration testing the entire core.

It's possible to break that down into two crates, certainly. I in fact just did it yesterday. Then I lost 15 minutes failing to understand why my binary wasn't behaving as I expected, only to realize I had executed cargo build on the lib crate, not the bin crate, and thus the binary of course didn't reflect the changes I had made.

It's one of those ergonomic costs that come from artificially splitting a logical unit in two just to escape the dread of spurious dependencies.

Sure, I was just spitballing to illustrate the point. Some syntax can surely be worked out.

5

In that post, I was referring to public outputs (binaries and libs) and not internal ones (build scripts, tests, benches, or examples)..

In your reply, you talked frequently of "benchmarks, examples, & tests'. For myself, I see less justification to re-design Cargo.toml enough to allow splitting out dependencies for these.

If a library exists just to isolate side effects from the binary, then I'm also not seeing the benefit to splitting up the dependencies. If the library is also being used in other packages, then it can make sense. However, at that point, your requirements have shifted. If you are using a registry, then the versioning requirements between the binary and library become different.

The linked post links out to RFC #3374 which caused problems with verifying the appropriate features are activated in light of feature unification (https://github.com/rust-lang/rfcs/pull/3374#discussion_r1235768792). I had considered calling out this problem specifically but per-target dependencies doesn't touch on package features. However, it does touch on dependency features. This is less likely to be an issue without the associated package feature enabled but there are still risks there.

Another consideration unique to this request is there is a concern over the ecosystem cost to adapting to new dependency tables. This doesn't mean we can't have new ones but that the benefits need to justify the costs.

ps I just remembered Allow dependencies that only apply to specific cargo targets (bin, example, etc.) · Issue #1982 · rust-lang/cargo · GitHub exists

6

There's more than one usecase :slight_smile:

With regard to a library/binary split for Sans IO, separating dependencies has two benefits:

  • Functional: prevents accidentally using IO libraries in the Sans IO core.
  • Compile-time: avoids dragging in pointless dependencies (tokio, the banana/gorilla/jungle) in integration tests which are mocking the IO, and thus not using tokio at all.

Another usecase I find myself having is with generated protocol libraries. The library itself just provides model/encoders/decoders for the protocol, and that's all that clients of the libraries really need, however it's also useful for developers to provide "minimal" binaries which can be used to decode a blob from the command line, or to connect to a stream of messages and process them on the fly -- filtering, printing, saving to file, etc...

At the moment, I (ab)use examples for these binaries. Not because they're good examples of library usage, but simply because it means I get separate dependencies that do not get dragged into the downstream dependencies. In exchange, those dependencies bloat the tests, because unfortunately examples & tests share the [dev-dependencies] section.

This is another example of "satellite" binary, which is not strictly intended to be a "build output" in the grand sense of the term. It's never intended to be used in production, it's just a little debugging utility.

I think it's very important to ensure backward compatibility here.

That is, I think the existing sections should be extended but not replaced. In fact, the ability to specify common dependencies only once is quite useful even after splitting. For example, when doing a binary/library split for Sans IO, chances are that the binary will need to use quite a few of the crates the library depends on (directly).

Thus I would see specialized sections as adding dependencies to the existing sections for certain build targets. Just like [dev-dependencies] adds to [dependencies] for "dev" build targets today:

[dependencies]

# All build targets depend on a, b, and c.

a = "0.1"
b = "0.2"
c = "0.3"

[bin-dependencies]

# Binaries in this crate depend on d in addition to a, b, and c.

d = "0.4"

[dev-dependencies]

# Dev binaries in this crate depend on x in addition to a, b, and c.
# They do not depend on x by default.

x = "0.5"

[examples-dev-dependencies]

# Example binaries in this crate depend on y in addition to a, b, c, and x.

y = "0.6"

[benches-dev-dependencies]

# Benchmark binaries in this crate depend on criterion in addition to a, b, c, and x.

criterion = "0.7"

[unit-tests-dev-dependencies]
[doc-tests-dev-dependencies]
[integration-tests-dev-dependencies]

In the absence of [dev-dependencies-unit-tests], unit tests are compiled with just [dev-dependencies] and [dependencies]; just like today.

Ah! I hesitated in calling this post "Pre-RFC" as I knew it must have come up before, and it was probably just my Google-fu being weak.

7

Is this a UI issue? cargo technically can compile an arbitrary subset if you specify multiple -p flags. There's also default-members workspace setting.

Dependencies are cached in workspaces. In such large workspace, I presume you would have a lot of common dependencies shared between crates, and their tests and examples. Would you actually save much build time if you specified deps very granularly? Your test helpers and bench frameworks could be compiled once and reused everywhere (unless they get different feature flag, which is made worse by more fragmented dependency declarations).

1 Like
8

All 3rd-party dependencies are specified at the workspace level, with base features, and "inherited" with dep = { workspace = true }. A few dependencies such as tokio get extra features in specific crates (net, in particular), but that is the exception, rather than the norm.

So, yes, in theory most dependencies should be compiled once (or perhaps twice, for tokio). In practice, compiling multiple packages one after another tends to lead to some dependencies getting recompiling. I have not investigated any further, but suspect that some of the 3rd-party dependencies may enable feature that I do not, and depending on whether the package includes them or not...

This helps with compile times, but bloat still means extra link time (for nothing), especially with DI, and extra space consumed in the target folder, which is already regularly pushing the 100GB.

Very much.

The default-members is of no help. The workspace is really only used to harmonize the 3rd-party dependency versions being used, and the members being compiled very much depend on the task at hand, and vary completely from one time to the next.

The -p flag is in theory possible, but specifying the package names one by one is very inconvenient. Sometimes, I'll use a little shell help: for d in *; do $(cd $d && cargo build --release); done.

Double-checking the docs, they do mention that -p supports wildcards, though there's no example with them, so it's not clear where they're supported. I'll make a note to experiment with that, it may help.

9

Specifying dependencies at workspace level helps, but it isn't sufficient, because features of indirect dependencies can be enabled or disabled depending on the set of packages being built together (e.g. you only use tokio without net, but build with or without another crate that uses reqwest which enables tokio/net).

for d in *; do $(cd $d && cargo build --release); done

This is the worst-case scenario for extra rebuilds, because feature flags of transitive dependencies will be adjusted for each build (OTOH you get potentially leaner executables due to each using minimum features).

You don't need wildcards in -p, you can specify it any number of times:

cargo build -p crate1 -p crate2 -p crate3 -p crate4

and this will unify feature flags across the crates, so their dependencies will be built once.

Cargo supports aliases, so if you're frequently building or testing a certain subset, you could have an alias with all the -p flags needed.

If you have heavy deps that are only needed for examples or rarely used binaries, you could hide these behind feature flag, and use required-features when defining each [[example]] and [[bin]], although IMHO it's disappointing that it can only break builds, instead of configuring them to work automatically.

1 Like
10

You could also use aliases to get the “build this subdirectory” behavior (though with a lot of manual management): within each subdirectory define a buildset = ["build", "-pfoo-bar", "-pfoo-baz"] alias for that directory’s packages, then wherever you run cargo buildset from will use the nearest alias.

11

A way to deal with this is the cargo workspace hack technique: cargo_hakari::about - Rust explains the idea, and using cargo-hakari is the most reasonable way I have found for doing this.

12

I know.

The problem is that it's very unergonomic.

I usually build subsets, but which subsets tend to vary depending on the task at hand...

TIL feature unification will unify features across binaries, thereby bloating binaries by default.

It's probably nice for Debug builds, but a bit unexpected for Release builds. I'll have to double-check how performance sensitive binaries are build.

13

Most extraneous features should not have any effect on the binary because they only add items to the library, and as long as those items go unused, they have no effect on the contents of the binary.

The typical exception to this principle would be when enabling the feature requires the library having the feature to make changes to existing items to support the feature — this should be kept uncommon.

14

Actually, even addition is in itself a potential optimization barrier.

Specifically, the issue mostly manifests when transitioning from 1 to 2 possibility:

  • A single implementation of a trait means that the dynamically dispatched calls can be optimized to the only implementation, this isn't the case any longer when there are 2 implementations.
  • A single way to define a return value means that the value can be optimized to its only known value, this isn't the case any longer when there are 2 ways.

Pretty niche, admittedly, but still :slight_smile:

15

That optimization is only possible to do with LTO and with LTO each binary gets optimized separately and the optimization would almost certainly run after throwing away all unused vtables.

This probably also needs LTO for the compiler to know.

16

TIL that wildcards are supported. Here is an example I've just tried: cargo check -p '*-enclave'.

(slightly off-topic): Now I just wish there'd be a cli flag to disable/choose feature unification between packages [1] (which would also make this behavior more explicit, while reusing the -p syntax): [2]

// Same as individual cargo build calls, same  as current behavior with
// only one -p and no wildcards, could also be called '--no-feature-unification'.
cargo build --release -p '*-enclave' --feature-unification=''
// Same feature unification as --workspace but don't build everything
// Unless I'm mistaken that is what hakari is effectively doing.
cargo build --release -p '*-enclave' --feature-unification='*'
// Consider all packages ending with '-enclave' for feature unification,
// but only compile those ending with '-whatever-enclave',
// as if only packages ending with '-enclave' existed in the workspace.
cargo build --release -p '*-whatever-enclave' --feature-unification='*-enclave'

// Not quite sure what should happen in this case. It probably should
// behave like '--no-feature-unification' except for the (in-workspace)
// dependencies ending with '-types':
// If 'a-enclave' needs feature 'some-types:X' it will not be enabled for
// 'b-enclave' (because we didn't tell it to include 'a-enclave' for feature
// unification. But if 'other-types' needs feature 'tokio:net', both 'a-enclave'
// and 'b-enclave' will have 'tokio:net'.
cargo build --release -p '*-enclave' --feature-unification='*-types'

// These are/would be the same (I think)
cargo build --release --worksapce
cargo build --release -p '*'
cargo build --release --worksapce --feature-unification='*'

// For backwards compatibility these would need to be equivalent
// (should work for all wildcards and '--workspace').
cargo build --release -p '*-enclave'
cargo build --release -p '*-enclave' --feature-unification='*-enclave'

This is effectively telling cargo (and whoever sees/uses the command) which crates he has to be aware of if he wants reproducible builds. Another way would be to specify which individual dependencies should have all their features be unified like with '--workspace', but I think this is more intuitive and requires fewer entries (thanks to the wildcards). Especially due to its similarity with -p.

This way you can also decide when exactly you want it (perhaps with a way to specify a default behavior in the future). Splitting the "what to build" from the "which features should be unified" when needed.

As long as you don't change the feature-unification list each dependency is only compiled once (unless you have multiple major versions of course).

Another situation where feature unification is unwanted is when the crates are intended for different compilation targets (e.g. embedded) or if there is a security boundary between them and you want reproducible builds that are not affected by other workspace members.

  1. Argument name is bikesheddable of course. ↩︎

  2. Shoud also work with wildcards like something-*, something-*-whatever and *-foo-*. ↩︎