Explicit "test" dependency grade?

I've been (linux) vendorizing a lot of crates, and a common problem I face is that "dev-dependencies" is too general.

As a vendor, its desirable to execute tests in some way, and be able to communicate the dependencies required for tests to the build tooling ( which operates outside cargo, and intrinsically limits cargo's ability to do anything that requires network IO on its own )

But currently, the design of Cargo.toml implies that only upstream maintainers should ever run tests.

And because "dev-dependencies" communicates dependencies for both examples and tests, additional effort has to be performed to identify the "dev-dependencies" that are actually needed.

To make things worse, often, upstream maintainers declare dev-dependencies in their Cargo.toml, and then publish them, but their build tooling actually strips the reason those dev-dependencies were there in the first place (for instance, they may have examples in their source repository, but they're not part of the published crate)

By association, this means that dev-dependencies end up being required to run tests, even though they're not used for running tests.

And this also discourages wide-spread testing of crates, which is hardly a Good Thing.

( For instance, I myself would consider writing a cargo tool that, when run, recursively visits all your dependencies and in turn, makes sure they pass tests, in order to weed out any potential platform-specific bugs, but the way cargo currently is, that would pull in excessive dependencies that had no use )

To summarise, I would greatly appreciate a [test-dependencies] (or something along those lines), that, when present, was used instead-of the declared [dev-dependencies] when it came to running tests.

( I would similarly appreciate some kind of explicit declaration that a dependency is required only for examples, because some vendors also may wish to ship built examples, and therein, relegate the "dev-dependencies" to communicate "dependencies you only need if you're a maintainer and/or working on a git checkout", but that's a different bike-shed )

2 Likes

To clarify, this can involve the need to patch the Cargo.toml of the published crate just to get tests working, where all the patch does is strip the spurious deps.

Similar issues that I ran into recently: I had a build dependency that was only needed for tests. In the end, I built a feature for "tests" that it could be conditional on. In a sense, making this a well-defined feature might be a more flexible/orthogonal/extensible design.

This also goes back to the notion of global or well-known features as discussed in Pre-Pre-RFC: making `std`-dependent Cargo features a first-class concept.

Having it as a feature would also be nice because you could compile library targets with some additional APIs that enable integration tests to do more fine-grained things.

3 Likes

For an example of how this currently massively sucks: