# A way to run tests for all your dependencies, recursively?
**URL:** https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584
**Category:** cargo
**Created:** [April 2, 2024, 7:46pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584 "2024-04-02T19:46:32Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@Vorpal](https://internals.rust-lang.org/u/Vorpal)
#### Post date: [April 2, 2024, 7:46pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/1 "2024-04-02T19:46:32Z")
## Problem
Currently `cargo test` only run tests for the current crate or workbench. Even the `-p` flag only seem to be able to select between packages in the current workspace.
There would be value in being able to run cargo tests for dependencies as well. For example, you may be targeting an architecture or operating system that the author of dependency did not test for. Or the dependency might no longer pass tests on the newest version of Rust (or on the older one you are using, MSRV isn't always correctly declared).
This would also be useful when packaging for distributions. E.g. Arch Linux does _not_ create separate packages for each Rust library like Debian does, and as a result it will only run the tests for the final binary crate. See [the Arch Linux Rust packaging guideline](https://wiki.archlinux.org/title/Rust_package_guidelines) for more background info.
This should not be the default (can you imagine the memes about compile times if it was), but perhaps an option such as `cargo test --dependencies -p '*'` could work (e.g. `--dependencies` would enable also testing dependencies outside the current workspace).
## How would this work?
It would use the test files that are included in the `.crate` files. Not all packages may currently include the tests that way. That is OK, but if this feature existed it would put pressure on the maintainers to include those tests in the crates, which would be a good thing.
The tests could build and run either as part of your target directory, or in a temporary directory, I have no strong opinions on those sort of details.
## Discussion
Is this something that has been discussed before? I wasn't able to find anything. Would there be any major technical problems with implementing this? What would the pros and cons be?
---
### Author:  [@josh](https://internals.rust-lang.org/u/josh)
#### Post date: [April 2, 2024, 9:35pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/2 "2024-04-02T21:35:42Z")
I'd love to see a way to do this, and running tests on the specific target seems like a great motivation.
That said, I do know that some crates don't include tests in the .crate files because the tests and test data are _large_ and they don't want every user to have to download those when most users don't need them.
One potential way to solve that would be to check out the git repository of the crate and run the tests from there.
(Relatedly, we should really have a way to verify that the uploaded .crate file matches the git repository.)
---
### Author:  [@kornel](https://internals.rust-lang.org/u/kornel)
#### Post date: [April 2, 2024, 10:30pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/3 "2024-04-02T22:30:39Z")
[Here's a thread where this came out](https://users.rust-lang.org/t/psa-check-if-your-cargo-crates-are-clean-and-tagged/109264/1).
To me there are a few problems with the current state of things:
- `cargo publish` doesn't run tests, but tests that depend on other files or their workspace layout can be broken during packaging. This feels like a workflow that is not supported in Cargo. As a crate author I'm unhappy about having to maintain a test setup that is untested by me. This might be solvable by running the tests during `publish`, and maybe `cargo test --packaged` or an extra `cargo hack` mode, but that's an additional CI time spent, and a slower publishing workflow.
- I prefer not to publish data files for tests, especially when they can be pretty large. I work with image codecs so I have files that test largest possible images, and sometimes submodules with official test suites that can be huge. Large crates are annoying, especially in environments like hosted CI and Docker where it's hard to cache any of Cargo's directories.
- While it's easy to exclude external files from packages, it's not possible to remove inline `#[test]` functions from `src/` files that may need these files, which leaves the packaged tests broken. Due to private APIs not all tests can be in `tests/` dir. Commenting out tests for `cargo publish` is problematic, because a "dirty" repository doesn't get `.cargo_vcs_info.json` file, which is useful for associating release with a repository, and sometimes needed to process relative links in README properly.
- CVE-2024-3094 has shown that test files can be used to hide malware payload, and such files can be easily obfuscated to be impossible to review. I have to review tarballs of crates that I'm using for my work. Source code review is already difficult and tedious, and I'd rather just not have to worry about additional risk of binary blobs in tarballs.
- The tests for crater runs and Linux distro packaging need to be downloaded only a handful of times, while downloads for regular `cargo` builds can happen literally million times more frequently. crates-io downloads are growing exponentially. Now in a single day it serves more crates than it did in the whole two years after Rust 1.0. The extra data is likely to get expensive. It feels like a huge waste to make crate tarballs larger for a once-in-a-million use.
---
### Author:  [@kornel](https://internals.rust-lang.org/u/kornel)
#### Post date: [April 2, 2024, 10:41pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/4 "2024-04-02T22:41:34Z")
Some potential solutions:
- Maybe [crates.io](http://crates.io) could have two tarballs per crate — one with minimal files for building a dependency, and an extras tarball with remaining files for tests and benchmarks.
- or support "storing" files in [crates.io](http://crates.io) tarballs similar to git LFS. For files excluded from the regular package, the tarball would store hashes of files, and have some mechanism to obtain them either from [crates.io](http://crates.io) or a git repo.
- or change the publishing workflow so that tarballs are not uploaded by users directly (which makes it possible to manipulate the tarball and put arbitrary files in there), and make [crates.io](http://crates.io) fetch itself a specified git tag (or mercurial, etc.). This would be a stronger guarantee that the [crates.io](http://crates.io) package is the same as a given tag, and the excluded files could be fetched from the repo.
---
### Author:  [@mathstuf](https://internals.rust-lang.org/u/mathstuf)
#### Post date: [April 4, 2024, 4:56am UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/5 "2024-04-04T04:56:17Z")
> [@josh](#):
>
> That said, I do know that some crates don't include tests in the .crate files because the tests and test data are _large_ and they don't want every user to have to download those when most users don't need them.
Some of my (git-using) crates store test data in the history of the project itself (merged into the history using `-s ours`) because `git bundle` files are a pain to juggle and amend. Not to mention being binary blogs to commit. These crates just don't test properly at all outside of their git repository. To that end…
> [@josh](#):
>
> That said, I do know that some crates don't include tests in the .crate files because the tests and test data are _large_ and they don't want every user to have to download those when most users don't need them.
How can one detect that it is a `.crate` build and not a `git`-based build without `build.rs`?
---
### Author:  [@mathstuf](https://internals.rust-lang.org/u/mathstuf)
#### Post date: [April 4, 2024, 5:00am UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/6 "2024-04-04T05:00:56Z")
> [@kornel](#):
>
> This might be solvable by running the tests during `publish`
Unlikely. I'm not really interested in `publish` being gated on a single configuration. My project's publish job is CI-dependent on the builds I _do_ care about passing (various feature flag selections, MSRV, `clippy`, etc.). If I cared about other platforms for these projects more, I'd gate publishing on those passing too, not just what happens to be doing the publishing.
---
### Author:  [@josh](https://internals.rust-lang.org/u/josh)
#### Post date: [April 4, 2024, 8:24am UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/7 "2024-04-04T08:24:51Z")
> [@mathstuf](#):
>
> > [@josh](#):
> >
> > That said, I do know that some crates don't include tests in the .crate files because the tests and test data are _large_ and they don't want every user to have to download those when most users don't need them.
>
> How can one detect that it is a `.crate` build and not a `git`-based build without `build.rs`?
I'm not sure what you mean by this? You don't have to detect those cases; the .crate file just doesn't support running `cargo test` from it at all, or alternatively doesn't support some of the tests.
---
### Author:  [@mathstuf](https://internals.rust-lang.org/u/mathstuf)
#### Post date: [April 4, 2024, 12:35pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/8 "2024-04-04T12:35:01Z")
Oh, you're saying that the `test` action is just broken there? I though that those explicitly excluding the files had also manage to disable the code using a similar mechanism.
---
### Author:  [@josh](https://internals.rust-lang.org/u/josh)
#### Post date: [April 4, 2024, 12:57pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/9 "2024-04-04T12:57:47Z")
Depends on the crate, but sometimes it's just broken. Other times it works and self-disables when it sees the files it needs are not present.
---
### Author:  [@Vorpal](https://internals.rust-lang.org/u/Vorpal)
#### Post date: [April 5, 2024, 1:30pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/10 "2024-04-05T13:30:27Z")
> [@josh](#):
>
> I'm not sure what you mean by this? You don't have to detect those cases; the .crate file just doesn't support running `cargo test` from it at all, or alternatively doesn't support some of the tests.
Not quite true. Both Fedora and Arch packages run tests on rust code built from downloads from [crates.io](http://crates.io).
---
### Author:  [@afetisov](https://internals.rust-lang.org/u/afetisov)
#### Post date: [April 5, 2024, 4:02pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/11 "2024-04-05T16:02:25Z")
> [@kornel](#):
>
> I prefer not to publish data files for tests, especially when they can be pretty large. I work with image codecs so I have files that test largest possible images, and sometimes submodules with official test suites that can be huge. Large crates are annoying, especially in environments like hosted CI and Docker where it's hard to cache any of Cargo's directories.
I certainly wouldn't want to download hundreds of megabytes of test vectors for all my crates either. In fact, I recall posts on Reddit and URLO where people with flaky metered connections complained about basically everything non-essential in a crate, including example code and docs, and they have a certain point.
That said, advanced test frameworks typically have multiple available test configurations. There are the usual tests, which are assumed relatively light and fast. But there can be also multiple separate test profiles for e.g. tests which require heavy external data, or a specific environment, or just take too long to execute. These heavyweight tests wouldn't be run by default, but can be opted in. It would be nice if Cargo supported something like that out of the box. At the moment this kind of test separation needs to be hacked in with something like special features, or build scripts.
If we had a built-in solution for this kind of test separation, we could also have extra metadata attached to heavyweight tests. I'm not sure whether [crates.io](http://crates.io) would be a good place to store test data, but one could certainly download it manually from relevant sources.
I believe that most crates would be fine using only light tests, which don't require any extra support from Cargo, [crates.io](http://crates.io) or the user. Even for crates which need heavyweight tests, it's likely that light tests could provide a nice first approximation to validation even in the absence of test data.
---
### Author:  [@system](https://internals.rust-lang.org/u/system)
#### Post date: [July 4, 2024, 4:02pm UTC](https://internals.rust-lang.org/t/a-way-to-run-tests-for-all-your-dependencies-recursively/20584/12 "2024-07-04T16:02:30Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.