Feature request: Specifying test resources in Cargo.toml

1

I'd like to get some preliminary feedback on a potential feature request to cargo (as suggested by the issue template).

My idea: adding a field (say test_resources) in Cargo.toml similar to includes. Cargo would then re-compile the tests if any of those files changed. Do you think it would be a good idea?

Background

I have a project where integration tests depend on many resource files (plain text files that specify the inputs and expected output of my system). Using the rstest crate, I automatically generate one test case for each sub-directory of my test resource directory.

At the moment, changes to my test resource directory are not be detected by cargo test. I would like that when I make any changes to this directory, cargo test will recompile my tests, picking up new directories added there (for instance).

The problem has been identified on the rstest side: New test files are only recognized after `cargo clean` · Issue #256 · la10736/rstest · GitHub.

Alternatives

I have considered the following options instead:

  • Use a build.rs script with cargo::rerun-if-changed=PATH. It's probably the best solution so far, but introducing a build.rs just for that feels a bit overkill, and it would also influence cargo build (which it shouldn't, as my binary doesn't depend on those files)
  • adding my test resources to includes. This will publish my test cases when I push my crate to crates.io, which seems wasteful.
  • Migrating away from rstest and listing the test directories at test runtime. The downside of that is that this generates just one Rust test case, which will contain all actual test cases defined by the sub-directories. This means that I would also need to re-implement the ability to run tests in parallel myself, for instance.
1 Like
2

There is actually an unstable feature for proc-macros to inform the compiler about their file-system dependencies: proc_macro::tracked_path::path.

1 Like
3

I see this as unlikely and instead am working towards runtime test case enumeration in a custom test harness. rustest and test-r are similar to what I want to achieve. I've not looked deep to see if they could replace rstest for your need.

1 Like
4

Thanks both, that's very helpful!

It looks like the tracked_path::path feature would be a neat fit, if it were able to track changes to directories (added or removed directory entries, not just changed file contents), which seems unclear so far looking at the tracking issue.

The option of dynamically generating test cases using a custom harness, as test-r does, seems to be a good one, I'll try to migrate to that for now. Curious to try out yours when it's available, @epage. :slight_smile:

5

We've talked in the past about having a Cargo.toml mechanism to provide any of the same directives that build.rs can emit. That could work for this.

Unless they're huge, please consider doing this anyway. It's nice when tests work based on the sources from .crate.

1 Like
6

We've talked in the past about having a Cargo.toml mechanism to provide any of the same directives that build.rs can emit. That could work for this.

That would be fantastic - I have also been looking for something like that. Do you have any pointers to such discussions? I could imagine trying to help there.

Unless they're huge, please consider doing this anyway. It's nice when tests work based on the sources from .crate.

That makes sense. I realize now that it would also be useful for downstream packaging (such as on Debian). But I just checked and that's not actually enough to have Cargo recompile the tests when they change (which actually makes sense now that I think of it).