I am developing a service based product and the integration tests are fairly expensive to run up. I would like to be able to use --exclude-integration and --integration flags to filter out integration level tests and run only integration level tests.
The underlying data structures are in place already, so it was just a case of adding in the right hooks, my changes are here:
I think this is a useful feature - does anyone else and what would be the next steps?
Thanks!
What controls TestType? Can you actually have integration and unit tests inside the same binary?
My understanding was this was something you would always control at the Cargo level, cargo test --lib to run only unit tests, cargo test --tests ---doc to run integration tests (including doc-tests).
desc.test_type is controlled by the directory name the files are in, if they are in ./tests then they get classified as integration tests.
That is done here: rust/test.rs at master · rust-lang/rust · GitHub
cargo test --lib does not detect unit tests within a binary, cargo test --tests runs unit and integration tests. (console example below)
The current 'solution' is to either use test naming filtering, setting the integration tests as features and using the --all-features option to enable, or to abuse the #[ignore] attribute.
martyn@workstation1:~/projects/rust-test$ tree ./
./
├── Cargo.lock
├── Cargo.toml
├── src
│ └── main.rs
└── tests
└── integration.rs
...
martyn@workstation1:~/projects/rust-test$ cargo test --tests
Compiling rust-test v0.1.0 (/home/martyn/projects/rust-test)
Finished test [unoptimized + debuginfo] target(s) in 0.12s
Running unittests src/main.rs (target/debug/deps/rust_test-3693454312d4c1d1)
running 1 test
test tests::do_unit_test ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running tests/integration.rs (target/debug/deps/integration-e5731f60a2373a41)
running 2 tests
test an_ignored_integration_test ... ignored
test a_integration_test ... ok
test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s
martyn@workstation1:~/projects/rust-test$ cargo test --lib
error: no library targets found in package `rust-test`
Literally via the path matching src/ or tests/ 
. That presumably breaks with auto-detected binaries, benchmarks, examples, and using <target>.path = to change where the library/integration tests are stored.
An option would be to add new flags at the cargo level. One that run all [lib] and [[bin]] unit tests that exist. Another one that runs all [[test]] integration tests without running the [lib] and [[bin]] unit tests.
cargo test --lib --bins is almost the first flag, but only if you have a library, for a binary-only package it errors. Maybe that should be considered a bug and changed to just a warning.
I had assumed --tests would be the second flag, referring to the [[test]] targets in the same way --bins refers to the [[bin]] targets. Rather it seems to run all kinds of targets with test = true (which [lib], [[test]] and [[bin]] enable by default) while excluding doctest and the "compile but don't execute" behavior of other targets.
cargo test --test '*' will execute just integration tests.
As mentioned, cargo test --lib --bins will run just the unittests.
Thanks!
cargo test --help does not mention the --test flag, although it is in cargo help test and mentioned in passing in the rust book - but does not mention that it is glob matching.
As noted above cargo test --lib --bins will fail when there is bin but no lib. So an automation system would have to determine if the project is bin only, lib only or both in order to separate unit and integration testing.
Changing this to a warning would solve that. Similarly --test '*' errors out if there are no integration tests.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.