cargo test -- --test-threads=1 can be used to run all tests consecutively. I'd like to label individual tests as being run consecutively, while still letting most tests run in parallel by default. Would be nice to have some kind of attribute parameter for doing this, like the following:
#[test(serial)]
fn testing() {
let left = func1();
let right = 5;
assert_eq!(left, right);
}
Note: Cross-posted on URLO
I made a similar post, but not identical. I asked if this feature was available in URLO, and the answer was no, so then I decided to ask in the internals forum if we could impl this feature. Did I do something wrong?
Linking to the original thread that you were asking for stdlib-level support would have been useful as a signal. In any case, I don't see an issue with serial_test covering this use case (I use it myself). What would be the benefit of putting it into libtest proper (which has a very foggy stability path anyways)?
The problem with serial_test is that it can only serialize with respect to other tests marked with its macros; it can't serialize with respect to all tests.
Integrating this into our built-in test machinery would allow it to serialize with respect to all tests.
Ah, yes, that makes sense. FWIW, CMake calls this RUN_SERIAL.
There is exactly that:
and if you don't want a crate:
static DELICATE_IMPLIED_SHARED_STATE: Mutex<()> = Mutex::new(());
#[test]
fn t() {
let _lock = DELICATE_IMPLIED_SHARED_STATE.lock().unwrap();
Why would you want that? The rest of the tests shouldn't be stateful, otherwise they can't execute in parallel either.
In projects I work on (that use CMake/CTest), the property is used for tests that do things like:
Inserting a mutex into every test because one of these tests exists ends up penalizing all tests against each other rather than some specific tests against the rest as a group.
Totally serializing unit tests leads to much longer wait times than necessary, i.e. it's quite unergonomic. So serial test execution should default to only serializing threads within a group, but not blocking tests that don't need serialization.
And for the heavy stuff we already have integration tests, which are executed serially at the outermost level. I.e. each file in tests/*.rs is a separate test crate.
I wonder if that's something which can be relied on? They're currently serial, because the integration between Cargo and rustc's test runner is weak, but hopefully this could be improved someday. For example, doc tests recently stopped being separate serialized executables.
The CMake project in question has quite a few integration tests. While #[test] tests in Rust are typically unit tests, they can be integration-like tests. See this test which uses infrastructure to drive the test using mockup implementations of the traits that do minimal, but substantial, work to emulate what forges do when interacted with (git hooks and "webhooks") but within the test process. These aren't performance sensitive, but I can certainly imagine existing projects wanting to preserve testing in a Rust port where such tests might want to say "I need execution exclusivity" without wanting to block other tests from running in parallel with each other. I also have these tests which are separate because they tickle process-wide state at the kernel level that interferes with other tests. That project also uses serial_test to make sure quota-tickling tests don't out-quota each other into spurious failures.
From reading the docs of serial_test, I have the impression that applying serial_test::serial attribute on 10 tests will make all of them run on the same CPU core consecutively, waiting for one to finish before starting the next test (correct me if I'm wrong). If so, this would be an unideal situation for my use case (I'm testing the parsing of environment variables) because when I use serial_test::serial, my intention is to just keep them isolated from each other so that an environment variable created in one test won't be read by another test that also gets affected by that environment variable.
So running all the tests in parallel should still be possible to do for my use case, but I just want to prevent cross-contamination between them. Using the serial attribute solves the cross-contamination (as long as I remove the envs at the end of each unit test that initiated the envs), but it feels unnecessary to hinder the parallelism since you could in theory keep the unit tests isolated from each other without having to run them on the same core. At least there should be some way to let each unit test have exclusive access to one core until it's finished, while still allowing the other tests to run on the other cores with or without exclusive access to their cores.
There probably should be a way to prevent cross-contamination between unit tests even by letting multiple tests run in parallel on the same core, as long as each execution thread has a dedicated set of environment variables that doesn't cross-contaminate with the other execution threads. Would be nice if we had an attribute we could put on the unit test to signify "this unit test should be isolated from the other unit tests but without blocking the other unit tests from running in parallel". Or perhaps this should be the default?
You're mixing up concepts and official naming here. We have
The latter two both use [test] attributes. What I'm saying is that if you're running into serialization problems then one easy fix is to not write your de-facto integration test as what is considered a unit test. Put it in the right place, which happens to result in the right behavior.
This would be a much simpler then trying to bend libtest.
Environment variables are per-process, not per-thread. There's been discussion about having an attribute to spawn tests into processes, but idk what the state of that is.
Integration tests help here too.
Since you're messing with process-wide state, I don't think a simple Mutex (which is what serial_test uses…I don't think there's any CPU pinning involved) will help. Such cleanup is hard in tests without avoiding all panic! codepaths within it. Suppose you have:
#[test]
fn env_test() {
let state = prep_env();
something_that_panics();
restore_env(state);
}
The prepared environment state is never restored properly and other such tests are still unhappy. One cannot just catch_unwind in general because there is also the #[should_panic] attribute (particular tests can do this, but then you're reimplementing libtest logic to separate "real" panics from "wtf" panics. Since you're interacting with process-wide state, I think tests/*.rs is better for this because then you're using the process isolation cleanup of the OS which is actually reliable for these tests.
I did not know this was currently the case and I have a project that would benefit from tests/*.rs being executed in parallel (many individual tests of a command line tool, and I already wrote the code to isolate invocations from each other because I thought they would be run in parallel).
By way of example, consider a test of a configuration environment variable. You'll could test that by setting the variable to a value and invoking functionality that will change as a result of that variable. But if doing do, you need to exclude every other test whose behavior might be affected by that variable, which may be every test you have of your library.
IIRC cargo nextest will run the different integration test binaries in parallel. It also has functionality for serial testing / mutual exclusion. Just be aware it doesn't run doctests yet; that has to go through the rustdoc+libtest integration since compilation is part of the test.
nextest also runs each test in its own process, fwiw.
Ignoring the "cores" part, it seems like the simple solution example could be extended to use RwLock, with all non-interfering tests getting a shared (read) lock, and interfering tests getting an exclusive (write) lock.