# Pre-RFC \`#\[test\]\` on \`mod\` **URL:** \ **Category:** language design\ **Created:** [June 3, 2022, 9:53am UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741 "2022-06-03T09:53:10Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![yanganto](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/yanganto/32/7061_2.png) [@yanganto](https://internals.rust-lang.org/u/yanganto)\ **Post date:** [June 3, 2022, 9:53am UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/1 "2022-06-03T09:53:10Z")
# Introduction > **(deleted)** > > Some people want to set the result of a test case fail but still, run full of the test as mentioned [here](https://github.com/rust-lang/rust/pull/96574#issuecomment-1117293041). If Rust can provide `#[test]` on the module, we can easier run all the functions in that module, let the module become a test case with a set of functions that need to be tested, and also have only one test result of the module. If some tests fail, the other tests still run. Following is the thing I want to propose. # Proposal Let `#[test]` apply to the module, and the module will be a single integrated test case, all the functions in the module will run, if any function fails, the test case fails. If more than one functions fail, the test case fails, and also all the error message are collected inside the test case, and still easy to debug on it, and the summary will keep as clean as possible. ```rust #[cfg(test)] #[test] mod infra { fn infra_1_works () {...} fn infra_2_works () {...} } #[test] fn it_works() {...} ``` ```plaintext running 2 tests test infra ... ok test it_works ... ok test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` If there is one error in there, it will probably be like this. ```rust #[cfg(test)] #[test] mod infra { fn infra_1_works () {...} fn infra_2_fails () {assert!(fail)} } #[test] fn it_works() {...} ``` ```plaintext running 2 tests test infra ... FAILED test it_works ... ok failures: ---- infra stdout ---- thread 'infra::tinfra_2_fails' panicked at 'assertion failed: false', src/lib.rs:0:0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: infra_test test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` If there are more error in there, it will probably be like this. ```rust #[cfg(test)] #[test] mod infra { fn infra_1_fails () {assert!(fail)} fn infra_2_fails () {assert!(fail)} } #[test] fn it_fails() {assert!(fail)} ``` ```plaintext running 2 tests test infra ... FAILED test it_fails ... FAILED failures: ---- infra_test stdout ---- thread 'infra::infra_1_fails' panicked at 'assertion failed: false', src/lib.rs:0:0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'infra::infra_2_fails' panicked at 'assertion failed: false', src/lib.rs:0:0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- it_fail stdout ---- thread 'it_fails' panicked at 'assertion failed: false', src/lib.rs:0:0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: infra it_fails test result: FAILED. 0 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` As you can see, all the functions in the `#[test]` module will run and the error message will be collected in the test case of `#[test]` module. The `#[test]` module is just a test case, and collect all the error message from each function. # Functions Inside the `#[test]` mod The `#[test]` mod is a test case that is a set of the test functions and only all the test functions that pass the test case of the mod will be passed. All the test functions in the `#[test]` mod run dependently. By nature, we use the null input function as a test. and some helper functions with inputs, we do not need to tell whether the function is a helper or not. If there is a useful function without inputs that should not be run in a test mod, we could use `#[test(not)]` to exclude it. We also discussed this in [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/.60.23.5Btest.5D.60.20on.20.60mod.60/near/284929052). Currently, the RFC does not propose `#[test(not)]`, because this kind function should not be in a test mod. The functions in `#[test]` `mod` do not guarantee to run in any form, the way to run these functions is the same as `#[test]` on `fn`. The `#[test]` on `mod` changes the test counting and summary on `cargo test`. The proposal does not change the way running the test functions, and the functions inside the `#[test]` `mod` are parts of a test case. This means the test filter works on `#[test]` `mod` but not any functions inside the mod. Because I think this may not be useful but has a higher complexity for implementation, If you highly recommended doing a filter please help me to refine this. πŸ™ # Nesting issues Currently, `#[test]` on mod let a set of functions become a test. The proposal do not addressing nesting mod issues, the similar is not addressing with `#[test]` on functions. ```rust #[test] fn it_works() { #[test] fn it_fails() { assert!(false) } } ``` ```rust running 1 test test it_works ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s ``` The behavior of nesting `#[test]` mod will be the same, only the first level no input function will become a set to be a test case. # Other test-related attributes A test-related attribute still works on the test function, the `#[test]` on the mod is building a test case with a set of the test functions. Following is the example to work woth `#[should_panic]` ```rust #[cfg(test)] #[test] mod infra { fn it_works () {..} #[should_panic] fn it_panic () {panic!("...")} } ``` Currently, the proposal does not raise the test-related attribute on the mod level. If you highly recommended doing this because I think this may not be easy to read, please help me to refine this. πŸ™ # What are the benefits of this design - keep the summary of `cargo test` clean and show the important things - Find issues with running `cargo test` once ---
**Author:** ![mathstuf](https://avatars.discourse-cdn.com/v4/letter/m/958977/32.png) [@mathstuf](https://internals.rust-lang.org/u/mathstuf)\ **Post date:** [June 3, 2022, 2:24pm UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/2 "2022-06-03T14:24:06Z")
`#[test]` is implemented as a proc macro. I think describing this in terms of what it does is useful. In particular: - what test function is registered to the list of test functions? - how does it call each test? - how should the generated function guard against cascade failures (e.g., `#[should_panic]` and unexpected panics). - can the `#[test]` on the module use `#[should_panic]` and friends? - does this nest? `#[test] mod foo { #[test] mod bar { /* … */ } }` - where should helper functions that are not tests live? ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 3, 2022, 2:30pm UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/3 "2022-06-03T14:30:56Z")
You linked a comment that says: > I've also heard folks requesting a feature to allow them to mark a test as failed without returning/exiting/panicking just yet. I don't know about what the other folks want, but for myself, the reasons I would like to see a feature like this are not about tidying up the summary or shortening the code. They're about **continuing to run a stateful process, logging multiple failures.** For example, I could write a loop over some possible input values, and make an β€œassertion” for each item. What I want at the end is a report of _every failure_ (not just the first one), because the first failure may not be sufficiently informative; a _pattern_ of failures can be much more so. Today, I can get that by having the loop explicitly print an error (which will be captured by the test harness) and set my own failure flag which I check at the end of the loop to decide whether to panic (and cause all the previous output to be reported). Improving on this verbose manual glue would be if I could just say `test::fail()` and continue producing results. (It'd also be possible to call more functions that did their own assertions in this style, without needing to explicitly pass success/failure back.) Even better β€” for my purposes β€” would be if, optionally, this could also report _multiple results_ in the summary, _as if_ the test were multiple test functions. For example, right now I have a use for running a test for each variant of an enum. I resorted to using a macro to read the enum definition and derive `#[test]` functions β€” if single tests could produce multiple results, then this could just be a loop instead. In all of these cases, I'm trying to produce information from a stateful process β€” that is, one function (and the functions it calls). A module-of-tests cannot be stateful in that way (without introducing mutable static state, which would _work_ but is quite inelegant), so it doesn't help. ---
**Author:** ![yanganto](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/yanganto/32/7061_2.png) [@yanganto](https://internals.rust-lang.org/u/yanganto)\ **Post date:** [June 4, 2022, 7:42am UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/4 "2022-06-04T07:42:38Z")
Hi @kpreid, Thanks for the instructions, there is something I can not get from your post. Could you help me think deeper about this issue? πŸ™ The functions in the `#[test]` mod may not always simple things to use a loop to change some value to test. From my side, a test is a scenario, and it may not good that less `loop` and `if` in the function but all statements. Also the test without `loop` and `if` helps beginners to join a Rust project. > [@kpreid](#): > > A module-of-tests cannot be stateful in that way Could you help to know how to make a module-of-tests that can not be stateful? I believe the `#[test]` on the mod is a test set collection and nothing about states. ```rust // Invalid rust code #[cfg(test)] mod test { let mut state = 0; #[test] fn it_works() { state = 1; assert!(state == 0); } #[test] fn it_works_2() { assert!(state ==1); } } ``` Rust is a performance, reliability, and productivity language, the proposal try to let developer writing less code and keep things readable. ---
**Author:** ![yanganto](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/yanganto/32/7061_2.png) [@yanganto](https://internals.rust-lang.org/u/yanganto)\ **Post date:** [June 4, 2022, 8:03am UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/5 "2022-06-04T08:03:41Z")
Hi @mathstufThanks for the input. The pre-RFC is updated about the functions inside the `#[test]` on mod and nesting issues. For the implementation (The `How` questions), I hope I can find these out. If I have an update on this. I will tag you again. If there is a thing the proposal did not answer, please kindly tell me. πŸ™‚ ---
**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\ **Post date:** [June 4, 2022, 1:25pm UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/6 "2022-06-04T13:25:18Z")
I shall try to restate my point: You wrote > [@yanganto](#): > > Some people want to set the result of a test case fail but still, run full of the test as mentioned [here](https://github.com/rust-lang/rust/pull/96574#issuecomment-1117293041). I am saying that your proposed mechanism (`#[test]` modules) does not provide the additional capabilities that being able to mark a test function failed without editing that function would. I agree with you that when possible, tests should be simple and straightforward in structure. However, sometimes more complex testing code is required for good coverage. Fail-but-continue would allow more useful reporting from such tests. Your proposal does not offer this benefit; rather it is a shortening of the syntax and output of multiple test functions as we currently have them. Perhaps this is worthwhile, but it does not solve the problems which fail-but-continue would. That is all I am disagreeing with. ---
**Author:** ![mathstuf](https://avatars.discourse-cdn.com/v4/letter/m/958977/32.png) [@mathstuf](https://internals.rust-lang.org/u/mathstuf)\ **Post date:** [June 4, 2022, 2:26pm UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/7 "2022-06-04T14:26:35Z")
> [@yanganto](#): > > The proposal does not change the way running the test functions. How do I use the name filtering mechanism on these? Or can I only access the module name for this filtering behavior? ---
**Author:** ![yanganto](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/yanganto/32/7061_2.png) [@yanganto](https://internals.rust-lang.org/u/yanganto)\ **Post date:** [June 5, 2022, 7:48am UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/8 "2022-06-05T07:48:14Z")
Hi @kpreid, Thanks for your response, and clear my problem. I had assumptions based on my experience with people using test cases in mind about why they ran into problems with test cases and wanted to fail-but-continue. So it seems not related from your end, that is my bad, I will remove the description about the issue in the introduction section to avoid the problem, and keep the proposal not related to the issue. Furthermore, I will explain what I am thinking about all of these, If you are willing to take more time on this, I will really appreciate it. Just think we are developing a storage system at version 0.1, and there is a store instance there, and we try to operate on it. A test case is the following try to do basic operations in one test case. ```rust #[test] fn test_basic_operations() { insert_chunk(); edit_chunk(); // Possible fails tag_chunk_deleted(); } ``` Why people want to fail with continue is that no matter whether `edit_chrunk` works or not we still want to know if `tag_chrunk_deleted()` still works or not. Such that `#[test]` on `mod` solves this issue in the following way. ```rust #[cfg(test)] #[test] mod test_basic_operations { fn insert_chunk() {...} fn edit_chunk() {...} fn tag_chunk_deleted() {...} } ``` Furthermore, This is easy to manage test cases when go releasing and focusing on different features without touching the test case itself. For example, the version 0.1 focus on the basic chunk operation, the version 1.0 focus on the storage interface ```rust // version 0.1 #[cfg(test)] mod test_basic_operations { #[test] fn insert_chunk() {...} #[test] fn edit_chunk() {...} #[test] fn tag_chunk_deleted() {...} } ``` ```rust // version 1.0 #[cfg(test)] #[test] mod test_basic_operations { fn insert_chunk() {...} fn edit_chunk() {...} fn tag_chunk_deleted() {...} } mod test_storage_interface { #[test] fn s3_protocal() {...} #[test] fn samba_protocal() {...} } ``` From my side, this is productive not only a syntax candy, and more about the test management things. @josh also likes this proposal, there may be another good inside for this based on his experience in building something complex system. I am also willing to hear about that. If you have more ideas or suggestions on this, please tell me. Many thanks. ---
**Author:** ![yanganto](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/yanganto/32/7061_2.png) [@yanganto](https://internals.rust-lang.org/u/yanganto)\ **Post date:** [June 5, 2022, 8:33am UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/9 "2022-06-05T08:33:18Z")
Hi @mathstuf, Thanks for pointing out this, I believe that you are an expert and know more about the mechanism. The filter does not work on the functions inside the `#[test]` `mod`, because they are parts of a test case but not a test case itself. In the future, when we look much deeper at the implementation, there may be something that needs to conquer, please give a hand on this if you can. Thanks in advance. ---
**Author:** ![mathstuf](https://avatars.discourse-cdn.com/v4/letter/m/958977/32.png) [@mathstuf](https://internals.rust-lang.org/u/mathstuf)\ **Post date:** [June 5, 2022, 4:17pm UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/10 "2022-06-05T16:17:56Z")
> [@yanganto](#): > > I believe that you are an expert and know more about the mechanism. I've been in the implementation in order to work on the impl for my own pre-RFC: > [@Pre-rfc: Skippable tests](https://internals.rust-lang.org/t/pre-rfc-skippable-tests/14611): > > Summary Some test cases have indeterminite results in some cases. These cases should be distinct from a passing or failing state of a test today. It is possible to turn tests off at compile time by using features determined by a build.rs script, however, this does not work well for cross compilation or when the build and test environment are otherwise quite different. Motivation Currently, tests in Rust's libtest have only a handful of end results: success failure (panic) should panic (whic… > [@yanganto](#): > > The filter does not work on the functions inside the `#[test]` `mod`, because they are parts of a test case but not a test case itself. That's fine; I think it just needs to be considered and documented, that's all. > [@yanganto](#): > > In the future, when we look much deeper at the implementation, there may be something that needs to conquer, please give a hand on this if you can. Thanks in advance. I can certainly offer my viewpoint, but I have no real weight to get anything merged πŸ™‚ . Being able to see corner cases and holes in combinations of things is just a knack I have πŸ™‚ . ---
**Author:** ![system](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/system/32/14092_2.png) [@system](https://internals.rust-lang.org/u/system)\ **Post date:** [September 3, 2022, 4:17pm UTC](https://internals.rust-lang.org/t/pre-rfc-test-on-mod/16741/11 "2022-09-03T16:17:58Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.