Ah, that reminds me of something else. I was thinking that I would like to unify run-pass and compile-fail. I want to provide some kind of header indicating whether compilation should succeed and whether the test should be executed. This could also be done just by reference files. i.e., if each test could have multiple reference files, then:
- at least one reference file must exist.
- if a “compiler output” reference file exists, we will compare the compiler output (this lets you test for warnings on a run-pass test, for example, as well as capturing errors);
- if a “runtime output” reference file exists, we will check that the output when executed matches.
Often, when working on a feature, I would like to lump all the tests related to that feature together – some of them will be positive (code runs as it is suppose to), some of them will be negative (compilation fails when it is supposed to).
I guess in general I think it’s more useful to group tests by “feature” or “part of the code they test” than by how they should be executed.