Should the pretty test be fatal if source comparison fails?


#1

Hi! I’m trying to break into working on rustc, so I picked a random bug (https://github.com/rust-lang/rust/issues/12590) and started working on it. As I was working on creating a test case to reveal the bug, I noticed that revealing the bug does not actually fail the test run.

If I have a file like:

// pp-exact
// The next line should not be expanded
mod issue_12590_b;

The test fails, and it shows the message I want to see (that the output doesn’t match the input). But it actually has two error messages, the second being that the output doesn’t typecheck because of a lack of fn main.

But if I add fn main, then the whole test passes! So I track this down to src/tools/compiletest/src/runtest.rs line 462. The compare_source logs an error with TestCx#error, but it’s not a fatal error. Then the typecheck runs, and that error is fatal so it actually shows an error in the tests. If the typecheck clears, then no fatal error is logged and the entire spec passes, and I can’t even see the error message about the mismatched pretty output!

So:

  1. Do I fix compiletest to make compare_source be fatal?
  2. Do I introduce a new way to invoke compare_source in a fatal way so I can do my test?
  3. Is there a chance other tests in the pretty suite expected compare_source to be fatal when using pp-exact?
  4. Am I just doing it wrong?

edit: Just found this commit that changes the behavior to no longer panic: https://github.com/rust-lang/rust/commit/7948afdc53cabf9330662ec894bf668839880a3c


#2

I reported the issue, and filed a pull request: