Testing the compiler

For the purposes of making rustc as polished as possible I think it makes sense to move toward the ‘ui’ testing model of capturing all of rustc’s output for every failure case.

There are two things though I’d like you to keep in mind.

First, alternate implementations. As more implementations of Rust appear they will need to test themselves against the reference implementation. Realistically, this means they will be using our test suite and our test suite needs to evolve into a usable reference test suite. The JetBrains people are already using our test suite this way (cc @matklad). Exact UI testing is in opposition to this goal - it seems likely to me that alternate implementations would not be inclined (or maybe even able) to reproduce rustc’s output, and certainly it would not be the initial focus of any implementation of Rust. Some simple affordances can make this work though - e.g. the test harness can have a more forgiving mode where it’s only checking error codes (it’s reasonable for our error codes to be part of the language spec that other implementations have to match).

Second, internationalization. Although it isn’t certain that we will internationalize the compiler, there will continue to be pressure to do so. I don’t know what that would mean for testing - it may not even be worth running the test suite in other languages, or i18n might have its own set of tests.