(the following text was initially published in this issue)
Currently there is no standard way to specify what example code example writes to stdout in doctests, thus makinf it impossible to test this output. Borrowing from Python I would like to propose to add something like \\> foo to doctest to specify stdout output of snippets. For example:
Using it cargo test will be able to test standard output of the program, thus making some testing cases a lot easier. Additionally it will be an incentive for crate authors to use this functionality thus making examples easier to understand for crate users.
Because output will be checked after program execution this example will pass tests:
println!("foo");
println!("bar");
//> foo
//> bar
While it should be considered a bad practice to write doctests in such way, checking relations between code line and output will be technically challenging, so I think it’s better to leave it for crate authors to handle.
Unresolved questions:
Should stderr be considered as stdout for testing purposes, or is it better to add similar functionality for stderr as well, e.g. //stderr> foo?
How about testing other IO operations: network, databases, etc.? For example by defining custom handlers.
Should stderr be considered as stdout for testing purposes,
As a possible point of reference I think python doctest completely ignores stderr; but it also has fundamentally different syntax (leading sigils are on the code rather than the stdout).
In general I like the idea. Some concerns I feel need to be addressed:
Concerns
Backcompat: Tests without stdout
We need to decide how to deal with the existing tests that have unmarked stdout. Either:
Suppose they don't exist. I think this is a bad idea. The main issue will be functions that write stdout as a side-effect, which I fear might exist in some application code (which might not be on crates.io, so crater will miss it)
Be lenient. Allow for lines to appear in stdout which are not mentioned in the doctest.
What about the converse problem? (comments that incidentally begin with//>) Do we just crater for this?
Require an additional annotation. E.g. ```rust,check_stdout```
Seems easy to accidentally write tests that don't actually test anything. Can be fixed with a lint maybe? (seems prone to false negatives though; many doctests are just to test that something compiles)
The annotation seems to be the only defensible option to me, but I hate being able to write tests that actually do nothing.
Increased requirements on stability of Debug
Not too long ago the Debug impl for floats was changed to stop rendering 1.0 as 1; I doubt this would have been possible if doctests sensitive to Debug were more common.
I had in mind the following approach: if doctest does not have lines starting with \\> (with space at the end) stdout will not be checked. I think we can ignore risk of having such lines in already existing doctests. But check_stdout is a viable alternative as well.
Increased requirements on stability of Debug
Writing tests dependent on Debug output is possible today as well, but yes it's matter of popularity. I think keeping doctests (and tests overall) passable on every release is not part of the Rust stability promise. (although it's of course very nice to have) If I wrong I hope someone from Rust team will correct me.
I definitely think stderr and stdout should be testable independently (I would say that, I am the person who wrote the eprintln! spec ) but //2> requires one to know the Unix-specific factoid that stderr is file descriptor 2. I mean, lots of people do know that factoid, if only because sh makes you know it too, but maybe we shouldn’t be a new reason you have to know it? On the other hand, //stderr> is long enough to make me worry about both typing fatigue and visual clutter. So I propose //e> to go with the e at the beginning of eprintln.
) but