Thoughts about integrating testing into the Rust documentation


#1

Hey,

Before I proceed, I want to say thanks to @bjz for proof-reading this post and getting me started in Rust! He’s said that @steveklabnik and @carols10cents are working hard on a rewrite of the book, so it would be great to hear how this fits in with the new direction.


I’ve started looking at Rust and am very impressed by the type system. The Rust Book has been great and explains a lot of the language concepts.

As somebody who comes from both static and dynamic languages backgrounds with a heavy focus on TDD (esp in dynamic world), I noticed though that testing is only introduced later in a different section as a side-note.

It would be great to see how types and testing can be used together in the development process from the start (via test-first and type-first approaches). The fact that Rust has a unit testing framework built-in means that we have a great opportunity to develop this culture. This fits in with Rust’s goals of focusing on quality and safety.

With that in mind I would like to hear what others think and suggest the following to start with:

Start the book with the basics of testing

It is really great that Rust comes with built-in, basic testing framework. After explaining bare minimum, we could introduce the test framework to be used in most other parts of the Rust Book.

We could explain few basics without diving into the details:

  • how to run the tests (cargo test)
  • structure ([cfg(test)], [test], super:: should be enough)
  • no need to mention integration tests at the beginning yet
  • provide a more detailed section on testing, past the basics (reference quickcheck and other tools)

Supplement the Rust Book with appropriate test code

Probably most of the code, starting from 4.5 (if), can use tests instead of println!.

For example, instead of:

let x = 5;

if x == 5 {
    println!("x is five!");
} else if x == 6 {
    println!("x is six!");
} else {
    println!("x is not five or six :(");
}

We could offer something like:

fn if_example() -> &'static str {
  let x = 5;

  if x == 5 {
      "x is five!"
  } else if x == 6 {
      "x is six!"
  } else {
      "x is not five or six :("
  }
}

# [cfg(test)]
mod tests {
    # [test]
    fn if_example_returns_five() {
        assert_eq!(super::if_example(), "x is five");
    }
}

Then as the examples get more complex, the test-driven book would be even more useful as the readers would be able to think about structuring the code better too.

This would advocate experimentation through the automated testing, which will naturally transition over into the systems that Rust noobs like me would start working on.

Advocate the quality through automated testing in the Guessing Game

Currently the Guessing Game is a showcase of the Rust capabilities which is awesome. But the way I see it - it is 30+ lines of production code without a single test.

It is great to showcase it, which I imagine is the main purpose. But it isn’t great at building the quality into the culture of Rust Engineers’ minds.

If we will try to test it, it basically becomes impossible without a good refactoring (and we can’t refactor without tests).

So a better example would be to structure the Guessing Game with testing in mind and maybe apply some of the TDD techniques (or other test approaches).

For example, with testing in mind, we could implement some tests first.

But we’d have to think about the types to even compile, so we’d start with:

enum GuessResult {
    Smaller,
    Bigger,
    Guessed,
    BadInput
}

then try to write some unit tests for our system:

// Start with some tests

# [cfg(test)]
mod tests {
    use super::{game_over, GuessResult};

    # [test]
    fn game_over_is_true_when_guessed() {
        assert_eq!(game_over(GuessResult::Guessed), true);
    }

    # [test]
    fn game_over_is_false_if_not_guessed() {
        assert_eq!(game_over(GuessResult::Smaller), false);
        assert_eq!(game_over(GuessResult::Bigger), false);
        assert_eq!(game_over(GuessResult::BadInput), false);
    }

    # [test]
    fn process_guess_returns_correct_guess_result() {
      // ...
    }

    # [test]
    fn print_guess_prints_formatted_result() {
      // ...
    }
}

Then use unimplemented!() as a placeholder. And then start implementing it.

// parts are ommitted...

fn process_guess(line: std::io::Result<String>, number_to_guess: i32) -> GuessResult {
  unimplemented!()
}

fn print_guess(result: GuessResult) -> GuessResult {
  unimplemented!()
}

fn game_over(result: &GuessResult) -> bool {
  unimplemented!()
}


fn main() {
    let input = io::stdin();
    let lock = input.lock();

    let number_to_guess = rand::thread_rng().gen_range(1, 1001);

    let guesses = lock.lines()
        .map(|line| process_guess(line, number_to_guess))
        .map(|guess_result| print_guess(guess_result))
        .take_while(|guess_result| !game_over(guess_result));

    println!("Attempts: {}", guesses.count());
}

This, while not fully tested, is a big step towards it. Later on it can be covered with an integration test too, as a bit more advanced subject.

UPDATE: my noob attempt at writing the Guess Game in a more testable way.

Looking forward to hearing what other think on this matter!


#2

Yeah, as I said, I’m not sure the status of the rewrite of the docs, and how testing fits in with it. There is a challenging balancing act between providing a language reference and a guide on how to get started with the language. There are also challenges in figuring out how to slowly introduce language features to readers without overwhelming readers, and catering to many skill levels. This might mean that we might need to be necessarily conservative with the documentation - I dunno. :confused:

It might be useful to show somewhere how types can be used as an iterative part of the design process. I don’t know if this belongs in the docs or somewhere else… as opposed to pure TDD, I’ve found that it’s loosely more along the lines of:

  1. specify your invariants as best as possible in your types by writing a prototype API and verifying that it makes sense to the type checker
  2. write out your behavioral specifications in your tests, further refining your API as you learn about the domain
  3. define your implementation based on those specifications, refining the work in the previous two steps as you go

Anyway, much food for thought - it’s always good to get a fresh perspective. I’m very interested in what others have to say on this, especially those who are more familiar with where things currently stand in the land of documentation!


#3

Yeah, good balance between ease of reading and testing is very important and probably very hard to get right.

I feel like testing is a very small price to pay for the benefits, especially in comparison to the learning curve in other parts of the docs/Rust language in general. It feels to me (but I’m really new to Rust) that it makes more sense to lean towards the testing and API prototyping side (instead of sheer simplicity).


#4

Thank you for caring about the book! I really appreciate threads like this. :heart:

So, way back when, I actually did this, with my “Rust for Rubyists” project. I think this is a very valid approach for a book, but I’m not sure it’s a valid approach for the book. Here’s why:

TRPL has some odd constraints, given that it’s official documentation. Or rather, as a member of the core team and doc team lead, I feel that the book has some constraints. Here’s a non-exhaustive list:

  • The book is the primary way that new users learn Rust from the Rust project itself. That means that it cannot make assumptions about the backgrounds of users coming to Rust, unless we as a project decide that we want to make assumptions about Rust programmers. And we’ve decided that we explicitly want to be welcoming to a large background of programmers.
  • Because we cannot make this kind of assumption, we also can’t assume that people will be familiar enough with certain constructs to just gloss over them right away. Another way of putting this is “forward references should be minimal.”
  • We should not assume that everyone learning Rust is a professional developer, and skilled at related-but-not-exactly-programming tasks like testing.
  • While testing is useful in Rust, TDD is not something that we can assume that people want to do, know how to do, or find to be a valid approach.

So this is why I didn’t start TRPL out with testing stuff. While testing is useful and I love it, the point of the book is to teach Rust, not to teach testing. A “test-first” approach for the book would lead people to assume that Rust is often test-first, which I think is actually fairly misleading. And Rust is missing several important things that make a test-first approach work well, like mocking.

Beyond that, you can’t teach testing as literally the first thing in Rust. Teaching the output of cargo new, with the smallest, idiomatic setup for tests, would include:

  • functions
  • attributes
  • modules
  • conditional compilation
  • macros
  • use

It’s non-trivial. That’s why I prefer to introduce testing later, after you have a basic handle on the language. We can show you something that’s more idiomatic, without distracting you by saying “ignore all this stuff.”

In other words, I find “Time to learn Rust! Okay, before we do that, let’s talk about testing. Ignore this Rust code, you’ll figure it out” to be not a great start for the book that TRPL is trying to be. That doesn’t mean another book might not work that way!

With this example, you’ve almost doubled the lines of code, and introduced even more new features, like lifetimes. And, I would argue that this is a very, very brittle test. I’d flag it in code review if this were a “real” project.

The purpose of the Guessing Game is to bootstrap someone’s understanding of Rust if they’re a “dive in heads first” kind of person. Making the code “production quality” isn’t a goal. If it were, I’d have written some bits differently. Specifically to make it testable. But doing so introduces a ton of extra concepts that distract from the “here’s a fairly direct port of a classic coding example, see, Rust isn’t that hard or weird!” aspect of it.

The length here explodes as well. The book is already very long. Making everything TDD would make it much, much longer.

Another practical consideration

The book is too far along to re-work it from the ground up again. The initial chapters are basically locked-in at this point. I have already written two books on Rust and am going through this ground-up re-write right now, at some point, I would like to do something else with my life :smile: Re-working the book Yet Again would destroy all of this forward progress; doing it right would be a complete re-write. And a complete re-write would be Yet Another Book.

In conclusion

So yeah, I appreciate the idea here, and the thought you’ve put in. I would love to see a “test-driven Rust” book. But we cannot change TRPL in this way.


#5

I basically agree with everything that @steveklabnik has said here. I am also from a Ruby background, and I got started with Rust through Steve’s Rust for Rubyists, and I indeed found it to be very comforting to start with tests. That said, I know I’m not very representative of all the programmers we want to bring into Rust with the book :slight_smile:

We had another suggestion recently for adding fully-featured examples to each chapter of the book, which is a similarly large change and level of effort as this suggestion is. What I suggested to that person is that they create an online companion resource to the book, and I think that could work well in this case as well. If you wanted to, you could write an “Intro to Testing in Rust” chapter that people could choose to read early on in tandem with the book, and you could have alternate code examples that use tests instead. The larger code examples in the book are numbered, so finding the correspondence between the book and this other resource could work out nicely. Please feel free to create something like this!


#6

Great responses! I wonder if this kind of thing might be good in a series of blog posts - kind of going off the Rust book in parallel and showing how you can do it in a test driven style. I don’t envy you in having to try to get folks up to speed with so many concepts all at once!


#7

I would really like some more discussion regarding these topics, and sharing of knowledge on how people have tackled these problems. Folks coming from dynamic languages or languages with reflection would probably be very interested in how they can still maintain test coverage in a language like Rust. But I have a feeling that this is a space where we are all still learning and techniques are still evolving, so formalizing testing processes in the official documentation is probably premature.

Anyway, this kind of discussion probably makes more sense on The Other Forum™.


#8

For comparison I found the book http://www.obeythetestinggoat.com/ a useful guide to ttd. In the book he writes the Django tutorial in a ttd style.


#9

Thanks for the reply @steveklabnik and @carols10cents . Appreciate it your time and explanations.

What you mention makes sense, the Rust Book is designed to be a Language Guide and not the engineering practices guide.

I think a better way to tackle this would indeed be a separate Testing with Rust book/guide/tutorial.

Sorry for wasting your time :slight_smile:


#10

Not a waste at all!


#11

With all the authority of being a random person on the internet, I would definitely read “The opinionated rewrite of the Rust Book to use TDD.” I’d even recommend it, for peepal that have read the Rust Book and have some experience with the language and want some advice on better software design practices.