Discovering the current test (name)

libs
7

What are we contemplating here? Does the #[test] attribute have a stable interface that must be obeyed, or can we add to it in some way? If we can add to it, I'd love to see a trait defined that tests can implement that allows forward-compatible extension of tests. Purely as a strawman:

// Configurations

#[derive(Debug, Default)]
pub struct TestConfigurationVersion2 {
    test_name: String,
    // Whatever else is being contemplated for this version
}

#[derive(Debug)]
#[non_exhaustive]
pub enum TestConfiguration {
    Version1,
    Version2(TestConfigurationVersion2),
}

// Errors

#[derive(Debug)]
pub enum TestConfigurationErrorVersion2 {
    // Indicates that the test cannot handle the given version of
    // the test framework's incoming configuration.  Ideally, we'll
    // add some information here that the test framework can use
    // to try and redo the test using an earlier version of the
    // configuration, but it's entirely possible to simple try
    // all supported variants of `TestConfiguration` until one of
    // them works, and to report the test as a failure if that
    // doesn't happen.
    VersionError,

    Failed(), // Something that derives std::error::Error

    /// If this is returned, then the test passed, BUT the test has a
    /// suggestion for another test to try.  This makes it possible to
    /// build intelligent fuzzers that use the results of prior tests
    /// to narrow down what caused the test to pass.
    PassedTestNext(TestConfigurationVersion2),

    /// If this is returned, then the test failed, BUT the test has a
    /// suggestion for another test to try.  This makes it possible to
    /// build intelligent fuzzers that use the results of prior tests
    /// to narrow down what caused the test to pass.
    FailedTestNext(TestConfigurationVersion2),
}

#[derive(Debug)]
#[non_exhaustive]
pub enum TestConfigurationError {
    Version1,
    Version2(TestConfigurationErrorVersion2),
}

// The test trait

pub trait TestTrait {
    #[allow(unused_variables)]
    fn constructor(
        config: TestConfiguration,
    ) -> Result<Self, TestConfigurationError>
    where
        Self: Sized,
        Self: Default,
    {
        match config {
            TestConfiguration::Version1 => Ok(Default::default()),
            TestConfiguration::Version2(..) => {
                Err(TestConfigurationError::Version2(
                    TestConfigurationErrorVersion2::VersionError,
                ))
            },
        }
    }

    fn test(self) -> Result<(), TestConfigurationError>
    where
        Self: Sized;
}

The idea is that #[test] would become more like a derive macro, converting all current tests into objects that implements TestTrait, but which have a do-nothing constructor and a test method that is compatible with the current test methods (there will be some fiddling to get this right, I banged the code out pretty fast without any real testing).

The main idea is that the interface should be forward-compatible. Although the configurations for any given version of libtest are fixed, we can add in as many versions as we wish via new variants to TestConfiguration. Similar statements can be made for TestConfigurationError.

In addition, a test is able to return a new test configuration that can be used for a different iteration of the test. That could be useful for feedback-driven fuzzing tests where (once you've minimized the configuration causing the issues), you want to save the configuration in a regression database.

There are a whole host of other issues to solve here, like whether or not async tests would be a good idea, etc., but this may be a ways forward that doesn't break older tests.