Discovering the current test (name)

I am maintaining a snapshot testing library (insta) and for it to work best, it tries to discover the name of the test by inspecting the thread name.

Unfortunately this does not work if --threads is set to 1 on rust test (bug report here) but it also can run into platform limitations on the length of the thread name.

Since this issue is quite frustrating to me I was trying to see if there are no better ways to expose this information. In the ideal case there was a test support module one can import to access information about the test. Something like test::current_test_name().

Has there been some discussion already about providing minimal introspection to the integrated testing system?

2 Likes

Worst worst case we could provide an #[insta::test] macro which sets our own thread-local for the test name. Wouldn't even require syn/quote, theoretically.

(Practically: using syn with just the minimal derive support to extract the function name would be way easier than finding it manually, but it'd be reasonable doable.)

This reminds me of the recent discussions about adding an ability to skip/ignore the test based on run-time criteria. And that reminds me of the plans we had a couple of years back regarding custom test frameworks, to allow experimentation in the community.

I don't think there has been much progress since on the pluggable test framework, and, personally, I quite like that rust ended up having just one testing framework. I like that there's little variation in terms of testing across the projects, and it seems that libtest is actually good enough (contrast with Python's unittest/pytest). The "single test framework" goes against Rust's value of empowering the user to do whatever, but, similarly to the "single build system", I think in this case this brings a lot of productivity benefits.

So.... Maybe there's a space for an RFC which proposes some plan for the evolution of current libtest? As a couple of strawmans:

#[test]
fn my_new_test_param(t: &mut test::Tester) {
    if std::env::var("RUN_SLOW_TESTS").is_err() {
        eprintln!("skipping {}", t.name());
        t.skip();
        return;
    }
}

#[test]
fn my_new_test_thread_local() {
    if std::env::var("RUN_SLOW_TESTS").is_err() {
        let t = test::current().unwrap();
        eprintln!("skipping {}", t.name());
        t.skip();
        return;
    }
}
2 Likes

I'm with you in that it's nice that rust has a defacto test framework these days. I would not want this discussion to be a stepping stone to going back to that discussion :slight_smile:

In terms of your strawman proposal I vastly prefer the thread local option because a lot of systems already need to access this information very far removed from where the test is declared. I'm happy to try to work on an RFC for this.

As the author of the skippable test pre-RFC, I think that thread-locals are just about the worst possible solution to this because it means that one has to think differently about APIs such as async, thread::spawn, or just about anything under rayon. We already had issues with println!() and its thread local usage behind the scenes. Let's please not add new ones.

Personally, I think just having either a magic macro or symbol that is available in #[test] functions is suitable. I have crates which like to use the test name and I just pass it around explicitly from the top-level function (Vim's word-in-buffer completion helping out from there).

2 Likes

I very much disagree, most of my libraries with good test coverage end up with massive hacky macro_rules! based test-generators to do parameterized testing. Maybe it's possible to do this as some nice proc-macro framework that generates normal tests, but since you can't generate function names like add(1, 1) == 2 I still think there's a lot of room for more featureful custom test definitions.

What are we contemplating here? Does the #[test] attribute have a stable interface that must be obeyed, or can we add to it in some way? If we can add to it, I'd love to see a trait defined that tests can implement that allows forward-compatible extension of tests. Purely as a strawman:

// Configurations

#[derive(Debug, Default)]
pub struct TestConfigurationVersion2 {
    test_name: String,
    // Whatever else is being contemplated for this version
}

#[derive(Debug)]
#[non_exhaustive]
pub enum TestConfiguration {
    Version1,
    Version2(TestConfigurationVersion2),
}

// Errors

#[derive(Debug)]
pub enum TestConfigurationErrorVersion2 {
    // Indicates that the test cannot handle the given version of
    // the test framework's incoming configuration.  Ideally, we'll
    // add some information here that the test framework can use
    // to try and redo the test using an earlier version of the
    // configuration, but it's entirely possible to simple try
    // all supported variants of `TestConfiguration` until one of
    // them works, and to report the test as a failure if that
    // doesn't happen.
    VersionError,

    Failed(), // Something that derives std::error::Error

    /// If this is returned, then the test passed, BUT the test has a
    /// suggestion for another test to try.  This makes it possible to
    /// build intelligent fuzzers that use the results of prior tests
    /// to narrow down what caused the test to pass.
    PassedTestNext(TestConfigurationVersion2),

    /// If this is returned, then the test failed, BUT the test has a
    /// suggestion for another test to try.  This makes it possible to
    /// build intelligent fuzzers that use the results of prior tests
    /// to narrow down what caused the test to pass.
    FailedTestNext(TestConfigurationVersion2),
}

#[derive(Debug)]
#[non_exhaustive]
pub enum TestConfigurationError {
    Version1,
    Version2(TestConfigurationErrorVersion2),
}

// The test trait

pub trait TestTrait {
    #[allow(unused_variables)]
    fn constructor(
        config: TestConfiguration,
    ) -> Result<Self, TestConfigurationError>
    where
        Self: Sized,
        Self: Default,
    {
        match config {
            TestConfiguration::Version1 => Ok(Default::default()),
            TestConfiguration::Version2(..) => {
                Err(TestConfigurationError::Version2(
                    TestConfigurationErrorVersion2::VersionError,
                ))
            },
        }
    }

    fn test(self) -> Result<(), TestConfigurationError>
    where
        Self: Sized;
}

The idea is that #[test] would become more like a derive macro, converting all current tests into objects that implements TestTrait, but which have a do-nothing constructor and a test method that is compatible with the current test methods (there will be some fiddling to get this right, I banged the code out pretty fast without any real testing).

The main idea is that the interface should be forward-compatible. Although the configurations for any given version of libtest are fixed, we can add in as many versions as we wish via new variants to TestConfiguration. Similar statements can be made for TestConfigurationError.

In addition, a test is able to return a new test configuration that can be used for a different iteration of the test. That could be useful for feedback-driven fuzzing tests where (once you've minimized the configuration causing the issues), you want to save the configuration in a regression database.

There are a whole host of other issues to solve here, like whether or not async tests would be a good idea, etc., but this may be a ways forward that doesn't break older tests.

That assumes that test skipping will be done by random threads. IMO tests should only be skippable by the test thread itself. I've used test skipping a decent amount in Objective-C. I have never needed to skip a test somewhere deep within the test. My test skips have always been the first line of the test function (perhaps wrapped in an if-block). I'm baffled by the need or desire to skip tests deep within arbitrary code running on arbitrary threads.

1 Like

I'm still preferring either -> ExitCode as a return type (and skipping based on the value) or attributes myself. The return code does use "magic numbers", but also works for test executables under tests/. The attribute is what I already have implemented :slight_smile: .

I agree, but feel free to read through the thread for the arguments in favor of panic-based or thread-local settings for more details.

2 Likes