When RUST_BACKTRACE is set, panics always cause a backtrace to be immediately printed, irrespective of whether that panic is expected and will later be caught using catch_panic. There is, to my knowledge, no way to suppress the backtrace in "expected panic" test scenarios.
This can become a performance issue in property-based tests that assert that certain operations should panic and run repeatedly over many different inputs.
Obviously, one can sidestep the problem by not setting RUST_BACKTRACE. But I would rather not do that because I think setting export RUST_BACKTRACE=1 in .bash_profile is actually the right default for a developer system. You don't want to spend hours reproducing a rare test failure just because you forgot to set the environment variable the first time.
The question I am still asking myself to this day, however, is the following: shouldn't we have something like this in std? It seems strange to me that there is currently no way to avoid the std panic backtraces when triggering an expected panic in tests, as for panic-heavy tests (like numeric types newtype tests) that generally sounds like a good idea to keep test logs readable, irrespective of performance issues.
You can use std::panic::resume_unwind to silently throw a panic. Maybe you could replace your assertions with a macro that expands to std::panic::resume_unwind when #[cfg(test)] and panic!() otherwise? You could even pass say struct AssertionFailed; as argument to resume_unwind and in your catch_unwind check that the panic was indeed an AssertionFailed rather than a panic somewhere else that shouldn't have panicked.
On one hand, I love the idea of testing code being able to easily check if it triggered the expected panic, and not any other unrelated panic. This is certainly much harder in Rust than in exception-first languages, and I'll readily admit that I normally don't bother with custom panic payloads.
On the other hand, it's a rather large code rewrite compared to what I have today, that arguably goes into the direction of more clunky code:
Instead of being able to use normal panicking constructs from the "base" type of my newtypes, I now need to use an uncommon idiom that is going to look weird to the uninitiated reader ("Why is there a resume_unwind() without a preceding catch_unwind()? Isn't this function about resuming some ongoing unwinding process?").
It ultimately means that tests are not exercising the actual production code path. I can certainly convince myself that the production code path is logically equivalent by manually reviewing the code, but that's tedious and arguably goes a bit against the spirit of automated testing.
For all these reasons, part of me thinks that an "expected panic scope", as I implemented in my current tests, is probably a cleaner design overall.
Still I'm going to think some more about this approach some more, and try it to see how good/bad it actually gets in practice. Meanwhile, I'm curious to hear more opinions on this topic, if there are any other ones.
Generally panics shouldn't be part of normal control flow, they represent exceptional cases. We really don't want to normalize the "use exceptions as longjmp" thing that some other runtimes do, it's terrible for performance.
Tests exercising panics are an unusual case (they're not production code), so I don't think this needs to be represented in std. The panic hook mechanism exists so you can do this kind of customization yourself, which you're already doing.
It's unclear to me why you think it should be in std, it sounds more like something that belongs in test harnesses, like libtest's #[should_panic] already does.
I understand the general sentiment that std should handle as little of this as possible, and offload as much of it as possible to third-party panic hooks from testing crates.
However, having written such a third-party panic hook myself, it is pretty clear that the customization points that are exposed by std::panic on stable are not good enough to do this in a clean way. My current approach works well enough for the internal needs of my crate, but it has so many rough edges that I would not want to publish it and advertise it as a general-purpose crate.
Here is why:
In general, when I install a new panic hook, I have no idea what other panic hook I am replacing. It may be the one from std, or it may be a custom one from another crate that does something arbitrarily complicated. In the interest of composability, I therefore think that a general-purpose panic hook (i.e. one that I would distribute as a dedicated crate and advertise to others) should always finish by calling the previously installed panic hook.
However, this hits a first roadblock on stable, which is that std::panic::(set|take)_hook is racy. Between the moment where I take the previous panic hook and the moment where I set mine, a new panic hook could have been installed by another library, which I will discard. There is no way to prevent this race on stable, though there is hope on the nightly horizon in the form of std::panic::update_hook.
This race is made worse by the fact that there is no way to hook into the sequential initialization procedure of libtest. From the perspective of user code, all #[test] functions immediately start running in parallel, which means that all panic hook initialization from testing libraries must be done lazily and will thus potentially occur in parallel.
On my side, I handle this problem by avoiding the use of other crates that touch the panic hook. But I would rather not put that sort of advice in the docs of a general-purpose crate...
Once the race is taken care of, a second problem with this composable "stacked hook" design is that there is no way for my panic hook to suppress only std backtraces without other side-effects. So far, I found two approaches that are both unappealing:
Do not run the lower-level panic hook when backtraces are suppressed. This is what I currently do. It is not great because it breaks panic hook composability again: if the user installed another panic hook down the stack that does something unrelated to printing backtraces (like e.g. counting panics), that panic hook will not work as expected for the panics where I disable backtraces, as it will be disabled along with the backtraces.
On nightly only, lock a global mutex, use std::panic::(set|get)_backtrace_style to enable/disable backtraces as appropriate, call into the lower-level panic hook, then restore the former backtrace configuration and unlock the global mutex. The use of a global mutex is not only problematic for performance (which is arguably not a concern in test code that exercises panics), it is also a correctness hazard as one must be careful not to cause deadlocks through recursive panics. But a global mutex is necessary due to the global nature of std::panic::(set|get)_backtrace_style, otherwise I could e.g. inhibit backtraces from other test threads that should not be inhibited.
In my opinion, it would be much cleaner to have some equivalent of std::panic::(set|get)_backtrace_style that operates in a thread-local fashion (at least as a thread-local override of the global setting). But the tracking issue discussions about this are not encouraging: although the concern has been brought up several times before, it seems that no one in the std dev team feels like this is worth the complexity cost. So I pleaded for it once more, but would not hold my breath...
There is no such thing as stacked hooks. If you want to have the default panic hook to run after your own panic hook is done, the way you did do this is by calling the default panic hook at the end of your own panic hook.
In general, when I install a new panic hook, I have no idea what other panic hook I am replacing.
If you control main you control how things get set up.
When I wrote "belongs in test harnesses" I meant a specific thing, as used by nextest for example.
Those provide their own test main and can thus be in charge of panic hooks.