EDIT: Proposal Updated - See Updates
Summary of updates here:
I'm leaving the rest of this post as it is for posterity, so please see the update.
TL;DR
Define a new auto-trait Leave, which means that the type can be implicitly destroyed when it goes out of scope. Users can declare must-destroy types by through impl !Leave for Foo {}, which are statically guaranteed to be destroyed before their lifetime's upper bound is reached.
This super-charges the RAII pattern, which opens up static, zero-cost programming patterns that rely on deterministic destruction, such as:
- Scoped tasks and threads, i.e. concurrency with static borrowing
- State machines that run to completion, notably async functions & expressions, alleviating the need for async drop
Prior art
- The Pain of Real Linear Types in Rust [Must read!]
- RFC: Scoped Threads, Take 2
- Idea: Leak auto trait and Drop call guarantees
History
Today, implementors must assume that any type can be leaked -- in other words, that a type can live after its upper lifetime bound. Here's my take on how we got here:
- Nobody thought really hard about leaking, so
mem::forgetwas unsafe, just in case. Rc,Arcand a few other std APIs had leaks, in safe Rust, leaving two choices:- Prevent some leaks with language features and std API changes.
- Simply allow leaks generally.
- As a response, leaks were formally allowed, and
mem::forgetbecame safe. - It was discovered that scoped threads could result in unsafe code by leaking a
JoinHandle.- (Note: Scoped threads are powerful because they can borrow locals from their logical parent thread.)
- As a response, scoped threads were removed from std.
- Scoped threads was re-implemented outside std (using an extra closure) in crossbeam etc.
- The approach is much more limiting - no join handle is available for the scope.
- Async was introduced to Rust.
- The async equivalent of scoped threads, scoped tasks, can't use the closure-strategy without blocking.
- Async code is forced to workaround by relying on
Arc - (Mostly unrelated): "Simply drop a future to cancel it" causes silent logical bugs - hard to reason about
Why now?
Leaks were relatively harmless as long as data was leaked. It just consumes memory - it doesn't kill your dog. However, this view broke down when behavior was leaked - which is represented by task- and join handles. As a result, we can't statically borrow locals from logical parent tasks or threads. We work around this problem with Arc, which has three problems: it's not zero-cost, it is noisy, and it can itself cause leaks and non-determinism around destruction. As multi-threaded executors - like Tokio - became a dominant force, this anti-pattern is growing.
It's not just concurrency - we see verbose, defensive patterns (pre-pooping your pants) in other places too, originally discovered in DrainRange. Again, async has exacerbated this problem: futures can always be arbitrarily dropped - this is worse than it sounds. Even an async fn foo(&mut self) function can be silently interrupted - which can cause inconsistent state. While "pre-pooping" works in theory, it is verbose, easy to get wrong, and as a result -- largely unused.
Why an auto-trait?
An annotation wouldn't work, as it wouldn't be transitive. Also, this feature is on the hook for safety, so it must be integrated into the type system - as far as I can tell.
"But it's impossible to prevent leaks!"
It's true that we can't prove the absence of memory leaks; scopes may never exit, reference cycles, etc. However, there are many cases where it's possible to prove that a value is destroyed before some other piece of code runs. That other piece of code can be written without having to defensively assume that the first piece of code (the destructor) may not have run. As such, this proposal is about control flow, not leaks.
Proposal
Define a new auto-trait Leave, which means that the type can be implicitly destroyed by the compiler.
- All types are
Leaveby default. Leaveis a super-trait ofDrop- A
!Leavetype is called a must-destroy type. Opt-in usingimpl !Leave for Foo {} - A type which owns a
!Leavetype is also!Leave, transitively - Must-destroy types must be destroyed before going out of scope, at every exit point (return, break,
?, etc) - Destroying a must-destroy type is done through destructuring, generally through
fn foo(self) - Generic code, e.g. containers, can opt-in to support must-destroy types using
T: ?Leave.Arc,Rcandmem::forgetdo not support?Leave- Vectors, maps, combinators can support
?Leave, but not all methods (e.g.clear)
- References cannot be destroyed, so they are always
Leave - Panicking with a live must-destroy type causes abort
Implementation cost
Whenever the compiler inserts drop automatically today, it would throw an error if the type is !Leave. When unwinding during a panic, a live !Leave type causes an immediate abort.
Since Leave is implemented by default, this is a backwards compatible change. All existing drop-semantics etc stay the same.
That said, an auto-trait is a huge deal. Libraries, including std, futures, would need to add ?Leave support to several collections, combinators, etc. This is a lot of work, which I believe to be worthwhile. Support can be rolled out incrementally and phased out behind feature flags.
Mental model
A must-destroy type can be thought of as a shift of destruction responsibility: our existing (Leave) types can be explicitly or implicitly destroyed and - worst case - not destroyed at all. A must-destroy type, on the other hand, is always explicitly destroyed. Users and library authors should implement !Leave when deterministic destruction is required for safety and/or fundamental invariants.
Examples
Here's a basic must-destroy type:
// Resource could be a handle of sorts, e.g. to a kernel resource
struct Foo(Resource);
// Declare non-destroy. (Ergonomic derive-macro could be added?)
impl !Leave for Foo {}
impl Foo {
// A simple destructor.
fn destroy(self) {
let Self(res) = self;
// Ensure some invariant on the underlying resource
}
// Custom destructor, with bells and whistles!
fn fancy_destroy(self, b: bool) -> u64 { .. }
// Async destructor
async fn async_destroy(self) { .. }
}
Basic control flow:
{
let foo = Foo::new();
// Error: foo must be manually destroyed
let foo2 = Foo::new();
foo2 = Foo::new(); // Error: Re-assignment without destroying old value
}
{
let mut foo = Foo::new();
let _ = foo.fancy_destroy(true); // OK
}
{
// Vec with must-destroy support
let v = vec![Foo::new(), Foo::new()];
let f = v.pop(); // OK: Moved element out of vec
v[0] = Foo::new(); // Error: overwriting without destroying existing value
for e in v { // OK: All values are returned from the iterator
e.destroy(); // OK
if roll_dice() { break } // Error: breaking leaves elements in the iterator
}
}
Some functions:
fn bar() {
let mut foo = Foo::new();
if roll_dice() {
foo.fancy_destroy(true);
return; // OK: foo destroyed in exit point 1
}
foo.destroy(); // OK: foo destroyed in exit point 2
}
fn baz() -> Result<()> {
let mut foo = Foo;
can_fail()?; // Error: Exit point without destroy
foo.destroy();
}
We can bring back scoped threads:
fn bar() {
let v = String::new("world");
// The handle is must-destroy
let handle = thread::scoped(|| {
println!("hello, {}"); // Borrow from outer scope
});
handle.join();
}
If executors and combinators add ?Leave support, we can do neat things in async:
// This future uses a must-destroy type, so it can't be arbitrarily dropped
async fn bar() {
let foo = Foo::new();
let a = 32;
foo.async_destroy().await; // Async destruction
}
struct Bar(bool, bool);
impl !Leave for Bar {}
impl Bar {
// Future becomes must-destroy, since self is must-destroy
async fn bar(&mut self) {
self.0 = true;
sleep(1).await; // Can't be interrupted here
self.1 = true;
}
fn destroy(self) { .. }
}
// Scoped tasks
async fn baz() {
let v = String::new("world");
let task = Task::scoped(|| async {
println!("hello, {}"); // Borrow from outer scope
});
task.join().await;
}
Issues
- Multiple exit points, from
?,break,matchand other branchings, makes it unergonomic to use must-destroy types. This can be worked around using a wrapper closure. A defer-statement could help with this:let mut foo = Foo::new(); defer foo.destroy(); // Runs before every exit point can_fail()?; // OK
Unresolved questions
- Is the
Leavetrait declaredunsafe(even if negative impls are safe to add)? - Can must-destroy types recover from a panic? If so, how?
- Unsafe escape hatch to force-destroy a must-destroy type?
Variations
- "
ImplicitDrop" (does not super-traitDrop):!ImplicitDroptypes can implement drop, and it's invoked during unwinding, but never implicitly.- Pro: Customizable - opportunity for panic recovery.
- Con: Overloading -
?ImplicitDropcontainers can't specialize their!ImplicitDropimpl.
- "
Leak" (does not super-traitDrop):!Leakcan implement drop - implicit drops are OK, but cannot be ref-counted, forgotten, etc.- Pro: Least invasive.
- Con: Prevents the "async destruction" use-case (see above).
Feedback requested
I know this is a large change. To me, it appears worthwhile, but I don't expect everyone to agree. At the moment, I'm focusing less on the "is this truly worth it?" and more on the "let's make the best possible proposal". (Don't worry if you're not fond of this direction, there will be plenty of time to argue against the proposal when it's polished). In particular, I'm looking for:
- A more flexible panic/unwind story than abort
- Holes, mistakes or missing considerations & drawbacks in this proposal.
- Alternative approaches for the problems listed above.
