Thanks for your interest. I've written a draft regarding what expired references are and how they are useful. I'll update the document and add more details as new discoveries are made.
The name was originally "Out-of-lifetime references", but after some thinking I find the term "expired references" somehow more intuitive. If someone has a better name, feel free to leave suggestions.
Expired references
Motivation
The idea originally came up in my attempt to design self-referential structs. Self-referential structs turned out to be inevitably complex, but expired references remain an interesting topic to explore.
The intent is not just to enable specific use cases; it is expected to make the logic around lifetimes explicit, and simplify the implementation of the compiler and remove corner cases like Sound Generic Drop and Member constraints.
Guide-level explanation
Currently, Rust ensures that all usage of a reference is within its lifetime. It does so by checking a function as a whole, and inferring every reference's lifetime based on its occurrences.
let x = 0usize; // 1
let y = &mut x; // 2
*y = 1; // 3
let z = &mut x; // 4 at this point, y can no longer be used
*z = 2; // 5
*y = 1; // 6 Rust's borrow checker will complain if this line is included
From the above code it concludes that:
- The lifetime of reference y begin before 2 and end after 6
- The lifetime of reference z begin at 4 and end at 5
- The lifetime of reference y and z shall not coincide because they are mutable borrows from the same variable, x
It thus concludes that the code is invalid and rejects the code.
References can also appear nested, such as in vectors or in a struct field. This is commonly used to create iterators, for example: (elided lifetimes added for clarity)
struct Iter<'a, T> {
arr: &'a [T],
index: usize,
}
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next<'b>(&'b mut self) -> Option<&'a T> {
if self.index == self.arr.len() {
None
} else {
let item = Some(&'a self.arr[self.index]);
self.index += 1;
item
}
}
}
fn make_iter<'a, T>(arr: &'a [T]) -> Iter<'a, T> {
Iter { arr, index: 0 }
}
fn main() {
let arr = vec!["this", "is", "a", "sentence"];
let iter = make_iter(&arr);
for word in iter {
println!("{}", word);
}
}
A reference to arr is used to create an iterator, and that reference is used in the next function of the iterator to generate a reference for the next function.
How does the compiler ensure that the reference is valid in the next function? It does so by propagating reference lifetimes. While creating the iterator with the make_iter method, the method accepts a lifetime parameter called 'a and creates a type called Iter<'a, T>. Since Iter<'a, T> contains a reference &'a T, the compiler propagates that lifetime to Iter<'a, T> and ensures that any usage of Iter<'a, T> can only appear within its lifetime 'a. That is, for any reference &'b mut Iter<'a, T> or &'b Iter<'a, T>, the type checker enforces that 'a: 'b.
As the borrow checker checks main function, while passing make_iter function, it is able to infer from the function signature make_iter<'a, T>(arr: &'a [T]) -> Iter<'a, T> that:
-
The lifetime 'a refers to the lifetime of arr.
-
The 'a in the returned value Iter<'a, T> is the same as the lifetime of arr.
By doing lifetime propagation, the compiler ensures that Iter<'a, T> can only be used during 'a, that is, while arr exists. Hence, if we drop arr before the for loop:
fn main() {
let arr = vec!["this", "is", "a", "sentence"];
let iter = make_iter(&arr);
drop(arr);
for word in iter {
println!("{}", word);
}
}
The borrow checker complains, because the later for loop uses iter which cannot exist when arr no longer exists. This prevents a potential memory safety bug in which we access freed memory through iter.
The above is the way Rust handles lifetimes since its very beginning. It works well to prevent undefined behavior, and it has proven to be rigorous. However, it suffers from some limitations.
Consider an accumulator, which pulls numbers from a &Cell<u64> and sums them up, and outputs the sum without &Cell<u64> being available:
use std::cell::Cell;
struct State {
val: Cell<u64>,
}
impl State {
fn new() -> State {
State { val: Cell::new(0) }
}
fn observe(&self) -> u64 {
self.val.get()
}
fn set_val(&self, v: u64) {
self.val.set(v)
}
fn shutdown(self) {
}
}
struct Accumulator {
ptr: *const State,
sum: u64,
}
impl Accumulator {
fn new(ptr: &State) -> Accumulator {
Accumulator { ptr, sum: 0 }
}
unsafe fn add(&mut self) {
let state = unsafe { &*self.ptr };
self.sum += state.observe();
}
fn get_sum(&self) -> u64 {
self.sum
}
}
fn main() {
let mut state = State::new();
let mut acc = Accumulator::new(&state);
state.set_val(3); acc.add();
state.set_val(4); acc.add();
state.set_val(5); acc.add();
state.shutdown();
println!("{}", acc.get_sum()); // 12
}
Note how this can only be implemented in unsafe. If we try to implement this in safe code, Accumulator will need to contain a reference to state , like: (main and State code are exactly the same and omitted)
struct Accumulator<'a> {
ptr: &'a State,
sum: u64,
}
impl<'a> Accumulator<'a> {
fn new(ptr: &'a State) -> Accumulator<'a> {
Accumulator { ptr, sum: 0 }
}
fn add(&mut self) {
self.sum += self.ptr.observe();
}
fn get_sum(&mut self) -> u64 {
self.sum
}
}
If we try to compile this, we will get this error:
error[E0505]: cannot move out of `state` because it is borrowed
--> c.rs:47:5
|
43 | let mut acc = Accumulator::new(&state);
| ------ borrow of `state` occurs here
...
47 | state.shutdown();
| ^^^^^ move out of `state` occurs here
48 | println!("{}", acc.get_sum());
| --- borrow later used here
Our code did not pass the borrow checker! The code should be safe because, even though state is dropped at line 47, acc.get_sum() does not use state in its code. In fact, the unsafe variant complies with Stacked Borrows and passes Miri tests, so as we'll explain later, most usual compiler optimizations still apply when calling get_sum.
The problem is that the compiler cannot understand that the get_sum function does not use the ptr reference. Instead, it only understands "lifetime propagation" as we've explained above. The Accumulator<'a> struct inherits the lifetime of all its fields, one of which is 'a, and thus the lifetime of Accumulator<'a> is 'a. This ensures that as long as the object Accumulator<'a> exists, all its fields can be accessed, so in our code:
fn add(&mut self) {
self.sum += self.ptr.observe();
}
self.ptr is guaranteed to exist, since &mut self exists. However, in the following code:
fn get_sum(&mut self) -> u64 {
self.sum
}
self.ptr is guaranteed to exist as well, even though we did not use it. This means that after state.shutdown(), state no longer exists, and we cannot access self.sum because the other field self.ptr is holding us back! What if we want to tell the compiler, as well as users of the function, that get_sum works outside the lifetime 'a? That's when expired references come to play.
How expired references work
Expired references, or out-of-lifetime references, are references that cannot be used. This might seem weird, since references that cannot be used does not have any value, but it tries to prevent a single reference in a struct field from constraining the whole struct and making the other parts of a struct unusable outside a single reference's lifetime.
We achieve this by introducing two new special lifetime identifiers, called 'self and 'none, in addition to 'static. 'none is the antonym of 'static. 'static means to live forever, while 'none is used for references that is not alive at all. 'self has a context-specific meaning and is different everywhere. In this document, we give it a meaning in the context of functions, that the references lives for the duration of the function call.
We use the same syntax as in current Rust to denote lifetime subtyping. 'a: 'b has the following equivalent meanings:
-
'a is assignable to 'b: let x: &'b T = &'a T
-
'a is longer than 'b. It can be understood as 'a >= 'b
And thus:
-
'static is longer than any lifetime. For any 'a, 'static: 'a
-
'none is shorter than any lifetime. For any 'a, 'a: 'none
For the purposes of explanation, we'll force all lifetimes to be explicitly written. That is, if a lifetime argument is not constrained, it will default to 'none, and thus any references with that lifetime will not be usable. That is, the following function does not work:
fn work<'a>'(&'a mut self) {
// 'a could be 'none and thus the reference may not be valid at all!
}
Instead, it has to be written this way:
fn work<'a: 'self>(&'a mut self) {
// 'a couldn't be 'none because of 'a: 'self.
}
This is not the proposed syntax, since it breaks nearly all existing code. Instead, we will later explain how lifetime elision rules are modified to ensure that expired references are an opt-in feature and that existing code is not affected.
In order for the above example code to work, we need to break a rule that has been fundamental to the Rust compiler: that the existence of a reference implies the validity of the reference. The impact of this is a bit hard to understand, but for now, think like this: if inserting a usage of reference a at some point of code causes the code to fail the borrow checker, then the reference is considered to be "expired", and treat it as if it were a raw pointer.
First of all, we need to somehow express, at the function signature, the fact that the function get_sum does not use the reference &'a State. This has different meanings for both sides of the function:
- For the function caller, in order to call
acc.get_sum(), we do not need to be within 'a . The borrow checker needs to understand that the call to acc.get_sum() does not cause the lifetime 'a to be extended till this point.
- For the function body,
get_sum() must not dereference self.ptr, not even a phantom dereference. self.ptr may point to invalid memory at this point, and it may contain invalid data or point to an invalid page, so any attempt to dereference it can cause undefined behavior. Instead, it must be treated like a raw pointer or opaque data.
We express the fact using the 'none and 'self lifetime specifier as we've just defined. The Accumulator is implemented in this way:
struct Accumulator<'a> {
ptr: &'a State,
sum: u64,
}
impl<'a> Accumulator<'a> {
// note that the 'a: 'self bound is not necessary here
fn new(ptr: &'a State) -> Accumulator<'a> {
Accumulator { ptr, sum: 0 }
}
fn add(&'self mut self) where 'a: 'self {
self.sum += self.ptr.observe();
}
fn get_sum(&'self self) -> u64 {
self.sum
}
}
We'll explain the function signatures add and get_sum with the new syntax.
-
fn add(&'self mut self) where 'a: 'self
This function borrows Accumulator for exactly the duration of the function body, and hence the input argument is &'self mut self. However, in our model. it is no longer safe to dereference self.ptr directly, because even though self is valid within the function body, it is no longer guaranteed that self.ptr is valid at all (imagine self.ptr being *const State at this point)! Instead, we have to explicitly specify in the function signature that 'a is indeed valid during 'self. Hence 'a: 'self.
-
fn get_sum(&'self self) -> u64
This function borrows Accumulator for exactly the duration of the function body, and hence the input argument is &'self mut self. Since sum is not a reference, we can access sum directly with a reference to Accumulator, and the only constraint we need is that the reference must be valid during function execution. This is explicitly written out by the 'self argument.
Relationship with the current lifetime model
As we said in the first section, currently Rust expects a struct/slice/array/vector to capture the lifetime of all its references. That is, if struct S contains a field &'a T , either directly or indirectly through another struct/array/vector or through PhantomData, then the whole struct becomes 'a: S. This means that when attempting to create a reference &'b S or &'b mut S to S, the type checker adds a constraint 'a: 'b for every field with lifetime 'a. Note that this is exactly the constraint that the proposal is going to remove.
Rust also enforces that all generic parameters must be used within the struct body, so the set of lifetime parameters is the same as the set of all lifetimes.
That is, for the following struct:
struct Example<'a, 'b, 'c> {
a: &'a T,
b: &'b mut T,
c: PhantomData<&'c mut T>,
}
impl<'a, 'b, 'c> Example<'a, 'b, 'c> {
fn work<'d>(&mut self, val: &'d mut usize) {}
}
The signature of fn work is equivalent to this in our new model:
fn work<'d>(&'self mut self, val: &'d mut usize) where 'd: 'self, 'a: 'self, 'b: 'self, 'c: 'self;
Basically, all lifetime parameters that appear:
- in a reference in one of the struct fields, enum variants, tuple fields, slice items, PhantomData reachable from a function argument
currently outlive 'self implicitly.
Impact on trait lifetimes
This proposal is likely to result in a change to how traits work. Currently, the lifetime parameter of &self or &mut self in every trait automatically includes the lifetime of all references reachable from self, even if the lifetime is not specified as a lifetime parameter in the trait. This is an implicit lifetime parameter that is actually used in the trait but not expressed well. For impl Trait in function return values, this has resulted in Member constraints. The lifetime for dyn Trait has also been a special case; it defaults to 'static and is specified using + 'lifetime syntax, as shown in the following example:
struct Ref<'a>(&'a usize);
trait Trait{}
impl<'a> Trait for Ref<'a> {}
fn f<'a>(val: Box<dyn Trait + 'a>) {
}
fn g(val: Box<dyn Trait>) {
}
fn main() {
let num = 0;
let data = Ref(&num);
let r = Box::new(data);
f(r);
// In this case, the lifetime parameter for `dyn` object defaults to `'static`, so it doesn't work.
g(r);
}
With this proposal implemented, such a lifetime would have to be explicitly specified in a trait's parameters rather than implicitly capture all the fields accessible from self.
For example:
use std::marker::PhantomData;
struct Iter<'a, T> {
slice: *const T,
remaining: usize,
_data: PhantomData<&'a T>,
}
fn make_iter<'a, T>(slice: &'a [T]) -> Iter<'a, T> {
Iter { slice: &slice[0], remaining: slice.len(), _data: PhantomData }
}
impl<'a, T> Iterator<'a> for Iter<'a, T> {
type Item = &'a T;
// There is a problem with traits here. With the proposal implemented, the
// Iterator trait's signature is:
// fn next(&'self mut self) -> Option<Self::Item>
// Before the implementation of the proposal, the lifetime 'x of the reference
// &'x mut self implicitly captures all struct fields, that is, there is an implicit
// lifetime constraint 'a: 'x if the struct contains references of lifetime 'a.
// However, with the proposal implemented, the lifetime 'self does not include
// 'a by default, so we cannot dereference `slice` from the function in safe
// code, and we cannot ensure that a reference to `slice` is valid in unsafe code.
// Instead, we need to write the signature as:
// fn next(&'self mut self) -> Option<Self::Item> where 'a: 'self.
// This means that we need to specify `'a` as a lifetime parameter of the trait!
// This is not extra complexity caused by expired references, but rather
// a sign that Rust lacks explicitness in the trait definition. The trait only works
// within a certain lifetime, but currently, this "certain lifetime" is derived from
// the concrete type rather than explicitly expressed on the trait signature.
fn next(&'self mut self) -> Option<Self::Item> where 'a: 'self {
if self.remaining == 0 {
None
} else {
// To properly dereference it, we need to ensure two things:
// * The reference is within its lifetime. This is enforced by 'a: 'self which is implicit.
// * We give the reference a correct lifetime so it will be usable during that lifetime.
// Since the iterator was constructed from &'a [T], we know that the &T reference
// will be usable during 'a.
let item = unsafe { &(*self.slice) as &'a T };
self.remaining -= 1;
Some(item)
}
}
}
In order to solve the above problem, we need to add an implicit lifetime to all traits that potentially contain a lifetime for self references. Currently, all traits have a hidden generic parameter called Self as the first parameter. We add one more hidden lifetime parameter in addition to that, called 'ref. In a trait's signature, for every function that takes a reference to self, such as &'a self and &'a mut self, the lifetime bound 'ref: 'a is automatically added (in addition to 'a: 'self). While implementing traits, the 'ref parameter will automatically catch all lifetime parameters of the current type, unless the user explicitly overrides it. Whether and how that lifetime parameter should be exposed to the user is left as a question.
// Old way of writing traits:
trait Iterator {
type Item;
fn next(&mut self) -> Option<Self::Item>;
}
// The above desugars to this:
trait Iterator<'ref> {
type Item;
fn next(&'self mut self) -> Option<Self::Item> where 'ref: 'self;
}
Avoid special cases like Sound Generic Drop
With a new lifetime parameter for all traits, such a design gets the logic clearer and shows a clearer way to understand corner cases like Drop Check!
Consider the following program in the book:
struct Inspector<'a>(&'a u8);
impl<'a> Drop for Inspector<'a> {
fn drop(&mut self) {
println!("I was only {} days from retirement!", self.0);
}
}
struct World<'a> {
inspector: Option<Inspector<'a>>,
days: Box<u8>,
}
fn main() {
let mut world = World {
inspector: None,
days: Box::new(1),
};
world.inspector = Some(Inspector(&world.days));
// Let's say `days` happens to get dropped first.
// Then when Inspector is dropped, it will try to read free'd memory!
}
The example is a case where a type implementing Drop differes from one not implementing it! In order to solve the problem, the language introduced a concept called Sound Generic Drop. The rule is as follows from the link:
For a generic type to soundly implement drop, its generics arguments must strictly outlive it.
However, the concept strictly outlive is very confusing, since up to now, all lifetime notations 'a: 'b allows equality (which is analogous to 'a >= 'b), and there is no special symbol to say 'a != 'b. Why exactly do we need to introduce such a concept, and how an item implementing Drop is breaking the safety guarantees and therefore cannot be implemented safely?
The reason becomes clear if we explicitly write out the call to drop:
fn main() {
let mut world = World {
inspector: None,
days: Box::new(1),
};
world.inspector = Some(Inspector(&world.days));
World::drop(&mut world); // doesn't pass the borrow checker!
}
The main function first immutably borrows world.days , and then writes it into world.inspector . However, at this point, it is no longer possible to take a mutable reference to world, since world.days is mutably borrowed by something else: the world struct is borrowed by itself. Thus, it cannot be mutably borrowed until the borrow to itself is released, which is impossible because we don't have a mechanism to tell the compiler that we want to forget a specific reference on a struct field!
However, with the proposal implemented, it makes sense for the Drop trait to take a lifetime parameter because it is no longer assumed 'a: 'self in a function that takes &mut self . We no longer need a special dropck, because the distinction between a type without a drop handler and a type with a drop handler is clearer; the former is as if Drop<'none> while the latter is Drop<'b> where 'b: <catches all lifetime parameters> .
struct Inspector<'a>(&'a u8);
trait Drop<'a> {
fn drop(&'b mut self) where 'b: 'self, 'b: 'a;
}
impl<'a> Drop<'none> for Inspector<'a> {
fn drop(&mut self) { // fn drop(&'b mut self) where 'b: 'self
// Now we are able to tell Drop that we do not need access to self.0.
// This means that we are able to implement a drop handler!
println!("I cannot see anything!");
}
}
struct World<'a> {
inspector: Option<Inspector<'a>>,
days: Box<u8>,
}
fn main() {
let mut world = World {
inspector: None,
days: Box::new(1),
};
'a: {
let days = &'a world.days;
let inspector = &'a mut world.inspector;
*inspector = Some(Inspector(days));
}
// In order to call `world`'s drop handler, we need a mutable reference to `world`.
// This means that the lifetime of both `days` and `inspector` has to end here.
// Therefore, `world`'s drop handler must be Drop<'none>, and so is `Inspector`'s
// drop handler.
call_drop(world);
}
Opt-in mechanism and migration notice
Naively implementing it will break existing code, and worse yet, unsafe code that relies on the assumption that their code can only be called in the lifetime of their struct fields, such as PhantomData, because such code will not cause compiler errors and silently break the program. We need a way to opt-in to this feature and also force authors of unsafe code to make lifetime parameters explicit.
In order to avoid breaking existing code, we apply the following rules by default:
- All types continue to inherit lifetime from all its fields, except when the field or the entire type was marked with
#[maybe_expire], in which case its lifetime is ignored while computing the lifetime of the whole type. In particular, if the entire type was marked with #[maybe_expire], its lifetime becomes 'static.
- For functions, the constraint
T: 'self is applied to every argument T that was not marked #[maybe_expire]. The mark may also be applied at the function level, impl level, module level or cargo level.
For example, if in most cases our struct need to access self.a: &'a T and self.b: &'b mut T but not self.c:
struct Example<'a, 'b, 'c> {
a: &'a T,
b: &'b mut T,
#[maybe_expire]
c: PhantomData<&'c mut T>,
}
impl<'a, 'b, 'c> Example<'a, 'b, 'c> {
fn work(&'self mut self) {
// cannot use self.c but can use self.a and self.b
}
fn work_full(&'self mut self) where 'c: 'self {
// we explicitly want to use self.c
}
}
Note its interaction with struct lifetimes and hidden lifetime parameters in traits:
fn has_lifetime<'a, T>() where T: 'a {}
has_lifetime::<Example<'a, 'b, 'c>, 'a>() // doesn't compile
has_lifetime::<Example<'a, 'b, 'c>, 'c>() // doesn't compile
fn bar<'a, 'b, 'c, 'x> where 'a: 'x, 'b: 'x { has_lifetime::<Example<'a, 'b, 'c>, 'x>() } // compiles
trait Foo {
fn work(&mut self);
}
// The implicit lifetime parameter is: 'x where 'a: 'x, 'b: 'x
impl<'a, 'b, 'c> Foo for Example<'a, 'b, 'c> {
fn work(&'self mut self) {
// cannot use self.c but can use self.a and self.b
}
}
// We need to explicitly specify the lifetime parameter if
// we want to use a reference in a trait
impl<'a, 'b, 'c> Foo<'ref> for Example<'a, 'b, 'c> where 'c: 'ref {
fn work(&'self mut self) { // implicit 'ref: 'self
// can use self.c!
}
}
In order to notify library authors of unsafe code to make lifetimes explicit in function signature, we can emit compiler warnings for PhantomData fields that are neither #[maybe_expired] nor 'static, and tell them how to apply #[maybe_expired] and make lifetimes explicit.
Lifetime parameters unused in the struct field should no longer cause hard errors. Instead, the compiler should emit warnings if these parameters were not used in the function signatures.
Compiler optimizations and compliance with Stacked Borrows
Note that all kinds of optimizations still work. For example, a compiler might store &u64 as u64 directly. However, it must not read memory at &'none u64 because it may point to invalid memory or invalid pages, causing segfaults. Apart from preventing dereference operations with &'none u64 , a compiler implementation must also take care not to perform out-of-thin-air reads with such a reference and must ignore it completely. Since lifetime constraints are statically known, the compiler should be able to figure out whether such reads are allowed. A way to avoid this is to treat them as a raw pointer, and any operations that are safe with raw pointers are also safe with 'none references. For example, &'none u64 can implement the Copy trait.
Its compliance with Stacked Borrows is trivial. Stacked Borrows is strictly more permissive than current Rust, and since Stacked Borrows only track reference usages, by preventing usage of 'none references, the reference usages are equivalent to current Rust. Thus, Stacked Borrows should allow 'none references without complaining.
Outline of implementation
Start by reserving the special lifetime identifiers 'self, 'none and 'ref, and implement the mark #[maybe_expired] to say that the references in a field or a struct may be expired and therefore not accessible. Add the constraint T: 'self for every function argument that was not marked #[maybe_expired]. Change the compiler to account for potentially expired references, check the lifetime parameter for every dereference operation, tell LLVM to ensure that unwanted optimizations doesn't happen. Look for any unsoundness caused by this change, such as where the compiler assumes that any reference in a struct field is accessible.
Then add a new hidden lifetime parameter to every trait. When implementing a trait, automatically add the lifetime parameter 'a and add constraint 'a: T for every struct field of type T that is not marked with #[maybe_expired]. Find a way to expose that parameter for both trait definitions and implementations. Somehow handle the case of dyn and impl Trait.
Finally, try to encourage use of #[maybe_expired] and make lifetime parameters explicit and deprecate struct-level lifetimes. We can remove the special case for Sound Generic Drop and check it with the borrow checker directly. Remove Member constraints.