Well hello there!
I'm soliciting for final feedback on the third draft of a Reborrow / "autoreborrow traits" RFC that I've been drawing up over the previous months. The RFC is up in my repository and the most recent version is included below. If you have the time to read it, I'd welcome any and all comments you have!
- Feature Name: autoreborrow-traits
- Start Date: 2025-04-15
- RFC PR: rust-lang/rfcs#0000)
- Rust Issue: rust-lang/rust#0000)
Summary
Enable users to opt into automatic reborrowing of custom types with exclusive reference semantics.
Motivation
Reborrowing of exclusive references is an important feature in Rust, enabling much of the borrow checker's work. This important feature is not available for reference wrappers or user-defined reference-like types, which hampers the usefulness of the language when working with these kinds of types. Manual reborrow-like functions can be implemented but they must be explicitly called making them verbose, and are importantly always limited to creating reference-like types bound to the local function that cannot be returned to the caller.
Examples of features unlocked by this RFC are:
- Automatic reborrowing of
Option<&mut T>
andPin<&mut T>
. - Automatic reborrowing of custom exclusive reference types like
Mut<'_, T>
. - Returning of values deriving from reborrowed parameters, extending to the source lifetime.
A reborrow is a bitwise copy of a type with extra lifetime analysis performed by the compiler. The lifetime of the reborrow result is a subset of the source value's lifetime (not a strict subset, the result's lifetime is allowed to be equal to the source's lifetime). When reborrowing as exclusive, the source value is disabled for the lifetime of the result. When reborrowing as shared, the source value is disabled for mutations for the lifetime of the result.
Status quo - reference reborrowing
This section gives basic Rust reference usage examples with reborrowing. This is to serve as a comparison for later examples of autoreborrow traits usage. The following kind of methods are assumed to exist on all reference-like types in these examples.
impl<T, U> T {
/// Requires exclusive access to self and returns some derived exclusive reference.
fn mutate(self) -> U;
/// Requires shared access to self and returns some derived shared reference.
fn observe(self) -> U;
}
Note that the methods are written as taking self
by value; because we're
talking about reference-like types, this is intended to be read as
fn mutate(&mut self)
and fn observe(&self)
for normal references and eg.
fn mutate(self: MyMut)
and fn observe(self: MyRef)
for custom reference
types.
The following examples of reborrowing will be used later:
fn mixed_reborrowing(source: &mut T) {
// Exclusive reborrow disables source.
{
let result: &mut U = source.mutate();
// The following line would not compile because source is disabled for
// the duration of the result's lifetime:
// source.observe();
result.observe();
}
// Note: source can be reborrowed as shared or exclusive after the
// previous exclusive reborrow is dropped.
// Shared reborrow disables source for mutations. Source can be reborrowed
// as shared multiple times.
{
let result1: &U = source.observe();
let result2: &U = source.observe();
// The following line would not compile because source is disabled
// for mutations for the duration of the results' lifetime:
// source.mutate();
// Note: both shared reborrow results can be kept and used
// simultaneously.
result1.observe();
result2.observe();
}
// Note: after all shared reborrow results are dropped, source can be
// reborrowed as exclusive again.
source.mutate();
}
fn returning_derived_mut(source: &mut T) -> &mut U {
// Note: the reborrow that happens when calling mutate() extends its
// lifetime to the lifetime of source, not to some local value conceptually
// create here:
// let reborrow = &mut *source;
// reborrow.mutate()
source.mutate()
}
fn returning_derived_ref(source: &mut T) -> &U {
// Note: exclusive reference is reborrowed as shared for the return value.
source.mutate()
}
// Some function returning a result where the both variants refer to the
// source data; think eg. a parser that wants to return a structure containing
// references to source text slice or an error pointing to invalid source text.
fn result_function(source: &T) -> Result<&X, &Y>;
// A function taking an exclusive reference, reborrowing it as shared to call
// other functions that return derived references, and then conditionally
// returning some results of those calls.
fn branching_function(source: &mut T) -> Result<&U, &Y> {
// Note: the early Err return requires the Err value's lifetime to extend
// to the source lifetime, ie. to the end of the call. The mutate call
// later requires the Ok value's lifetime to end before that call. The
// compiler is currently not smart enough to split Ok and Err from one
// another, meaning that this does not work in current stable Rust.
// However, this does compile on Polonius!
let result = result_function(source)?;
println!("Example: {:?}", result);
source.mutate()
}
The important things to note here are:
- Reborrowing is done on exclusive references only; shared references are Copy and need no reborrowing semantics.
- Reborrowing has two flavours, shared and exclusive.
- Reborrowing happens implicitly at each coercion-site.
- If the function being called returns a value derived from the reborrowed reference, the returned value's lifetime extends in the caller's scope to match the original reference's lifetime.
- Branching function returns run into issues with lifetime extension that needs Polonius to resolve.
Wrapper type reborrowing
Wrapper types like Option<T>
cannot be reborrowed today. This makes using
optional exclusive references tedious as they are automatically moved when used.
Reborrowing of Pin<&mut T>
has been provisionally added into the nightly
compiler, but it is done via custom handling and relies on internal knowledge of
the type.
We'll revisit the earlier examples, now with Option<&mut T>
as our example of
a wrapper type with exclusive reference semantics. Note that this means that the
function signatures for our helper functions are now
fn mutate(self: Option<&mut T>) -> Option<&mut U>
and
fn observe(self: Option<&T>) -> Option<&U>
.
fn mixed_reborrowing(source: Option<&mut T>) {
// Exclusive reborrowing
{
// Note: In current Rust this would work, but after this the source has
// been moved out of this function and could no longer be used. This
// call site must change from using a move to using a reborrow.
let result: Option<&mut U> = source.mutate();
// Again, we want the following line to fail to compile, and it does
// but currently for the wrong reasons:
// source.mutate();
result.observe();
}
// Shared reborrowing
{
// We want to enable the following calls to still work: this means that
// the first mutate() call must reborrow source instead of moving it.
// The following calls must also implicitly reborrow source as shared.
let result1: Option<&U> = source.observe();
let result2: Option<&U> = source.observe();
result1.observe();
result2.observe();
}
// After all shared reborrow results are dropped, source must be
// reborrowable as exclusive again.
source.mutate();
}
fn returning_derived_mut(source: Option<&mut T>) -> Option<&mut U> {
// This does work today, but mutate() must internally map() the Option
// instead of eg. using as_deref_mut() to compile. See below.
source.mutate()
}
fn returning_derived_ref(source: Option<&mut T>) -> Option<&U> {
// Note: This does not work today, and suggests using as_deref() which also
// doesn't work, citing "returns a value referencing data owned by the
// current function". Using eg. Option::map() to inject a coercion site
// fixes the issue:
// source.map(|v| -> &T { v })
source.mutate()
}
// Some function returning a result where the both variants refer to the
// source data; think eg. a parser that wants to return a structure containing
// references to source text slice or an error pointing to invalid source text.
fn result_function(source: Option<&T>) -> Result<&X, &Y>;
// A function taking an exclusive reference, reborrowing it as shared to call
// other functions that return derived references, and then conditionally
// returning some results of those calls.
fn branching_function(source: Option<&mut T>) -> Result<&U, &Y> {
// Note: this would again move source and hence does not work.
// let result = result_function(source)?;
// What about manually reborrowing? No, this does not work either!
// The Option<&T> now has a lifetime bound on the temporary as_ref()
// borrow and cannot be returned from this function (via the Err variant)
// as it "returns a value referencing data owned by the current function".
let result = result_function(source.as_ref().map(|v| -> &T { v }))?;
// Do something with the result.
println!("Example: {:?}", result);
// Then mutate the source again.
Ok(source.mutate())
}
The important thing to note here is that while manual unwrapping and rewrapping
of an Option does help through many issues, it too fails in the branching
function returns case, though it gives a different error: the error is now
related to the lack of true reborrowing as opposed to the compiler not knowing
how to split Ok and Err branch lifetimes from one another. Even Polonius does
not help in this case, as the source.as_ref()
call has made the result
lifetime a strict subset of the source lifetime: even though we can see that the
transformation from Option<&mut T>
to Option<&T>
does not retain any
references to local values, the compiler cannot be quite so sure and thus does
not allow this code to compile.
Custom reference type reborrowing
Custom reference types are as many as there are users, but a common theme
amongst them is that they usually hold one (though sometimes multiple or none)
data field and a PhantomData field (or multiple) for holding a lifetime in. The
previous examples are once again revisited with custom MyMut
and MyRef
reference types. The function signatures of our helper functions are now
fn mutate<'a>(self: MyMut<'a>) -> MyMut
and
fn observe<'a>(self: MyRef<'a>) -> MyRef
.
In the examples, "FIX" and "NOT" comments are given to show how some of these
examples can be made to work today and what sort of issues are related to those
fixes. These fixes use rb()
and rb_mut()
helper methods that are
"reborrow-like" but do not perform true reborrowing, as will become apparent in
the examples.
/// Exclusive reborrowable reference to some data. Can be turned into a MyRef
/// by reborrowing as shared.
///
/// Note: this type is not Copy or Clone.
struct MyMut<'a>(...);
impl Reborrow for MyMut { /* ... */ }
/// Shared reference to some data.
#[derive(Clone, Copy)]
struct MyRef<'a>(...);
/// Reborrow-like helper methods, in the form of a trait.
trait Reborrow {
/// Type that represents the Self exclusive reference reborrowed as shared.
type Target;
fn rb_mut(&mut self) -> Self;
fn rb(&self) -> Self::Target;
}
fn mixed_reborrowing(source: MyMut) {
// Exclusive reborrowing
{
// Note: mutate() takes self by value, so in current Rust MyMut would be
// moved here. Instead we want it to be reborrowed as exclusive.
// Note: explicit "reborrow-like" function works today:
// FIX: let result = source.rb_mut().mutate();
let result = source.mutate();
// Again, we want the following line to fail to compile, and it does
// but currently for the wrong reasons:
// source.mutate();
result.observe();
}
// Shared reborrowing
{
// We want to enable the following calls to still work: this means that
// the first mutate() call must reborrow source instead of moving it.
// The following calls must also implicitly reborrow source as shared.
// Note: the explicit "reborrow-like" function works today:
// FIX: let result1 = source.rb().observe();
// FIX: let result2 = source.rb().observe();
let result1 = source.observe();
let result2 = source.observe();
// Note: source can be reborrowed as shared multiple times.
result1.observe();
result2.observe();
}
// After all shared reborrow results are dropped, source must be
// reborrowable as exclusive again.
// Note: even in current Rust we don't need to use the "reborrow-like"
// function here, as source is not used after this and we can thus move it
// out of the function.
source.mutate();
}
fn returning_derived_mut(source: MyMut) -> MyMut {
// This does work today, but only if source is moved into mutate() instead
// of using the "reborrow-like" function:
// NOT: source.rb_mut().mutate()
source.mutate()
}
fn returning_derived_ref(source: MyMut) -> MyRef {
// This does not work today, but can be made to work with a helper, such as
// the Into trait, that turns an owned MyMut into an owned MyRef. Again,
// using the explicit "reborrow-like" function does not work:
// FIX: source.mutate().into()
// NOT: source.rb_mut().mutate().into()
source.mutate()
}
// Some function returning an error where both variants refer to the source
// data; think eg. a parser that wants to return a structure containing
// references to source text slice or an error pointing to invalid source text.
fn result_function(source: MyRef) -> Result<MyRef, MyRef>;
// A function taking an exclusive reference, reborrowing it as shared to call
// other functions that return derived references, and then conditionally
// returning some results of those calls.
fn branching_function(source: MyMut) -> Result<MyRef, MyRef> {
// Note: here again we'd want automatic reborrowing as shared. If we use
// the explicit "reborrow-like" function then the resulting MyRef has a
// lifetime strictly bound to the temporary borrow of source. That is what
// we want for the Ok value, as we must drop result before source gets
// reused as exclusive, but for the Err value we conversely want the
// lifetime to be extended to the source lifetime so that we can return it
// from the function. This could work with Polonius, but only with true
// reborrowing instead of the "reborrow-like" functions. In effect, this
// function cannot currently be compiled without using lifetime transmutes.
// NOT: result_function(source.rb())?;
// This also does not work, since source is moved here but reused below.
let result = result_function(source)?;
// Do something with the result.
println!("Example: {:?}", result);
// Then mutate the source again.
Ok(source.mutate())
}
The most important point here is that just like with Option<&mut T>
, custom
exclusive reference-like types cannot be used in functions returning values with
derived lifetimes from branches. The derived lifetime is long enough (equal to
the source lifetime) if the reference-like type is moved out of the function and
into the subroutine, but then it cannot be reused. If reference-like type is
explicitly "reborrowed" using the helper functions, then the resulting derived
lifetime is strictly bound within the local function and cannot be returned from
it.
Let's look at usage of reborrowing, or the "reborrow-like" functions, of custom reference-like types in the wild. The following are a smattering of custom reference-like types and traits that either make use of "reborrow-like" functions, implement the "reborrow-like" function APIs, or are interested in using reborrowing in the future.
PeripheralRef
PeripheralRef
is an custom exclusive reference-like type that provides
exclusive access to hardware peripherals in an embedded system codebase. Of note
is that the size is usually either a ZST or a single byte in size as many
peripherals are either statically known and need no runtime behaviour, or have
only a small number of possible values to refer to.
Also note that PeripheralRef
does not have a corresponding shared
reference-like type.
- Size: Generic type dependent, but commonly ZST or 1 byte.
- Kind: Zero-sized marker type, or a small value for runtime decision making.
- Usage: Provide exclusive access to statically knowable (or mostly knowable) hardware peripherals in an embedded system codebase.
GcScope and NoGcScope
GcScope
and NoGcScope
are a custom exclusive and shared reference-like type,
respectively, that provide access to the Nova JavaScript engine's garbage
collector. The types are zero-sized and have no runtime existence, serving only
as a means of compile-time verification of garbage collector safety, or
use-after-free checking in other words. The "garbage collector lifetime" of the
type is observed by all garbage-collectable data handles, and is taken as
exclusive by any method that may trigger garbage collection. The result is that
methods that may trigger garbage collection always invalidate all
garbage-colletable data handles on call. Of note is that these types actually
carry a second lifetime as well, the "scope lifetime", which is used when
handles are rooted to the current scope. This second lifetime is always shared
reference-like, and thus does not participate in reborrowing.
- Size: ZST.
- Kind: Marker types with no runtime existence.
GcScope
has two lifetimes, one lifetime is exclusive and one is shared.NoGcScope
has two shared lifetimes. The always-shared lifetime takes no part in reborrowing and can be ignored here. - Usage:
GcScope
provides exclusive access to a garbage collector, while its shared reborrowNoGcScope
is used to create handles to garbage collected data. Handles are thus invalidated ifGcScope
is used as exclusive.
MutableHandle and Handle
MutableHandle
and Handle
are custom reference-like types to garbage
collectable data in the Servo browser's SpiderMonkey mozjs
wrapper.
- Size: Pointer-sized.
- Kind:
MutableHandle
is conceptually an exclusive reference type.Handle
is a wrapper around a shared reference. - Usage:
MutableHandle
is a reference type wrapping a raw pointer that can unsafely be turned into an exclusive reference on the condition that the garbage collector does not run while the exclusive reference exists.Handle
is a plain shared reference wrapper; the pointed-to value is (aspirationally) only mutated by the GC where it is internally mutable.
Mat, MatMut, MatRef, ColMut, and ColRef
Mat
is an owned rectangular matrix while MatMut
, MatRef
, ColMut
, and
ColRef
are various kinds of custom reference types over the source Mat
.
- Size: Usually 5 pointer-sizes for
MatMut
andMatRef
, 3 forColMut
andColRef
. - Kind:
MatMut
andColMut
are exclusive reference types.MatRef
andColRef
are shared reference types. - Usage: Enable various near-zero-cost operations over matrices by binding the
iteration to the original
Mat
through the reference types while statically ensuring that data access within the bounds of the borrow is valid.
Note: Mat
is also reborrowable into MatMut
and MatRef
, but also has a
Drop
implementation. If reborrowing is understood to act on lifetimes of
reference-like types, then reborrowing Mat
is outside that definition as it
neither has a lifetime nor is a reference-like type. It could be argued that
Mat
does have an implied 'static
lifetime, but the lack of reference-like
aspects of it at least cannot be denied. This use-case might be better thought
of as a generalised AsRef/AsMut
or perhaps Deref[Mut]
implementation.
reborrow crate
This is a library used in eg. faer (above Mat
and friends) that provides
traits for emulating reborrowing.
Honorable mention: Rust for Linux smart pointers
The Rust for Linux project has many custom smart pointer types with peculiar requirements beyond normal Rust usage. At least one type implements reborrowing, and according to some discussion on Zulip, there is some interest in having more of such types. One of the reasons why more of such types do not exist is said to be bad ergnomics around automatic reborrowing.
Honorable mention: CppRef and friends
These types do not implement reborrowing and do not always carry lifetimes, but cxx is interested in automatic coercion between the different C++ reference types. Since these are pointer wrapping types, some forms of autoreborrowing would make these coercions possible.
Guide-level explanation
When creating a custom exclusive reference types, users can derive a Reborrow
trait on the type which opts the type in to &mut
like semantics. The trait
checks that all of the fields of the type implement Reborrow
and return false
for needs_drop
. The trait has no methods and no parameter or associated types.
A blanket implementation of Reborrow
for Copy
types is provided by the
core
library.
For coercing a custom exclusive reference type into a shared reference type,
users can implement a CoerceShared: Reborrow
trait on the custom exclusive
reference type. The trait is simple and only requires specifying the Target
type:
trait CoerceShared: Reborrow {
type Target: Copy;
}
The Copy
bound on Target
ensures that all shared reference types are always
trivially copyable. The trait checks that for all fields in T::Target
, a field
with the same name exists in T
such that type Field
of the field in T
implements is Field: CoerceShared
where the type SharedField
of the field in
T::Target
is equal to the type <Field as CoerceShared>::Target
. A blanket
implementation of CoerceShared<Target = Self>
for Copy
types is provided by
the core
library.
When a value T: Reborrow
is used at a coercion site expecting T
, a new T
is created from the original value using the functional record update syntax and
the original value is disabled for reads and writes. The original value stays
disabled for writes until all fields of T
that are Reborrow
but not Copy
are dropped, and stays disabled for reads until all fields of T
that are
Copy
and carry a lifetime are dropped. This is considered a write on the
original value.
When a value T: CoerceShared<Target = U>
is used at a coercion site expecting
U
, a new U
is created from the original value with each field in U
performing an implied CoerceShared
method on a copy of the corresponding field
in T
. The original value stays disabled for writes until all fields of U
that are Copy
and carry a lifetime are dropped. This is considered a read on
the original value.
Custom exclusive to shared reference type method resolution
Implementing T: CoerceShared<Target = U>
makes it possible to call methods of
U
on T
. For example, for a type
MyMut<'a, T>: Reborrow + CoerceShared<Target = MyRef<'a, T>>
with a method
impl<'a, T> MyRef<'a, T> {
fn get(self) -> &'a T;
}
it is possible to call this directly on a value of type T
:
fn example<T>(t: MyMut<T>) {
let data: &T = t.get();
}
This implicitly applies CoerceShared
on MyMut<'a, T>
to create a
MyRef<'a, T>
and finds the get
method on that type.
Building custom reference types
A basic custom exclusive reference type is as follows:
struct MyMut<'a, T> {
ptr: NonNull<T>,
lifetime: PhantomData<&'a ()>,
}
The MyMut
struct does not derive Copy
and thus requires a Reborrow
implementation. When MyMut<T>
is reborrowed, the compiler translates
fn inner_mut(source: MyMut<T>);
fn outer_mut(source: MyMut<T>) {
inner_mut(source);
}
into something akin to
fn inner_mut(source: MyMut<T>);
fn outer_mut(source: MyMut<T>) {
let MyMut {
ptr,
lifetime,
} = source;
inner_mut(MyMut {
ptr,
lifetime,
})
}
where the original source
is marked used as exclusive by the compiler for the
lifetime of the resulting MyMut
, similarly to how it would work if lifetime
was a &mut
reference.
The corresponding custom shared reference type for MyMut
is as follows:
#[derive(Clone, Copy)]
struct MyRef<'a, T> {
ptr: NonNull<T>,
lifetime: PhantomData<&'a ()>,
}
and using CoerceShared
to convert a MyMut
into a MyRef
is translated into
something akin to
fn inner_ref(source: MyRef<T>);
fn outer_ref(source: MyMut<T>) {
let MyMut {
ptr,
lifetime,
} = source;
inner_ref(MyRef {
ptr,
lifetime,
})
}
where the original source
is marked used as shared for the lifetime of the
MyRef
.
Automatic reborrowing of structs with multiple fields
The field-wise definition of Reborrow
and CoerceShared
traits enables
automatic reborrowing of structs that have multiple fields and even multiple
lifetimes with different reference semantics associated with them.
A basic example, simplified from faer
's reborrowing use case, with a single
lifetime and multiple data fields is given below.
struct MatMut<'a, T> {
ptr: NonNull<T>,
rows: usize,
cols: usize,
row_stride: usize,
col_stride: usize,
__marker: PhantomData<&'a ()>,
}
#[derive(Clone, Copy)]
struct MatRef<'a, T> {
ptr: NonNull<T>,
rows: usize,
cols: usize,
row_stride: usize,
col_stride: usize,
__marker: PhantomData<&'a ()>,
}
fn inner_mut(source: MatMut<T>);
fn outer_mut(source: MatMut<T>) {
// inner_mut(source); is turned by the compiler into:
let MatMut {
ptr,
rows,
cols,
row_stride,
col_stride,
__marker,
} = source;
inner_mut(MatMut {
ptr,
rows,
cols,
row_stride,
col_stride,
__marker,
})
}
fn inner_ref(source: MatRef<T>);
fn outer_ref(source: MatMut<T>) {
// inner_ref(source); is turned by the compiler into:
let MatMut {
ptr,
rows,
cols,
row_stride,
col_stride,
__marker,
} = source;
inner_ref(MatRef {
ptr,
rows,
cols,
row_stride,
col_stride,
__marker,
})
}
It is also possible for the CoerceShared::Target
type to have a different
layout than the source type does. An example described by a user in their closed
source code base is given below.
struct ImbrisMut<'a, T> {
ptr: NonNull<T>,
metadata: usize,
__marker: PhantomData<&'a ()>,
}
#[derive(Clone, Copy)]
struct ImbrisRef<'a, T> {
ptr: NonNull<T>,
__marker: PhantomData<&'a ()>,
}
fn inner_mut(source: ImbrisMut<T>);
fn outer_mut(source: ImbrisMut<T>) {
// inner_mut(source); is turned by the compiler into:
let ImbrisMut {
ptr,
metadata,
__marker,
} = source;
inner_mut(ImbrisMut {
ptr,
metadata,
__marker,
})
}
fn inner_ref(source: ImbrisRef<T>);
fn outer_ref(source: ImbrisMut<T>) {
// inner_ref(source); is turned by the compiler into:
let ImbrisMut {
ptr,
metadata,
__marker,
} = source;
inner_ref(ImbrisRef {
ptr,
__marker,
})
}
It is also possible for there to be multiple lifetimes in the type with some of
those using exclusive reference semantics and others using shared reference
semantics. The reborrowing structure can also have multiple levels of depth. An
example from the Nova JavaScript engine's GcScope
is given below.
struct GcMut<'a> {
__marker: PhantomData<&'a mut GcToken>,
}
#[derive(Clone, Copy)]
struct GcRef<'a> {
__marker: PhantomData<&'a mut GcToken>,
}
#[derive(Clone, Copy)]
struct ScopeRef<'a> {
__marker: PhantomData<&'a ScopeToken>,
}
struct GcScope<'a, 'b> {
gc: GcMut<'a>,
scope: ScopeRef<'b>,
}
struct NoGcScope<'a, 'b> {
gc: GcRef<'a>,
scope: ScopeRef<'b>,
}
fn inner_mut(source: GcScope<'a, 'b>);
fn outer_mut(source: GcScope<'a, 'b>) {
// inner_mut(source); is turned by the compiler into:
let GcScope {
gc,
scope,
} = source;
let GcMut {
__marker,
} = gc;
inner_mut(GcScope {
gc: GcMut {
__marker,
},
scope,
})
}
fn inner_ref(source: NoGcScope<'a, 'b>);
fn outer_ref(source: GcScope<'a, 'b>) {
// inner_ref(source); is turned by the compiler into:
let GcScope {
gc,
scope,
} = source;
let GcMut {
__marker,
} = gc;
inner_ref(NoGcScope {
gc: GcRef {
__marker,
},
scope,
})
}
There is an interesting point here: the scope
field can effectively be copied
directly into the resulting GcScope
or NoGcScope
in either exclusive or
shared reborrowing case, as it is Copy
. Its functional record update syntax
could be written out but would not make much of a point. The important bit there
is that the resulting scope
should not keep the original source
used at all.
The gc
field on the other hand needs to keep the source
used. This is
achieved by having the compiler mark the source.gc
field as used, either as
exclusive for Reborrow
or as shared for CoerceShared
, for the lifetime of
the resulting GcMut
/ GcRef
type.
Why recurse into non-Copy reborrowable fields?
In the above GcScope
example it could be argued that GcRef
should be
reborrowed as exclusive by the compiler, without looking into its fields and
performing the functional record update syntax on it. This would work for
GcRef
, and would even be more clear-cut than the destructuring since the code
"turned by the compiler into", as written above, would effectively compile with
minor changes (let GcMut { .. } = &mut source;
) in today's Rust because all
fields within GcRef
are Copy
but it would not mark the source
as being
used.
The reason for recursing becomes clearer if we take a more complicated example. Imagine a larger wrapper type with multiple reference-like fields, some of them with exclusive reference semantics, some of them with shared reference semantics, and some of the containing other wrappers that themselves contain exclusive and shared reference semantics.
struct Context<'a, 'b, 'c, 'd> {
gc: GcScope<'a, 'b>,
arena: &'c mut Arena,
validity_bool: &'d AtomicBool,
}
Now if this type was made reborrowable and that reborrowing act only performed
an exclusive reborrow on GcScope<'a, 'b>
without looking at its fields, then
the result would be a new GcScope
where both of the lifetimes keep the
original GcScope
"used".
fn outer_mut(source: Context) {
let Context {
gc,
arena,
validity_bool,
} = source;
inner_mut(Context {
gc,
arena,
validity_bool,
})
}
Now if the 'b
lifetime escapes in some Value<'b>
, that Value
would keep
the gc
field borrowed as exclusive. This is not what the GcScope
internally
defines; its internal view of the 'b
lifetime is that it has shared reference
semantics. This is effectively the same issue that mutating gc.gc
and reading
gc.scope
at the same time is okay, but calling a method on gc
that mutates
self.scope
and reading gc.scope
at the same time is not okay. In the first
case, only gc.gc
is used exclusively and gc.scope
is used as shared, whereas
in the latter case the whole gc
is used exclusively and gc.scope
is used as
shared, which is an aliasing violation.
The correct formulation for transitive reborrowing is then this: for each field
in the reborrowed type that is !Copy
, recurse into the type and check if it
has any !Copy
fields. If it does, recurse into it. If not, then mark that
field used. An example using Context
is given below.
fn outer_mut(a: Context) {
// `Context: !Copy` so recurse.
let Context {
// `GcScope: !Copy` so recurse.
gc,
// `Arena: !Copy` but has no fields: mark as used.
arena,
// `&T: Copy` so skip.
validity_bool,
} = source;
let GcScope {
// `GcMut: !Copy` but has no `!Copy` fields: mark as used.
gc,
// `ScopeRef: Copy` so skip.
scope
} = gc;
let GcRef {
__marker,
} = gc;
inner_mut(Context {
gc: GcScope {
gc: GcMut {
__marker,
},
scope,
},
arena,
validity_bool,
})
}
Reference-level explanation
core
libs
Two new traits, Reborrow
and CoerceShared
, are added to and exposed from
core::ops
.
pub trait Reborrow {}
pub trait CoerceShared: Reborrow {
type Target: Copy;
}
A blanket implementation is provided for any type that implements Copy
:
impl<P: Copy> Reborrow for P {}
impl<P: Copy> CoerceShared for P {
type Target = Self;
}
The traits are also implemented for &mut T
and &T
:
impl<T: ?Sized> Reborrow for &mut T {}
impl<T: ?Sized> CoerceShared for &mut T {
type Target = &T;
}
Blanket implementations for Pin<&mut T>
and Option<&mut T>
are also added:
impl<T: ?Sized> Reborrow for Pin<&mut T> {}
impl<T: ?Sized> CoerceShared for Pin<&mut T> {
type Target = Pin<&T>;
}
impl<T: ?Sized> Reborrow for Option<&mut T> {}
impl<T: ?Sized> CoerceShared for Option<&mut T> {
type Target = Option<&T>;
}
Compiler changes: method probing
The existing Rust reference section for method calls describes the algorithm for assembling method call candidates, and there’s more detail in the rustc dev guide.
The key part of the first page is this:
The first step is to build a list of candidate receiver types. Obtain these by repeatedly dereferencing the receiver expression’s type, adding each type encountered to the list, then finally attempting an unsized coercion at the end, and adding the result type if that is successful. Then, for each candidate
T
, add&T
and&mut T
to the list immediately afterT
.
We add to the "for each candidate T
" list expansion the step that if
T: CoerceShared
where <T as CoerceShared>::Target
is not equal to T
, and
T
is not equal to some &mut U
then <T as CoerceShared>::Target
is added to
the list immediately after &mut T
. Note: the condition on T
not being some
&mut U
is because of &mut T: CoerceShared<Target = &T>
, which would
otherwise produce unnecessary duplicates.
This does change the list of candidate receiver types, adding shared coercion targets to it when applicable. These should be rare, so the effective change is minimal.
Compiler changes: reborrow adjustments
In the compiler, when resolving coercion a new possible reborrow adjustment is
added. The condition for performing this coercion is that the source type is
Reborrow: !Copy
and the target type is equal to the source type, or that the
source type is CoerceShared: !Copy
and the target type is equal to the source
type's CoerceShared::Target
type. In the first case an exclusive reborrow
adjustment is inserted, and in the second case a shared reborrow adjustment is
inserted. This replaces the PinReborrow
adjustment currently in place for
Pin<&mut T>
reborrowing specifically.
This adjustment creates a new target type struct by performing the following steps with the source type and the target type as its parameters:
- Let
should_mark
betrue
. - For each field in the target type, do:
- If the target field type is
Copy
and the corresponding source field type isCopy
, then- Assert that the two types are the same.
- Perform a copy from source to target.
- If the target field type is
Copy
and the corresponding source field type is!Copy
, then- Assert that the reborrow adjustment kind is "shared".
- Assert that the source field type implements
CoerceShared
and itsCoerceShared::Target
is equal to the target field type. - Set
should_mark
false. - Note: marking
should_mark
false is superfluous, we could mark any number of reborrowable source fields used as shared. - Recursion: perform these steps (starting at the top) with the source field type and the target field type as parameters.
- Note: the recursion is needed for handling changes in memory layout between source and target field types.
- If the target field type is
!Copy
and the corresponding source field type is!Copy
, then- Assert that the reborrow adjustment kind is "exclusive".
- Assert that the two types are the same.
- Set
should_mark
false. - Recursion: perform these steps (starting at the top) with the source field type and the target field type as parameters.
- Note: the recursion is needed for ensuring that only necessary parts of the source type are reborrowed as exclusive.
- If the target field type is
- If
should_mark
is true, then- If the reborrow adjustment kind is "shared", then
- Mark the the source type used as shared.
- If the reborrow adjustment kind is "exclusive", then
- Mark the source type used as exclusive.
- If the reborrow adjustment kind is "shared", then
Drawbacks
The main drawback of autoreborrow traits is the conversion from the T
to
<T as CoerceShared>::Target
and the related addition to method probing logic.
This adds a new piece to an already complicated Deref coercions set.
Reborrowable custom reference types are also relatively niche, though very
useful.
The algorithm for the compiler to crunch through reborrowed vs copied fields is not too bad, but it's still a bit involved and if it were to be reproduced at each reborrowing coercion site then the performance penalty would probably be significant. Luckily, the results of the algorithm never change so it should be fairly easy to effectively cache the results.
Aside from that, reborrowing is not too complex and rustc already has existing
code for autoreborrowing Pin<&mut T>
, which this would supersede.
Rationale and alternatives
Rationale
The rationale for choosing a single trait with a single target GAT and no methods is as follows:
- Reborrowing only makes sense for exclusive reference types: reborrowing of shared reference types as shared is exactly equal to copying the shared reference type.
- An exclusive reference should only be exclusively reborrowable into itself,
ie.
&mut T -> &mut T
orMatMut<'_> -> MatMut<'_>
. A&mut T -> &mut U
should be thought of as a reborrow followed by a dereference. Hence, a separateReborrow::Mut
GAT is not needed. - An exclusive reference type that doesn't need to be reborrowable as shared may
exist: the proposed
Reborrow
trait does not directly support this, but it can implemented by creating a dummy type (with same layout as the exclusive reference) which is never used and has no methods on it. The exclusive reference type will thus be reborrowable as shared, but into a type that cannot be used for anything. - Adding a method to reborrowing would make all coercion sites into implicit injection points into user-code. This seems both too powerful, too magical, and too subtle to allow, hence no method is proposed.
Compiler-level reasoning around custom type autoreborrowing
Exclusive reborrowing of custom types is fairly straightforward as the only
complexity lies in implementing the lifetime replacement and extension, and the
disabling of the source T
. This is already implemented in the compiler for
Pin<&mut T>
specifically, and extending this for all types implementing the
Reborrow
trait ought not be too complicated.
Shared reborrowing is much less so, as it requires the compiler to copy the
source T
into <T as Reborrow>::Ref
. The usual case for reborrowable types is
that they have only one (or no) data field of the same kind for both the
exclusive and shared reference types, meaning that a transmute would be
trivially sound (at least for #[repr(transparent)]
types), but for multi-field
types the soundness of the transmute cannot be guaranteed. A basic reasoning
mechanism safe transmutes is provided by the
safe transmute project,
which can detect grievous mistakes like transmuting padding bytes into data.
More complex reasoning, like making sure that two usize
fields do not
logically flip in the transmute, cannot be performed by the compiler and would
require #[repr(C)]
usage on both sides of the transmute. As this requires
manual checking by the user of the Reborrow
trait, the trait is proposed as
unsafe
with the requirement being that T
is transmutable into T::Ref
.
Hence, the trait's safety requirements are inherited from std::mem::transmute
.
Mitigating the unsafety of an injected transmute
As the transmuting of T
to T::Ref
is unsafe and requires manual checking,
we'd like to avoid the injected transmute. The obvious thing to do would be to
add a method fn reborrow(self) -> T::Ref
(note: the method must take self by
value, not by reference) to the Reborrow
method but as discussed above, this
is not proposed because it would mean injecting implicit user-controlled code
with arbitrary side-effects to every coercion site. This runs counter to what
reborrowing is generally though of being, "copy of a pointer with extra lifetime
analysis", and we'd thus want to avoid it. (Contrast this with adding an
fn copy()
method to the Copy
trait.)
With the safe transmute project, the implicit transmute could be improved by
requiring a type Ref: TransmuteFrom<Self, _>
trait bound on the GAT: with this
the source and target would at least be confirmed to be safely transmutable. As
mentioned earlier, this would still not guarantee that the transmute is sound
but would at least go a long way in mitigating the issue.
An alternative to an explicit reborrow
method would be to add a
type Ref: From<T>
bound to the GAT and have the compiler inject a
<T as Reborrow>::Ref::from(T)
call at the reborrow site. The downside here is
that the From
impl is still user-controlled code and could likewise have
arbitrary side-effects. That being said, From
trait implementations are well
understood (as compared to a custom reborrow
method, or even eg. Deref
vs
AsRef
vs Borrow
), are held to a high standard, and misusing said traits is
likely to cause various other issues; they are thus unlikely to contain
side-effectful code.
Finally, in the future if/when const fn
s in traits become available,
improvements on the above method-based approaches appear: the custom reborrow
method could be made into a const fn reborrow()
or, if trait bounds requiring
const
implementations of traits or trait methods become possible, the From
bound could be turned into Target: const From<Self>::from
. With this the fear
of entirely arbitrary side-effects is further mitigated, though not entirely
removed.
The safe transmute trait bound would not be enough for the Reborrow
trait to
be marked as safe, due to the soundness issue. Hence, it is not a complete
mitigation.
Of the possible complete mitigations, this RFC considers only the
type Ref: From<T>
bound worthy of further study. An explicit reborrow method
would have high chance of misuse, but a From
trait conversion can be fairly
well trusted to be valid and safe to call implicitly. Adding const
bounds
would restrict the possible misuse of a reborrow
or from
method, but as the
capabilities of const
expand the restrictions placed by this bound would
slowly erode. Hence, those bounds would not really be reliable soundness
guarantees.
Detecting the reborrowed lifetime
In the above reborrow formulation, the Reborrow
trait itself has a lifetime
parameter that the compiler uses to detect which lifetime is being reborrowed,
in case there are multiple (or none). This seems like a neat solution, but if it
is deemed problematic then an alternative would be to require that reborrowable
types and their Ref
targets have exactly one lifetime.
This would cause some trouble for some users (including the author's GcScope
),
but if it is the price that has to be paid for true reborrowing of custom types,
then so be it.
Alternatives
Alternative 1: Reference-based method(s)
The most obvious alternative to the proposed Reborrow
trait would be to take
what we already have in userland and make it part of the core library and rustc.
This would mean implementing a trait or traits in the vein of the reborrow
crate:
/// Shared reborrowing.
trait Reborrow<'short, _Outlives = &'short Self> {
type Target;
#[must_use]
fn rb(&'short self) -> Self::Target;
}
/// Exclusive reborrowing.
trait ReborrowMut<'short, _Outlives = &'short Self> {
type Target;
#[must_use]
fn rb_mut(&'short mut self) -> Self::Target;
}
Note that two traits are given here to match what the reborrow
crates does,
not as an explicit alternative proposal.
The main benefit with this approach is that this follows the current userland state-of-the-art way of creating "reborrow-like" functions and would effectively bless a particular, well-liked userland implementation into the language together with compiler support for automatically injecting the trait method calls, relieving the user of having to call the methods manually.
The main issue with this approach is that this does not implement true
reborrowing: the 'short
lifetime gets bound to the temporary borrow created
when calling these methods, meaning that lifetime extension does not work out of
the box. The 'short
lifetime is a strict subset of the source lifetime (call
it 'long
), instead of being able to extend to equal it. To overcome this
issue, rustc would need to special-case this trait implementation to allow
lifetime extension of 'short
into 'long
. But doing so is unsound!
As the implementation of these methods is user-controlled, users often find neat
ways to stretch implementations beyond the limits of what was intended but not
explicitly made impossible. In a pure reborrow frame of mind, turning
&'short Self<'long>
into Self<'short>
with 'short
being allowed to
lifetime extend to equal 'long
seems sound, but it is not. The &Self
reference is a pointer onto the stack (most likely) and allowing lifetime
extension from Self<'short>
to Self<'long>
would mean that rustc would allow
the &Self
reference to escape its call stack. This is equal to returning a
pointer pointing to released stack memory, ie. undefined behaviour.
Thus, in current Rust syntax a Reborrow
trait taking self
by reference while
allowing lifetime extension would be unsound. To solve this issue, a new syntax
for taking a value by reference that cannot be used as a place or turned into an
address would be needed, perhaps something like:
trait Reborrow<'short, _Outlives = &'short Self> {
type Target;
fn reborrow(^'short self) -> Self::Target;
}
trait ReborrowMut<'short, _Outlives = &'short Self> {
type Target;
fn reborrow_mut(^'short mut self) -> Self;
}
As this sort of a reference would be something entirely new (though it probably
relates to Pin<& (mut) T>
somehow), it is deemed too complicated a change for
this RFC. Because true reborrowing is one of the focal aims of this RFC, any
reference-based trait method is thus considered a non-starter due to the
soundness issue.
Alternative: Generic type parameter
The Reborrow
trait (any version of it) could be defined with a generic type
parameter. This would be more generic than having a generic associated type,
though it would likely make the implementation correspondingly more complex as
well. This doesn't help with the soundness issues of a reference-based method or
anything else related to the trait. It also has a new issue lurking within it: a
Reborrow
trait that can be implemented multiple times over gives users the
power needed to implement automatic type coercion. Eg. for integer coercion, you
need only to implement Reborrow<uN> for MyInt
and Reborrow<iN> for MyInt
for
all N
of your preference and now your MyInt
type will be reborrow-coerced
into whatever integer type is needed by a given call.
For this reason, this RFC does not propose a Reborrow
trait with a generic
type parameter.
Alternative: Special reborrow syntax
If userland reborrowing was made explicit in the source code using special syntax, then the compiler would not need to implicitly inject user-controlled code at coercion sites. This would make user-controlled reborrow methods perfectly acceptable, as they would be visible in user code.
The
ergonomic clones
feature has added a foo.use
suffix operator for (effectively) calling
clone()
in an ergonomic way. A similar thing could be done for reborrowing
using eg. the ref
and mut
keywords:
fn mixed_reborrowing(source: MyMut) {
let result = source.mut.mutate();
result.ref.observe();
let result1 = source.ref.observe();
let result2 = source.ref.observe();
result1.ref.observe();
result2.ref.observe();
source.mut.mutate();
}
This does not solve the problem of lifetime extension in any way, ie. this does
enable implementing true reborrowing. It also takes away the "auto" from
"autoreborrowing". (The ref
and mut
suffix syntax might also be better used
for other purposes, such as generalised Deref
/ DerefMut
or AsRef
/
AsMut
calls.) For these reasons, this RFC does not propose adding a custom
syntax for reborrowing.
Alternative: "Event handler" methods
The soundness issue related with returning a new value from a reference-based
reborrow method makes reference-based reborrow methods a non-starter. If
user-controlled code during reborrowing is still wanted, an alternative to the
conversion methods is to have "event handler" methods like on_reborrow
and
on_reborrow_mut
. These methods would take the reborrow result by reference but
would have no return value, thus avoiding the soundness issue at the cost of
still requiring the compiler to inject bitwise copies and transmutes.
These would allow compiler-injected reborrowing to trigger user-controlled code,
which would be beneficial for eg. reference counted types. This would be useful
for eg. RefCell<T>
's Ref<'a, T>
and RefMut<'a, T>
types, which perform
(cheap) reference incrementing/decrementing on Clone and Drop. If the reference
counting of these types could be performed as part of reborrowing, then methods
like RefMut::map_split
could be generalised over both reference counting and
reborrowing types.
The downside here is that cloning is explicit, and this explicitness and clippy linting makes it easy for users to opt out of cloning when it is not needed. With compiler-injected reborrowing, this becomes much more complicated. Consider the following code:
fn method(r: RefMut) {
method_a(r);
method_b(r);
}
In current Rust, this will not compile as r
is moved into the method_a
call.
To fix this, an explicit .clone()
must be inserted:
fn method(r: RefMut) {
method_a(r.clone());
method_b(r);
}
Now imagine the original code being compiled with RefMut
implementing
Reborrow
(and still being Drop
like previously): both method calls will
implicitly reborrow r
, effectively performing two clone calls and one drop
:
fn method(r: RefMut) {
method_a(r.reborrow()); // calls on_reborrow() on the result
method_b(r.reborrow()); // cals on_reborrow() on the result
// implicit `drop(r);` happens here
}
A sufficiently smart compiler could theoretically optimise the extra clone out,
but because of the on_reborrow
method it would be very hard or impossible to
prove that the combination of r.reborrow()
and drop(r)
is equal to moving
r
into the second call. The "ergonomic clones" RFC
does include a provision
for allowing the compiler to optimise out extra clones; the same kind of
provision could be asserted for Reborrow
to allow the compiler to remove the
extra calls, but it is an added complexity.
Therefore, this RFC does not propose any on_reborrow
like trait methods and
instead suggests keeping reference counting and reborrowing separate from one
another.
Related efforts
Generalised Deref
and DerefMut
traits
The Deref
and DerefMut
traits are quite magical and powerful, but they are
not quite powerful enough to implement the userland reborrow-like functions
because they must always return references. A generalised
Deref(Mut)Generalised
trait would be one where the return value is the
Target
GAT directly and not a reference of it. These traits would be strong
enough to replace the reborrow
crate and would have the benefit that a
DerefGeneralised
trait enables passing reborrowable parameters with a simple
&my_ref
or &mut my_mut
to explicitly trigger dereferencing. This would be
visible in the source code (usually; method calls wouldn't show them) and thus
the trait methods would not be as scary, compared to implicitly inserted
user-controlled code.
Again, these traits on their own do not and cannot solve the lifetime extension issue. In this sense, the current userland's "reborrow-like" functions could actually be more accurately called "generalised deref functions" as the two are exactly equivalent.
Reborrowing of reference counted references
Briefly discussed above in the "event handler" methods alternative, this section expands on how and why reference counted references might want to tap into method-based reborrowing, and why this is probably not the best of ideas.
The standard library's RefCell<T>
gives out Ref<'a, T>
and RefMut<'a, T>
custom reference types; each of these carries with it a pointer to the reference
counted data. Ref
s are Clone
and increment the RefCell
's ref counter,
while RefMut
have an internal clone
method that gets used in eg. the
RefMut::map_split
method and increments the RefCell
's mut counter. Both
types then have Drop
impls that decrement their respective counters again at
the end of their lifetimes. These types of references are not reborrowable in
the traditional sense as they need to run custom code during reborrowing, but
they are reborrowable in the sense that a Ref
or RefMut
created based on an
earlier RefMut
should be allowed to outlive its source.
eg. Given the following kind of code:
fn example() {
let rc = RefCell::new(0u32);
let result = {
let ref_mut = rc.borrow_mut(); // mut counter +1
{
let (ref_mut_1, ref_mut_2) = ref_mut.map_split(...); // mut counter +1
drop(ref_mut_1); // mut counter -1
ref_mut_2
}
};
println!("ref_mut_2: {:?}", result);
// mut counter -1
}
On the type level, the ref_mut
is moved out of in map_split
, meaning that
ref_mut_1
and ref_mut_2
are both dependent on a borrow that, in a sense,
dropped already. Despite this, they can not only be used but also returned from
their current scope and moved to parent scopes up until the scope where the
originating RefCell
is.
This is very similar to how reborrowed references can use lifetime extension to
return to their "true source of origin" despite being borrowed deep inside a
call graph. If RefMut
's clone method were to be rewritten in terms of a
Reborrow
trait with a method incrementing the mut counter, and if the
Reborrow
trait could be extended to allow types containing data with Drop
impls, then the RefMut::map_split
could be generalised further.
There is a complication to this: since the borrow: BorrowRefMut
field of
RefMut
implements Drop
, the above code could see three drop calls instead of
the current two. Right now ref_mut
is never dropped as it is moved out of and
into map_split
but if that move was made into a reborrow, then an extra
reborrow-injected clone would happen at the map_split
call site and an extra
drop would happen when the scope in which ref_mut
was create is left. As an
optimisation, the compiler could choose to only perform a reborrow when a move
isn't possible, but that might be very complicated to implement.
Prior art
- Formalise reborrows RFC: rust-lang#2364
- Reborrow trait pre-RFC: Rust Internals Forum #20774
- Feature request to simulate exclusive reborrows in user code: rust-lang#1403
- Experimental support for Pin reborrowing: rustc#130526
- Obscure Rust: reborrowing is a half-baked feature haibane_tenshi's blog
Unresolved questions
Matching the custom type's lifetimes to the reborrowed lifetime can be done via a lifetime bound on the trait, but it is a magical feature that doesn't really otherwise have much of a use. Whether this matching could be done in a better way is an unresolved question, possibly in a manner allowing multiple reborrowed lifetimes if that happens to be reasonable as that would make passing of "collection of exclusive references" kind of context structs easier. This is not considered a high priority for this RFC, however.
Future possibilities
It seems probable that autoreborrowing will open wide the doors for AsRef
,
Deref
, Borrow
, and similar traits to be re-examined and extended in a world
where custom reference types are commonplace. As an example, currently a
&mut T
can be passed to a method that takes &mut U
where T
dereferences
into a U
. With the proposed Reborrow
trait, this would not be possible. A
Reborrow
-aware generalised Deref
trait that enables this would need to take
the custom reference type T
by value and produce a U
as a result, ie. an
Into
trait. Allowing Into::into
calls to be injected at reborrow sites seems
like a possibly bad idea, and is thus not proposed in this RFC. Revisiting this
is, however, possibly worthwhile at a later time if eg. ^(mut)
-like references
that cannot be used as places make it into the language.
Additionally, some custom reference types require a context parameter for
performing actual dereferencing. Far into the future, a true generalised Deref
trait might thus look something like this:
impl<'a, T> DerefGeneralised for ArenaHandle<'a, T> {
type Context = &'a Arena;
type Target = &'a T;
fn deref_generalised(^self, ctx: Self::Arena) -> Self::Target {
// ...
}
}
But, as the traits become more generic they become more and more complex and error-prone, and thus it is unclear if these sorts of generalisations truly deserve to make it into the language or if they should forever stay in userland only.