This is intended to be an eRFC, or perhaps now it would be an MCP, for a short-to-medium-term change to unstable behavior on the way to a long-term solution. cc @Ericson2314
This resulted in a design notes PR: Note design constraints on hypothetical `DynSized` by CAD97 · Pull Request #166 · rust-lang/lang-team · GitHub
Summary
(Temporarily?) Resolve the "what do size_of_val
and align_of_val
do" question on extern type
by punting it to the developer defining that specific type.
This could in the future become an implementation of DynSized
and perhaps Pointee
for custom metadata. As such, and because this eRFC is specifically intended to be an experimental unstable stepping stone between the current state and eventual stable extern type
, we provide two potential spellings: both with and without DynSized
.
Motivation
extern type
and size_of_val
/align_of_val
do not play well together. size_of_val
is defined over T: ?Sized
, so under the current rules, this means that it can be called with an extern type
. However, the hole point is that the Rust world doesn't know the size of an extern type
, and perhaps that it's actually unknowable!
As of the writing of this eRFC, there is weak team support for making the use of extern type
in generics fail to compile via an internal default-bound DynSized
trait which extern type
s do not implement.
The purpose of this eRFC is to allow developers to define extern type
which are DynSized
(i.e. the size/align is knowable dynamically), thus supporting use cases like a thin CStr
or simple custom DSTs.
Guide-level explanation
By default, when you define an extern type
, you cannot use this type in a generic context. This is because Rust knows nothing about the type, so it can't monomorphize code to use the type.
In order to use an extern type
in a generic context, you need to tell Rust two things: how to determine the size of a pointee and the required alignment of a pointee. If these two facts about the type are known, we say that the type is dynamically sized. (If the size of the type is statically known, i.e. T: Sized
holds, then the type is statically sized.)
With DynSized
In order to provide this information, you implement DynSized
:
extern {
type CStr;
}
unsafe impl DynSized for CStr {
fn align_of_val_raw(_: ()) -> usize { 1 }
unsafe fn size_of_val_raw(this: *const Self) -> usize {
libc::strlen(this.cast()) + 1
}
}
trait DynSized
is defined in the implementation-level section.
Without DynSized
In order to provide this information, you supply two items at the definition of your extern type
:
extern {
type CStr {
fn align_of_val_raw(_: ()) -> usize { 1 }
unsafe fn size_of_val_raw(this: *const Self) -> usize {
libc::strlen(this.cast()) + 1
}
}
}
The signature of align_of_val_raw
is explained in the alternatives section.
By defining size_of_val_raw
and align_of_val_raw
, Rust now knows enough about the type to monomorphize code. There are a number of restrictions on size_of_val_raw
and align_of_val_raw
unlike normal functions to facilitate this:
-
align_of_val_raw
must be a power of two. -
size_of_val_raw
must be a multiple ofalign_of_val_raw
. -
size_of_val_raw
must accurately represent the size of this instance of the type and the size of the allocated object[1]. -
align_of_val_raw
is provided any pointer, potentially even a null pointer[2]. The pointer may be underaligned.[3] -
size_of_val_raw
is provided a raw pointer with read-only permissions (it may not be used to write to the pointee). The pointer may be underaligned[3:1]. -
size_of_val_raw
andalign_of_val_raw
must be pure and not have any observable effects beyond the time it takes to calculate. Calls may be inserted spuriously[4] and calls may be removed[5]. - The return value of
size_of_val_raw
andalign_of_val_raw
must not change for any given instance of the type, even as the instance gets mutated. -
size_of_val_raw
may be called with a pointer to droppedSelf
[6]. The author of the dynamically sizedextern type
is responsible to ensure thatsize_of_val_raw
still functions correctly after callingDrop::drop
.
If any of these requirements are not met, the program behavior is undefined, even if the type is not used.
The following changes are made to existing functions in the standard library and langugage[7]:
- Any generics
<T: ?Sized>
no longer acceptextern type
s that are not dynamically sized. -
std::mem::size_of_val
callsDynSized::size_of_val_raw
for extern types. -
std::mem::align_of_val
callsDynSized::align_of_val_raw
for extern types. -
std::mem::size_of_val_raw
(and thusLayout::for_value_raw
) requires that pointers to anextern type
sized tail are pointers to an allocation of a valid but potentially dropped instance of the extern type tail. -
std::mem::align_of_val_raw
becomes safe... modulo concerns about other unsized tails.
Implementation-level explanation
A new default-bound lang item unsafe OIBIT is added: DynSized
.
unsafe trait DynSized: ?DynSized {
fn align_of_val_raw(metadata: <Self as Pointee>::Metadata) -> usize;
unsafe fn size_of_val_raw(this: *const Self) -> usize;
}
We have Sized: DynSized
, and ?DynSized
implies ?Sized
. DynSized
is primarily intended as a forever-unstable implementation trait like Freeze
is, but depending on its utility in user code, it may be a candidate for future exposure.
All types except for extern type
(and perhaps other future custom DSTs) are DynSized
. extern type
s are not DynSized
, and DynSized
can be implemented for these types by user code.
The automatic implementation of DynSized
implements align_of_val_raw
and size_of_val_raw
using the implementation of the min_align_of_val
and size_of_val
intrinsics. (See their current implementation in rustc, which should now bug!
when used on ty::Foreign
.)
mem::align_of_val_raw
and mem::size_of_val_raw
are removed/deprecated in favor of DynSized::*
. mem::align_of_val
and mem::size_of_val
forward to DynSized::*
.
Drawbacks and rationale
DynSized
adds complexity to the language. However, it is the author's belief that DynSized
is a simplification over the ad-hoc additional rules required to support the RFC-accepted extern type
feature well. Both the current state where size_of_val
returns 0
and the proposed state where size_of_val
panics and a best-effort lint is provided are annoying footguns. In fact, the existence of a lint may require an implementation that looks a lot like DynSized
. A later tentative proposal was to ban the use of extern type
in generics altogether; this is what this eRFC is reifying. Even if DynSized
is never exposed to the developer, a mechanism that looks very similar to the OIBIT will have to exist anyway, so it makes sense to implement it as one.
If/when DynSized
is ever stabilized (which this eRFC is not proposing to necessarily happen), a lot of code will want to change from T: ?Sized
to T: ?DynSized
. To a first order of approximation, this would likely be most code dealing in just Borrowing<'_, T>
and no code dealing in Owning<T>
.
Disclaimer: the eRFC author is the author and maintainer of an FFI binding which optionally uses extern type
and makes great use of a generic Handle<T: LibExternType>
. This necessarily biases the author to the version of this proposal which allows T: ?DynSized
bounds in user code.
Alternatives
This eRFC uses DynSized::align_of_val_raw(<T as Pointee>::Metadata)
to be compatible with dyn Trait
(and future custom fat DSTs), which requires a valid DynMetadata
. The "don't say DynSized
" way of specifying a dynamically sized extern type
shows this in its signature for align_of_val_raw
, but if DynSized
is purely an implementation detail, it could be an associated const
. This would allow the power-of-two alignment to be easily enforced, but would preclude future extensibility using that syntax to custom Pointee
metadata.
Portions of the DynSized
trait could be made const
, and no consideration towards the implications of ~const DynSized
has been made by the author.
Pointee
and DynSized
could be merged into a single type, but this seems undesirable, since truly unsized extern type
are still pointees, and it would prohibit Thin
from being a simple trait alias.
Also, bikeshedding. At a minimum, since align_of_val_raw
takes metadata rather than a pointer, that's probably a bad name; perhaps align_for_metadata
?
Prior art
The entire purpose of extern type
is to emulate the use of incomplete types in C APIs for typesafe opaque handles. When a type is incomplete in C (or C++), the compiler refuses to emit any glue which requires knowledge about the type T
. The only thing which you can do with an incomplete type is talk about pointers to it, and anything you learn about the type is provided by a function. This is what is represented as ?DynSized
.
There's two ways main that a C FFI library can use an incomplete type:
typedef struct LIB_SYSTEM LIB_SYSTEM;
// fully opaque; allocated by the lib
LIB_SYSTEM* lib_create_system();
void lib_release_system(LIB_SYSTEM*);
// partially opaque; allocated by the caller
size_t lib_system_size(); // NB: C alloc typically doesn't specify alignment
void lib_create_system(LIB_SYSTEM*);
void lib_release_system(LIB_SYSTEM*);
The latter is primarily used in resource-constrained or hard realtime libraries where introducing a malloc
call is undesirable, to allow the caller to use their own allocation strategy, whether that be a stack buffer, arena, or just a call to malloc
. With DynSized
and custom allocators, a similar pattern becomes possible in Rust[8].
Unresolved questions
- Unknown unknowns
Future possibilities
- We could allow explicit implementations of
Pointee
onextern type
s to provide custom fat pointer metadata for full custom DST support. - Fixing the FFI ecosystem, which is currently using various iterations of patterns for representing opaque structs. Notably, implementing object-safe traits for opaque types represented this way is a giant footgun because it creates a vtable that says the size/align is 0/1. This is probably not a soundness hole, but this relies on the absence of code which tries to do something clever with the size/align, and it is an open question whether accessing past the
Sized
bytes is allowed by the Rust memory model.
-
This means that pointer reads within
size_of_val_raw
bytes must be valid, and reads outsidesize_of_val_raw
bytes are assumed to be invalid. In addition, copyingsize_of_val_raw
bytes must copy the entire object, andsize_of_val_raw
bytes must be the correct size to allocate/deallocate memory for this instance of this type. ↩︎ -
Alignment needs to be known before indirection (without looking at the type instance) in order to determine the field offset of an unsized tail. This is most commonly found in the implementation of
Rc
/Arc
, which internally store roughly a pointer to roughly astruct RcInner<T: ?Sized> { ref_count: RefCount, data: T }
. Without knowing the alignment ofT
, it's impossible to offset a pointer to the field in order toalign_of_val
it. ↩︎ -
Consider
#[repr(packed)] struct Packed { tail: ExternTail }
. ↩︎ ↩︎ -
Being able to insert spurious calls is required for common code motion optimizations, but the compiler will attempt to avoid inserting unnecessary calls, as calculating the size/align is allowed to be nontrivial (e.g.
strlen
walks the length of a null-terminated string). ↩︎ -
E.g. when
std::mem::size_of
is called multiple times, it is an allowed optimization to remember the results of the first call and reuse them, rather than recalculating the size. ↩︎ -
This is for the purpose of deallocation, and specifically for
Rc
/Arc
. The order of deallocating aBox<T>
callsdrop_in_place(*mut T)
first and thendeallocate(*mut T)
. ForBox
, since these are done linearly in a single function, it could just calculate theLayout::for_value(&T)
before dropping theT
. However,Rc
works differently because ofWeak
. When the lastRc
is dropped,drop_in_place(*mut T)
is called. Then, later when the lastWeak
is dropped,deallocate(*mut RcInner<T>)
is called. This means thatWeak
needs some way to get the size/align ofRcInner<T>
after theT
has been dropped. The simplest way is to allow getting size/align from a droppedT
, but there are alternatives[7:1]. ↩︎ -
When the author previously talked to T-lang about these requirements on size/align (w.r.t.
Weak::as_ptr
and friends) there was weak consensus that requiring statically known alignment and retrieving size from droppedT
was a reasonable restriction. However, alternative libs designs were discussed such that this guarantee was not finalized yet:Rc
could store a pointer directly toT
and use reverse offsets for the refcount to avoid needing to know the alignment statically, and dropping theT
inRcInner<T>
could overwrite it withLayout::for_value(&T)
such thatWeak
can just read that onDrop
, or additional fields could be added to the header data to store size/align as required. ↩︎ ↩︎ -
The need for this approach is limited in pure-Rust applications, because Rust is statically linked by default and does not consider changing the size of a struct as a breaking change, unlike in C where changing the size of a public struct is an ABI breaking change that can cause silent UB when upgrading dynamic libraries without recompiling the world. However, Rust is still used for FFI where this is a concern and should be able to interoperate with C libraries following this pattern. ↩︎