# Pre-RFC: mem::trailing\_padding!

**URL:** <https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681>\
**Category:** Uncategorized\
**Created:** [October 10, 2024, 2:41pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681 "2024-10-10T14:41:04Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![binarycat](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/binarycat/32/12210_2.png) [@binarycat](https://internals.rust-lang.org/u/binarycat)\
**Post date:** [October 10, 2024, 2:41pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/1 "2024-10-10T14:41:04Z")

</div>

# Problem statement

It is quite common in rust to have an array of enums that is iterated over. It is also quite common for some variants of that enum to be much larger than others. Implemented trivially, this will result in quite a lot of wasted space for padding. This can be solved by defining a full bytecode syntax, but this requires implementing a bunch of encoding and decoding logic, which wastes CPU cycles and introduces opprotunities for programmer error.

# Guide level explanation

Add a `trailing_padding!` macro to `core::mem`. This macro would take the name of a type or enum variant, and evaluate to an integer literal.

This literal would be the number of trailing "padding" bytes in that enum variant.

Padding bytes can have any value, so this would allow the creation of a crate that implements a datastructure that carefully overlays the padding at the end of one enum with the start of another. Because padding can have any value, and because this datastructure would be append-only, this would be sound. Combined with `#[repr(C)]`, this would essentially allow easily creating an opcode prefixed bytecode representation just through an `enum` and a derive macro.

# Alternatives

A const function that accepts a `mem::Discriminant` and returns the number of trailing padding bytes.

---

<div class="post-metadata">

**Author:** ![scottmcm](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/scottmcm/32/2355_2.png) [@scottmcm](https://internals.rust-lang.org/u/scottmcm)\
**Post date:** [October 10, 2024, 3:06pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/2 "2024-10-10T15:06:21Z")

</div>

Reminds me of [https://lang-team.rust-lang.org/frequently-requested-changes.html#size--stride](https://lang-team.rust-lang.org/frequently-requested-changes.html#size--stride), where I hypothesized something like this that would give a `Range<usize>` for a type or an instance saying which bytes are _sufficient_ to copy, with the trivial `0..sizeof(T)` implementation always being legal if a compiler wants.

---

<div class="post-metadata">

**Author:** ![binarycat](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/binarycat/32/12210_2.png) [@binarycat](https://internals.rust-lang.org/u/binarycat)\
**Post date:** [October 10, 2024, 3:08pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/3 "2024-10-10T15:08:23Z")

</div>

Importantly, this change would _not_ effect how `Vec` or any other pre-existing type functions.

Another alternative would be full layout reflection, or at least saying where all the padding bytes are.

---

<div class="post-metadata">

**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\
**Post date:** [October 10, 2024, 5:47pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/4 "2024-10-10T17:47:46Z")

</div>

You could almost implement this today in a derive macro by looking at the [offsets](https://doc.rust-lang.org/std/mem/macro.offset_of.html) and sizes of all fields and finding the last occupied byte. The catch is that you can't ask where the enum discriminant is stored (or any other non-field repr complications that might be introduced in the future), but that's well-defined for `#[repr(C)]` and `#[repr(uN)]` enums. That would be sufficient to write a prototype of this feature without modifying the compiler.

---

<div class="post-metadata">

**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\
**Post date:** [October 10, 2024, 6:34pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/5 "2024-10-10T18:34:18Z")

</div>

If you compact enums like that, won't this ruin alignment? You won't be able to `repr(C)`-read any except the first one?

---

<div class="post-metadata">

**Author:** ![CAD97](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/cad97/32/3460_2.png) [@CAD97](https://internals.rust-lang.org/u/CAD97)\
**Post date:** [October 10, 2024, 6:53pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/6 "2024-10-10T18:53:03Z")

</div>

Important thing to be careful of: if a tail field is not [`Freeze`](https://doc.rust-lang.org/nightly/std/marker/trait.Freeze.html) (is shared mutable; an `UnsafeCell`) then some of the trailing padding may be writable from safe code, and any calculations need to be careful to consider that.

Whether padding is valid to write to with `unsafe` doesn't matter since it's still unsound to write through `&T` not via the `UnsafeCell` API even if it happens to be generally valid.\[1\]

> [@binarycat](#):
>
> This can be solved by defining a full bytecode syntax, but this requires implementing a bunch of encoding and decoding logic, which wastes CPU cycles and introduces opportunities for programmer error.

Note that it's possible to use `repr(C)` and/or `repr(uN)` to create _a_ bytecode which can load safe enums with `ptr::read_unaligned`, although it might be somewhat less optimal than a manual bytecode definition. The RFC text defining these layouts: [https://rust-lang.github.io/rfcs/2195-really-tagged-unions.html](https://rust-lang.github.io/rfcs/2195-really-tagged-unions.html)

> [@binarycat](#):
>
> Add a `trailing_padding!` macro to `core::mem`.

Ideally this would be a function like `mem::size_of`, but that would require “enum variants as types” to be utilized as intended by the target problem space.

> [@binarycat](#):
>
> [Alternatively:] A const function that accepts a `mem::Discriminant` and returns the number of trailing padding bytes.

Or similarly, an “`*_of_val`” function. The limitation of course being that such needs an instance of the variant to get the trailing padding, just like currently you need to get a `Discriminant` value.

There's no real _requirement_ for this to be a macro like there is for `offset_of!`, so I'd personally prefer a solution which doesn't use a macro in the stdlib. A crate can of course provide a wrapping macro that sticks the fn in a `const{}` block\[2\].

> [@binarycat](#):
>
> Combined with `#[repr(C)]`,

Given the definition is as-if defined with `struct` and `union`, a proc-macro with access to the enum definition (e.g. a `derive`) could accomplish the goal with just a little bit of `Layout` math on said as-if types.

> [@kpreid](#):
>
> You could almost implement this today in a derive macro by looking at the [offsets](https://doc.rust-lang.org/std/mem/macro.offset_of.html) and sizes of all fields and finding the last occupied byte.

I didn't even think of that possibility, which gives a bit tighter of a bound than just using the size of the variant payload as-if types.

> [@binarycat](#):
>
> Another alternative would be full layout reflection, or at least saying where all the padding bytes are.

You can almost get this today with just `offset_of!` and `Layout` of the fields' types, if you know the field name/types (i.e. `#[derive]`). The macro auto(de)ref specialization tricks I'm conceptualizing to get a nice ergonomic behavior are honestly kinda frightening.

> [@binarycat](#):
>
> Importantly, this change would _not_ effect how `Vec` or any other pre-existing type functions.

Technically, that is all entirely private unstable impl details, so it doesn't matter whether they change their impl strategy or not. They should do whatever has the most consistent/predictable “zero overhead” performance characteristics.

> [@kornel](#):
>
> If you compact enums like that, won't this ruin alignment?

`read_unaligned` or `repr(packed)` solve that (although I don't think we actually define a `repr(C, packed)` for enums), or if the variants are sufficiently different in size (given the desire to overlap, they likely are), it's possible to align the next instance properly while still starting in the padding of the prior.

* * *

1. This is the stated position of T-opsem, and the used justification for immutable static constant promotion of `UnsafeCell`-free enum variants even when the full type is not `Freeze` and potential opsem would make writes valid when not promoted to immutable memory. (And a trivial proof/example: I can retag memory as a `Freeze` array of bytes and then refcast it back to the type in question, and this should be considered sound if the reachable safe API cannot cause a write to the now immutable memory.) 

2. And because of various impl details, doing so can actually be better for compile time, since it reduces the amount of MIR cost to the function.

---

<div class="post-metadata">

**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\
**Post date:** [October 10, 2024, 7:10pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/7 "2024-10-10T19:10:59Z")

</div>

> [@CAD97](#):
>
> Important thing to be careful of: if a tail field is not [`Freeze`](https://doc.rust-lang.org/nightly/std/marker/trait.Freeze.html) (is shared mutable; an `UnsafeCell`) then some of the trailing padding may be writable from safe code, and any calculations need to be careful to consider that.

This isn't a problem unless you are

1. using a “deep, not shallow” definition of trailing padding that looks into field types rather than treating them as opaque except for ordinary `size_of`, _and_
2. looking through `UnsafeCell<T>` to the padding of `T`.

I'd say this operation shouldn’t do (2), in exactly the same way looking for niches doesn’t, and _possibly_ shouldn't do (1) either.

---

<div class="post-metadata">

**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\
**Post date:** [October 10, 2024, 7:18pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/8 "2024-10-10T19:18:03Z")

</div>

> [@CAD97](#):
>
> > [@binarycat](#):
> >
> > Importantly, this change would _not_ effect how `Vec` or any other pre-existing type functions.
> 
> Technically, that is all entirely private unstable impl details, so it doesn't matter whether they change their impl strategy or not. They should do whatever has the most consistent/predictable “zero overhead” performance characteristics.

And so `Vec` (and even more so, slices) could not do this, because it is expected to offer O(1) random read-write access which depends on uniformly-sized elements. Collections like `BTreeMap` could try, but they’d have to expand out to full-size nodes on any `get_mut()` or `iter_mut()` (because you can never hand out a `&mut` reference to these padding-overlapped enums without potentially smashing following elements). In general, `std` doesn’t deal in append-only or “immutable” collections, so there aren’t many opportunities to use this kind of layout in `std`.

---

<div class="post-metadata">

**Author:** ![binarycat](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/binarycat/32/12210_2.png) [@binarycat](https://internals.rust-lang.org/u/binarycat)\
**Post date:** [October 10, 2024, 7:20pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/9 "2024-10-10T19:20:38Z")

</div>

> [@CAD97](#):
>
> Or similarly, an “`*_of_val`” function. The limitation of course being that such needs an instance of the variant to get the trailing padding, just like currently you need to get a `Discriminant` value.
> 
> There's no real _requirement_ for this to be a macro like there is for `offset_of!`, so I'd personally prefer a solution which doesn't use a macro in the stdlib. A crate can of course provide a wrapping macro that sticks the fn in a `const{}` block[[2]](#footnote-138797-2)

the reason i prefer a macro solution is because of the aforementioned limitation of needing an instance of the enum. i want this to be useable in derive macros, and a function-based solution would struggle, expecially if the enum variant contained runtime-only values.

of course, another alternative would be first-class curried enum variants. it's already possible to instantiate values that are a variant that is missing it's value, but only with tuple enums:

```rust
enum En {
    Um(u8)
}

fn main() {
    let _x = En::Um;
}

```

---

<div class="post-metadata">

**Author:** ![quinedot](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/quinedot/32/7294_2.png) [@quinedot](https://internals.rust-lang.org/u/quinedot)\
**Post date:** [October 10, 2024, 7:26pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/10 "2024-10-10T19:26:27Z")

</div>

> [@binarycat](#):
>
> it's already possible to instantiate values that are a variant that is missing it's value, but only with tuple enums

That's a function item or so, not a variant value. You can `let _y = |b| En::Em { b };` too.

---

<div class="post-metadata">

**Author:** ![pitaj](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/pitaj/32/11262_2.png) [@pitaj](https://internals.rust-lang.org/u/pitaj)\
**Post date:** [October 10, 2024, 7:27pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/11 "2024-10-10T19:27:00Z")

</div>

`offset_of` is a macro, so I don't see why this wouldn't be as well.

---

<div class="post-metadata">

**Author:** ![binarycat](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/binarycat/32/12210_2.png) [@binarycat](https://internals.rust-lang.org/u/binarycat)\
**Post date:** [October 10, 2024, 7:28pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/12 "2024-10-10T19:28:59Z")

</div>

it's an opaque type that implements `Fn`. i don't know a ton about the compiler, but theoretically it could implement other traits too.

---

<div class="post-metadata">

**Author:** ![kornel](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kornel/32/2711_2.png) [@kornel](https://internals.rust-lang.org/u/kornel)\
**Post date:** [October 10, 2024, 7:36pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/13 "2024-10-10T19:36:54Z")

</div>

`let _x: En = En::Um;` doesn't compile, and the error says says it's `fn(u8) -> En`.

---

<div class="post-metadata">

**Author:** ![kpreid](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/kpreid/32/8484_2.png) [@kpreid](https://internals.rust-lang.org/u/kpreid)\
**Post date:** [October 10, 2024, 8:15pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/14 "2024-10-10T20:15:47Z")

</div>

The type error should be saying not `fn(u8) -> En` but `fn(u8) -> En {En::Um}`. This is the (not nameable in surface syntax) [_function item type_](https://doc.rust-lang.org/reference/types/function-item.html) of the variant constructor. Every `fn foo(){}` item, and every tuple-struct or tuple-variant constructor, has a unique function item type.\[1\]

There is no type-system obstacle to deciding that constructor function item types should implement another trait that gives information about the variant (perhaps via returning `mem::Discriminant`?).

* * *

1. One reason why such unique types exist is so that they can be zero-sized types, so that when functions are passed to generic code and stored in generic data structures, the function takes up no space and is easily inlinable after monomorphization. Function pointers can also sometimes be bypassed by LLVM optimization of code, but not removed from struct layout.

---

<div class="post-metadata">

**Author:** ![steffahn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/steffahn/32/13288_2.png) [@steffahn](https://internals.rust-lang.org/u/steffahn)\
**Post date:** [April 3, 2026, 8:15pm UTC](https://internals.rust-lang.org/t/pre-rfc-mem-trailing-padding/21681/15 "2026-04-03T20:15:49Z")

</div>

This topic was automatically closed 540 days after the last reply. New replies are no longer allowed.