# \[Pre-RFC\] Add a new offset\_of macro to core::mem

**URL:** <https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273>\
**Category:** Uncategorized\
**Created:** [January 23, 2019, 5:04am UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273 "2019-01-23T05:04:46Z")\
**Posts on this page:** 15\
**Page:** 7

<div class="post-metadata">

**Author:** ![197g](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/197g/32/7276_2.png) [@197g](https://internals.rust-lang.org/u/197g)\
**Post date:** [July 9, 2019, 11:52pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/121 "2019-07-09T23:52:27Z")

</div>

> [@RustyYato](#):
>
> This wasn’t a major reason for writing up `Generic pointer to field` , but that is a good reason to bake it into the compiler. Only one issue, a user-land `Project` trait would make it impossible for the compiler to reason about the disjointness of the (smart/raw) pointers returned by `project` so this is a moot point. Basically you would have to special case every smart pointer that you want to allow disjoint borrows from.

I meant a reason for the other RFC, `Allow fields in traits that map to lvalues in impl'ing type´. Ideas on disjoint borrowing on custom pointer types (if required at all) fit better in a different thread.

> [@Dante-Broggi](#):
>
> Via the [`MemoryLayout`](https://swiftdoc.org/v4.2/type/memorylayout/) stdlib type, Swift provides the function `offset(of key: PartialKeyPath<T>) -> Int?` which provides access to the byte offset from a pointer to `T` to the property referenced by the keypath, or `.none` if the property is not stored trivially (e.g. computed, indirect, or overlaps other properties).

I actually find Swift's solution interesting (kind of like it, but would need to try it out for a verdict) but there are some crucial details. In comparison

- The input of `offset` is some well-typed object (a `KeyPath` from what I could gather) and not a string. It even has [a unique expression type](https://docs.swift.org/swift-book/ReferenceManual/Expressions.html#ID563) for its creation. It overall feels similar to a member pointer (less strictly typed than the current Rust proposal, not every member has a unique type).

- Secondly, `offset` is a method of a generic other type which captures the base type in a type parameter and not an ad-hoc intrinsic or macro. This suggests `MemoryLayout` and `KeyPath` are the core primitives, not `offset`.

- It seems that you are not supposed or at least discouraged to use the integer result of `offset` for manual computation of say pointers to members. Rather, you can do so type-safely with the `KeyPath` directly:

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [July 10, 2019, 5:26pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/122 "2019-07-10T17:26:46Z")

</div>

> [@CAD97](#):
>
> This could be exposed as either `offset_of<T>(const &str)` via magic, `offset_of<T, const &str>()` , or even only available as `offset_of!($:type, $:ident)` . There’s no intent to use a dynamic string for runtime reflection with this intrinsic.

I would very much prefer `offset_of!($:type, $:ident)`, which would allow writing `offset_of!(Foo, bar)` without quoting `bar`.

That said, we _might_ want to allow a subset of expression syntax rather than just an ident, to allow `offset_of!(Foo, bar.baz)` or `offset_of!(Foo, bar.baz[3].fnord)`. Debatable, but people _do_ use the C offsetof that way.

---

<div class="post-metadata">

**Author:** ![RustyYato](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/rustyyato/32/13627_2.png) [@RustyYato](https://internals.rust-lang.org/u/RustyYato)\
**Post date:** [July 10, 2019, 6:53pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/123 "2019-07-10T18:53:10Z")

</div>

I think as a first step, we should just allow fields, and we can expand to more general expressions later

---

<div class="post-metadata">

**Author:** ![Tom-Phinney](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/tom-phinney/32/3299_2.png) [@Tom-Phinney](https://internals.rust-lang.org/u/Tom-Phinney)\
**Post date:** [July 10, 2019, 6:54pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/124 "2019-07-10T18:54:35Z")

</div>

> [@josh](#):
>
> we _might_ want to allow a subset of expression syntax rather than just an ident, to allow `offset_of!(Foo, bar.baz)` or `offset_of!(Foo, bar.baz[3].fnord)` . Debatable, but people _do_ use the C offsetof that way.

I would prefer that generality, as it seems more likely to be forward-compatible with disjoint borrows. However I agree with @RustyYato that it need not be the first step, as long as the initial approach does not foreclose such finer resolution as a future extension.

---

<div class="post-metadata">

**Author:** ![josh](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/josh/32/5934_2.png) [@josh](https://internals.rust-lang.org/u/josh)\
**Post date:** [July 10, 2019, 7:04pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/125 "2019-07-10T19:04:03Z")

</div>

Sounds reasonable to me.

---

<div class="post-metadata">

**Author:** ![spunit262](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/spunit262/32/5677_2.png) [@spunit262](https://internals.rust-lang.org/u/spunit262)\
**Post date:** [July 10, 2019, 8:42pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/126 "2019-07-10T20:42:01Z")

</div>

AFAIK, disjointedness only matters with references and accesses, so any API that allows the projection to be done purely as math on raw pointers until the wanted borrow is crated, should be fully capable of allowing disjointed borrows.

---

<div class="post-metadata">

**Author:** ![RalfJung](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ralfjung/32/2415_2.png) [@RalfJung](https://internals.rust-lang.org/u/RalfJung)\
**Post date:** [July 12, 2019, 8:41am UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/127 "2019-07-12T08:41:07Z")

</div>

> [@197g](#):
>
> There is UB _within_ the `offset_of` macro crates, the most popular determines the offset subtracts pointers into uninitialized memory, and relies on pointer projection to fields of struct-pointers. But **the main UB here (invalid references) needs to be fixed regardless of `offset_of`** since it is also used for initializing structs. If it is fixed, the macro could be written **without UB** (at least @RalfJung made it sound like it could) and not require any intrinsic.

Correct. A basic `offset_of` for fields could be implemented UB-free with a solution for [RFC for an operator to take a raw reference by RalfJung · Pull Request #2582 · rust-lang/rfcs · GitHub](https://github.com/rust-lang/rfcs/pull/2582).

However...

> [@RustyYato](#):
>
> ```rust
> macro_rules! offset_of {
> ($parent:ty, $field:ident) => {{
> let $parent { $field: _, .. }; // protection against deref-coercion
> let ptr = 0_usize as *const $parent;
> (&raw ptr.$field) as usize
> }}
> }
> 
> ```

...this _does have UB_ even with the RFC! `ptr.field` perform an _inbounds_ pointer offset operation, but your pointer is not inbounds of any allocation.

I would like this not to be UB, but there are concerns that making raw pointer field offset a safe operation would lose a lot of optimization potential. Likely, making it UB only on overflow (as opposed to the more restrictive "inbounds") would be sufficient to mitigate that, but LLVM does not offer that option currently.

So, until then, the UB-free way to do `offset_of` with `&raw` is:

```rust
macro_rules! offset_of {
    ($parent:tt, $field:tt) => {{
        // Make sure the field actually exists. This line ensures that a
        // compile-time error is generated if $field is accessed through a
        // Deref impl.
        let $parent { $field: _, .. };

        // Create an instance of the container and calculate the offset to its field.
        // Here we're using an uninitialized instance of $parent. We avoid UB
        // by only using raw pointers that point to real (allocated, albeit uninitialized) memory.
        let val = $crate::mem::MaybeUninit::<$parent>::uninit();
        let base_ptr = val.as_ptr();
        #[allow(unused_unsafe)] // for when the macro is used in an unsafe block
        let field_ptr = unsafe { &raw (*base_ptr).$field };
        let offset = (field_ptr as usize) - (base_ptr as usize);
        offset
    }};
}

```

---

<div class="post-metadata">

**Author:** ![RalfJung](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ralfjung/32/2415_2.png) [@RalfJung](https://internals.rust-lang.org/u/RalfJung)\
**Post date:** [July 21, 2019, 6:05pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/128 "2019-07-21T18:05:38Z")

</div>

> [@RalfJung](#):
>
> …this _does have UB_ even with the RFC! `ptr.field` perform an _inbounds_ pointer offset operation, but your pointer is not inbounds of any allocation.

Follow-up: the actual reason this has UB is that [dereferencing dangling pointers is UB](https://doc.rust-lang.org/nightly/nomicon/what-unsafe-does.html). So It's the `*ptr` in your code that already causes UB.

At some point I want to pursue relaxing this for raw pointers, but there are more pressing matters. 😉

---

<div class="post-metadata">

**Author:** ![mjbshaw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mjbshaw/32/5103_2.png) [@mjbshaw](https://internals.rust-lang.org/u/mjbshaw)\
**Post date:** [November 3, 2019, 5:17pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/129 "2019-11-03T17:17:57Z")

</div>

Now that [`offset_from` is a `const fn` (behind a feature flag)](https://github.com/rust-lang/rust/pull/63810), this can be implemented as a const-friendly macro (I haven't tested this since I nightly hasn't yet been updated with `const_ptr_offset_from`; I may revise this in the next day or two when I get a chance to actually test it):

```rust
#![feature(const_transmute)]
#![feature(const_ptr_offset_from)]
#![feature(ptr_offset_from)]

macro_rules! offset_of {
    ($Struct:path, $($field:tt)+) => ({
        const OFFSET: usize = {
            extern crate core;

            let base_uninit = core::mem::MaybeUninit::<$Struct>::uninit();
            unsafe {
                // This is UB, but it's the best we can do for the time being.
                // If this is in `core`, we can just leave a comment to tell
                // people that this is an unsafe implementation detail that
                // works here in `core` but is UB outside of this macro.
                let base_ref = core::mem::transmute::<_, &$Struct>(&base_uninit);
                let base_u8_ptr = base_ref as *const _ as *const u8;

                let field_u8_ptr = &base_ref.$($field)+ as *const _ as *const u8;
                let offset = field_u8_ptr.offset_from(base_u8_ptr) as usize;
                
                // Make sure the offset computation stayed within the struct's size.
                let assert = [offset; 1];
                assert[(offset <= core::mem::size_of::<$Struct>()) as usize - 1]
            }
        };
        OFFSET
    })
}

```

Using `$($field:tt)+` allows this to support `offset_of!(Struct, field.sub_field.sub_array[3])`.

---

<div class="post-metadata">

**Author:** ![RalfJung](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ralfjung/32/2415_2.png) [@RalfJung](https://internals.rust-lang.org/u/RalfJung)\
**Post date:** [November 3, 2019, 5:47pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/130 "2019-11-03T17:47:03Z")

</div>

> [@mjbshaw](#):
>
> Using `$($field:tt)+` allows this to support `offset_of!(Struct, field.sub_field.sub_array[3])` .

You had to drop the deref protection for this, though.

If a deref coercion actually ends up in a different allocation, `offset_from` will detect that -- you crucially rely on this computation happening at compile-time though, so there should be a comment for that.

But it is also conceivable that a deref coercion stays inside the same struct... and I am curious how weird that can get? I first thought this could be impure, but since you are running this at const-time we'd actually get an error if the deref code wasn't const-compatible. Probably right now being in a const context entirely protects you against deref coercions as those are non-`const fn`-calls; that check will get relaxed eventually though.

---

<div class="post-metadata">

**Author:** ![mjbshaw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mjbshaw/32/5103_2.png) [@mjbshaw](https://internals.rust-lang.org/u/mjbshaw)\
**Post date:** [November 3, 2019, 5:50pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/131 "2019-11-03T17:50:59Z")

</div>

Yeah, I thought about that. But the deref protection syntax isn't compatible with tuple structs, so I consider it non-viable unfortunately.

---

<div class="post-metadata">

**Author:** ![RalfJung](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/ralfjung/32/2415_2.png) [@RalfJung](https://internals.rust-lang.org/u/RalfJung)\
**Post date:** [November 3, 2019, 6:04pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/132 "2019-11-03T18:04:28Z")

</div>

Then you should mark the macro as `unsafe` though.

---

<div class="post-metadata">

**Author:** ![Amanieu](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/amanieu/32/4095_2.png) [@Amanieu](https://internals.rust-lang.org/u/Amanieu)\
**Post date:** [November 3, 2019, 8:27pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/133 "2019-11-03T20:27:51Z")

</div>

> [@mjbshaw](#):
>
> Yeah, I thought about that. But the deref protection syntax isn't compatible with tuple structs, so I consider it non-viable unfortunately.

Actually it works [just fine](https://github.com/Gilnaa/memoffset/blob/59ed7dd178aebd45515310f9dbee9ee7fb1fef6e/src/offset_of.rs#L129) with tuple structs.

---

<div class="post-metadata">

**Author:** ![mjbshaw](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/mjbshaw/32/5103_2.png) [@mjbshaw](https://internals.rust-lang.org/u/mjbshaw)\
**Post date:** [November 4, 2019, 12:04am UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/134 "2019-11-04T00:04:17Z")

</div>

Indeed it does. I must have screwed something up when I tried it earlier. I stand corrected. Thanks.

---

<div class="post-metadata">

**Author:** ![steffahn](https://sea2.discourse-cdn.com/flex002/user_avatar/internals.rust-lang.org/steffahn/32/13288_2.png) [@steffahn](https://internals.rust-lang.org/u/steffahn)\
**Post date:** [December 22, 2024, 6:48pm UTC](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273/135 "2024-12-22T18:48:08Z")

</div>

This topic was automatically closed 540 days after the last reply. New replies are no longer allowed.

[Previous page](https://internals.rust-lang.org/t/pre-rfc-add-a-new-offset-of-macro-to-core-mem/9273.md?page=6)