This is UB: accessing a field asserts that the old and new pointer (computing the offset) are in-bounds of the same allocation. Your pointer is not in-bounds of any allocation.
There are some tricks to avoid Deref, like here and here.
Deref