This recently came up in discussion and I'd love some clarity on it. I'm talking specifically about code like this:
let ref: &'a T = /* .. */;
let static_ref: &'static T = core::mem::transmute(ref);
Clearly this is illegal if static_ref is ever accessed after the target object is dropped. What I'm asking about is whether that line in and of itself is UB. I also could have used any lifetime 'b: 'a, but let's stick to 'static since it seems to come up most often and is easier to reason about.
My initial thoughts were that this is UB because:
Lifetime is part of a type, therefore a 'static reference value is only "valid" if the pointed-to object actually lives long enough for 'static to apply. Therefore this code is in violation of the "values must be valid" rule.
I've never seen any documentation stating that this is defined behaviour, so this makes it "undefined" by definition.
I'd love some clarity on this. It also appears to me that any time you'd want to do this in a sound way, storing a raw pointer instead should be possible and avoids lifetime issues as they don't have lifetimes as part of the type.
As far as I know (and as far as multiple crates that use such approaches[1] interpret it), lifetime parameters are irrelevant, and creating a reference with a lifetime parameter thatās incorrect does mean youāre giving up protection from the borrow-checker, but itās not UB in and by itself. Also AFAIR, miri seems to agree with this interpretation.
Note that Iām not saying you can use the &'static T as long as you just make sure that itās never accessed after the target object is dropped. What you can do safely is use it for as long as the original reference would have been usable. For anything beyond, youād need to consider all the ways in which references can become invalidated. E.g. if you mutably access[2] the target object, your immutable reference immediately becomes invalid, so using the &'static T afterwards would be UB. If you mutate or move the owner of your object (e.g. move the Box<T> that's holding the T) then you might even be in a gray zone where itās not fully decided yet whether your &'static T (from a transmuted deref [&T] of the Box) get invalidated - the &Box<T> reference you would have started with certainly did get invalidated.
As far as I know (and as far as multiple crates that use such approaches interpret it), lifetime parameters are irrelevant
This is interesting, thank you. I know almost nothing about pointer provenance, but I was wondering if that would potentially have any bearing here.
My understanding is that any code doing this is unsound if it is possible for any safe code block to ever use the resulting reference once invalidated, which would be something to be exceptionally careful about. It would be nice if the nomicon could get a paragraph or two on this for clarity. My initial thoughts are that this shouldn't be on the docs for transmute because that might encourage more people to do a thing that feels very difficult to do correctly.
As a third point, rather than storing a &'a with a funky lifetime you've had to arm-wrestle the compiler into, wouldn't storing a raw pointer be better in every situation? Sure, on every safe interface you'd have to convert back to an appropriate reference, but at least you're doing so in an unsafe block that's encouraging you to think exactly about this validity problem.
Yes, raw pointers can be a nicer option. Note that all points still apply. A raw pointer *const T created from a &T reference will also be invalidate in the cases I listed above, like when the target is mutably accessed.
As a rule of thumb, shared + mutable access via raw pointers is possible, but then all the raw pointer usually have to be copies of the same original raw pointer (created originally e.g. from a &mut T reference). Another way to think of it is that copies of a raw pointer still are (in an object-identity kind of view) still the same pointer.
In the cases Iāve seen 'static be used like this e.g. yoke, itās used because raw pointers arenāt an option, since itās a more generic thing supporting all kinds of types with lifetime parameters, not just plain references. (In case you look at the code, thereās also an increasing amount thoughts/discussion about it in the comments relating to the āgray zoneā Iāve mentioned above of moving the owner (like a Box for instance) whilst keeping the reference alive, and workarounds like usage of MaybeUninit to improve the soundness, but this might be unrelated to your question.)
If you use raw pointers, note that those in Rust are, unfortunately, by default nullable, so you use some optimization opportunities like the null-pointer-optimization of Option for your structs holding a pointer, unless you use something like NonNull [which unfortunately ended up less ergonomic than the nullable pointer type] ā in case you do use NonNull and the pointer is considered mutable, keep variance in mind, perhaps even adding some appropriate PhantomData<ā¦> to restore invariance.
Another advantage that raw pointers have is that *const T or NonNull<T> donāt require a T: 'static bound (this is about whether the type T itself can yet-again contain (non-static) references), whereas &'static Tdoes come with this restriction.
This is turning a bit into a (albeit interesting) conversation about all of the variations of the question which are definitely not legal. I'm still curious if there's any official source on it being legal though (at least in some carefully managed situations).
In particular, if this doesn't fall afoul of the "no invalid values constructed" rule, it would be interesting to know exactly why. Is it because "invalid values" only refers to the bits and alignment of an object? Is the "lifetimes" part of the type system just excepted from this rule entirely?
For the sake of completeness, I should note that I did check this with MIRI and it doesn't blow up just by creating an invalid lifetime reference. This is evidence not proof though, as afaik MIRI is not considered complete in its ability to detect UB.
For the aliasing models we are currently considering, lifetimes do indeed not matter.
However, note that there is one caveat to this: when you pass a reference to a function, that reference must outlive the function. This does not follow from its lifetime, it follows from the fact that it appears in argument position. If you transmute a reference to 'static, pass that to a function, never use the reference again, but while the function runs you deallocate the memory the reference points to -- that is UB.
No. We don't yet make any guarantees about the reference aliasing model. If you want guarantees, use raw pointers.
The current model used by the compiler doesn't consider lifetimes as part of the definition of what is "valid" for a type. In this sense, creating a &'static which isn't actually valid for 'static is not instant UB in the same way that creating a bool with value 2 is.
However, that model is not guaranteed nor documented as a public "interface". The implication being that this could (in principle) change in the future or even with a different compiler implementation. So it is "undefined" in the sense of "not having a defined guarantee".
Then there are all of the ways in which using (or as you point out, even passing around) such a reference is definitely UB. I guess a kind of "defined UB" (in that UB is defined to be triggered, like it is with a bool of 2).
Yes that sounds pretty good. There's a large gray area here, from things that are almost certainly going to remain UB (like the example I mentioned) to things that are almost certainly going to be allowed (like transmuting the lifetime to 'static and then never using that reference ever again for anything). But it's all work-in-progress and there are no decisions by the lang team nor opsem team yet.
I'd view it from another angle:
The lifetime determines until when values of that type might exist because lifetimes only limit until when types stay valid. If you transmute a &'a into a &'static and only use it for 'a, the value itself never becomes invalid. You only get UB once it's freed or mutated while holding onto the reference.
That's similar to transmuting a &'a mut bool into a &'a mut u8. In itself that doesn't cause UB. The types have the same alignment and size and any value of bool is a valid bit pattern for u8. Only once you make the value invalid for bool it's UB:
let mut foo = true;
let bool_ref = &mut foo;
let u8_ref: &mut u8 = unsafe { core::mem::transmute(bool_ref) };
*u8_ref = 2; // invalid `bool` value created -> UB!
let value = Box::new(42);
let reference: &i32 = &value;
let static_ref: &'static i32 = unsafe { core::mem::transmute(reference) };
drop(value); // holding `static_ref` after this point makes the value invalid
dbg!(static_ref); // invalid `&'static` value created -> UB!
You can't even use it for 'a. In the context of normal UB, this is like saying a boolean value of 2 is fine if the optimizer decides to to use jump-if-not-zero instructions on it, but it's not fine if the optimizer does something different. I would have called this the optimizer "exploiting" UB; it's still UB even if the optimizer isn't doing bad things right now.
Normally it should be okay for the optimizer to reorder certain statements involving static references, e.g.
let value = Box::new(42);
let reference: &i32 = &value;
let static_ref: &'static i32 = unsafe { core::mem::transmute(reference) };
// It should be okay for the optimizer to reorder these two lines.
dbg!(static_ref); // ā static ref will always be valid, so we can dbg! it whenever
drop(value); // ā deallocation has no observable behavior and there are no uses of this
But obviously the optimizer can't reorder those lines or it "exploits" the UB. Or am I missing something else?
If we told the optimizer that the reference was live in perpetuity, yes that transform would be valid, and violating that assumption we told the optimizer to make would be UB, whether it's manifest or not.
The terminology I try to use is that if your program does an invalid operation, that causes that execution to be UB, and that UB is either manifest (things go wrong) or nonmanifest (you got lucky and everything happens in a reasonable interpretation of what would happen). The most common kind of nonmanifest UB is any operation which is invalid according to the Rust AM but rustc happens to currently lower the code to valid LLVM IR which is free of UB at that abstraction level.
However, we do at least imply that this is not a valid transformation, and it isn't a language requirement that &'staticactually remain valid until the end of time. Namely, the documentation for fn Box::leak<'a>(Self) -> &'a mut T says that
This function is mainly useful for data that lives for the remainder of the programās life. Dropping the returned reference will cause a memory leak. If this is not acceptable, the reference should first be wrapped with the Box::from_raw function producing a Box. This Box can then be dropped which will properly destroy T and release the allocated memory.
which implies that Box::from_raw(Box::leak::<'static>(it)) is sound. And in fact it is valid per SB and TB as implemented by Miri, though doing this does have some thorny edges due to function argument protectors. So I still recommend you don't use Box::leak but use Box::into_raw instead and work with the raw pointer if you're going to unleak the box.
This makes me uncomfortable, because isn't it the exact opposite of what lifetimes are there to do? The reason lifetimes work in (safe) Rust is because the lifetime annotates a value of a type at a particular location definitely does exist (though they may exist longer) *.
Doesn't this also fall into the exact "defined UB" vs "undefined UB" dichotomy from above? i.e. neither documented and defined to trigger UB nor with a documented definition to have any particular guaranteed behaviour (thus, "undefined behaviour" in a literal sense).
* If there is an exception to this, I suppose it is very occasionally 'static. But even then, the "usual" use of 'static is for things that live until the end of the Rust program (and are then cleaned up by the Rust runtime, OS, or just not at all). The cases I'm thinking of where this isn't the case are all examples of where a lifetime is transmuted to 'static for erasure purposes, as discussed in this thread.
This is interesting, but still far from the kind of documented guarantee that would make me as a user of the language comfortable that my code transmuting &'a to &'static is guaranteed to be (and remain) sound (except in the precise case documented in Box::leak).
I guess that's where I'm coming down to on this in general. If, as a user of the language, I don't have documentation to point to to prove that my unsafe code is sound and has stable defined behaviour, then it is (in a very literal sense) undefined behaviour. Therefore I should not do it if at all possible. If I need to do it in a narrow use-case, I should ask if a documented guarantee could be provided that covers that use-case.
Clearly this is not as bad as "defined UB" (such as dereferencing and invalid pointer, or constructing a value with a bad bit pattern). "Manifest UB", as you put it, clearly is worse still.
Maybe I should start a discussion/blog post/something on a taxonomy of undefined behaviour.
That's what follows from the lifetime rules, when you're talking about the value behind the reference. I talked about the reference as a value itself:
A reference type &'a T is only valid for 'a. That means that the reference (as a value) can at most live for 'a but doesn't need to exist for that long.
If you don't transmute lifetimes, then this enforces exactly what you described. If the referenced value lives for 'a, then only references of type &'a T or &'b T with 'a: 'b can be created referencing it. And those references cannot outlive 'a, so conversely the referenced value must exist at least for 'a.
I don't think so. Bools are documented to use the bit patterns 0x00 and 0x01 for false and true. The layout of a reference to a sized type is documented:
Pointers and references have the same layout. Mutability of the pointer or reference does not change the layout.
Pointers to sized types have the same size and alignment as usize.
And transmute has these safety conditions:
Both types must have the same size. Compilation will fail if this is not guaranteed.
transmute is semantically equivalent to a bitwise move of one type into another. [ā¦]
Both the argument and the result must be valid at their given type.
All of this holds when transmuting a &mut bool to a &mut u8 of the same lifetime. Only once you write a value through the reference that is not a valid bool you cause UB. The transmute itself is defined but wildly unsafe and allows to cause the UB in the first place.