`Pin` from `&mut` refs?

[zrk]

So I’ve been testing Pin on beta, and I’ve got some questions.

1. From my comprehension when reading the doc, a Pin<T> is a pointer type that offers the guarantee that its pointed value will not move as long as the Pin exists, unless “it doesn’t matter” that the pointed value moves (T implements Unpin. I’ll ignore Unpin T for the rest of this message). Cases where it does matter are typically self-referential structs that are in a way observing their own address. Pin<T> comes basically as a solution to the fact that &mut T is slightly too powerful to guarantee that an object cannot be moved: &mut T allows to swap the data of two references, possibly changing their address, hence resulting on a “move” of the pointed-to objects. Pin<T> aims to “behave like &mut T where possible”, and to restrict its powers to hold its guarantee. Is that clunky explanation correct?

2. Still in my comprehension, Pin<T> has two critical “unsafe points” in its usage: the first is when it is created from a pointer (Pin::new_unchecked), where the caller must ensure that the pointed-to object is in fact “pinned”, ie won’t be moved while Pin<T> exists. The second is when a Pin<T> is promoted into a &mut T (Pin::get_unchecked_mut()), giving it all the required power to misbehave with e.g. mem::replace or mem::swap. The safety of that second point relies on the caller promising not to misuse the obtained &mut T. Is this explanation still correct?

3. If 1. and 2. hold, then I believe we could add a safe Pin::new(&mut T) -> Pin<&mut T> safe constructor for Pin<T>, even for !Unpin T. The rationale for this is that it is always safe to call the existing Pin::new_unchecked(&mut T) function, because rules of aliasing ensure we only have that one &mut T to access T at this point, that &mut T is borrowed by the function for the entire duration of the Pin<&mut T>, and in that function we are obviously not moving out of T. See a code example in Playground. To my knowledge there is no such function. So what am I missing?

• Is that function somehow available and I missed it? I skimmed the doc for Pin and &mut ref, but there are many impl block with various bounds so maybe I just missed something?
• Am I misunderstanding something about Pin<T> that makes my proposed constructor unsound with regards to the guarantees provided by Pin?
• Any other reason for that function to be missing?

Otherwise, I find the Pin API to be a great step forward regarding unmovable/autoreferential structs, if only a bit hard to understand, wrt its guarantees.

Bonus question: If my understanding is correct, the example in the pin module documentation is unsound just if we remove the PhantomPinned member. While OK because of its use of unsafe, I find a bit dangerous that we have no way to mark autoreferential structs as !Unpin automatically. Would there be a way to do this?

[Yato]

The second is when a Pin<T> is promoted into a &mut T ( Pin::get_mut() ), giving it all the required power to misbehave with e.g. mem::replace or mem::swap

I know that you said that you are going to ignore Unpin, but it is critical to understand that because of the promises made by Unpin, Pin::get_mut is safe. You can only use Pin::get_mut on types that implement Unpin, which means that they don’t have any self referential shenanigans that would make it unsafe to move them around.

If 1. and 2. hold, then I believe we could add a safe Pin::new(&mut T) -> Pin<&mut T> safe constructor for Pin<T> , even for !Unpin T . The rationale for this is that it is always safe to call the existing Pin::new_unchecked(&mut T) function, because rules of aliasing ensure we only have that one &mut T to access T at this point, that &mut T is borrowed by the function for the entire duration of the Pin<&mut T> , and in that function we are obviously not moving out of T . See a code example in Playground. To my knowledge there is no such function. So what am I missing?

There is already a Pin::new, but it requires that <P as Deref>::Target: Unpin. I am not sure why it requires Unpin though.

[CAD97]

There are two problems with your understanding.

First, when discussing Pin, it should always be Pin<P>. This is because P is not the pinned value, the T which P points to is the pinned value. That is, Pin itself is not a smart pointer, but rather Pin is a smart pointer wrapper. You probably will never have a Pin<T> in useful code, and will see a lot of Pin<&mut T> around potentially !Unpin operations.

Second and more importantly, you have Pin's guarantee subtly wrong. Pin's guarantee isn’t that T isn’t moved while the Pin exists, but that T will never be moved once the Pin is constructed (unless it doesn’t matter and thus T: Unpin).

It’s for this reason that new_unchecked is unsafe. You have to guarantee that the T will never be safe to move (in code beyond the “unsafety scope”) from here on out, forever. (unsafe code that still knows what it’s doing to a concrete T is theoretically allowed to move the value, I suppose.)

As for the doc example, it’s a fact that for now, self referential structures are a niche, complicated subject that requires unsafe care to create, and the default assumption is that everything is still trivially movable.

[zrk]

I fixed my playground link (was a link to last saved playground instead of my specific playground).

I know that you said that you are going to ignore Unpin , but it is critical to understand that because of the promises made by Unpin , Pin::get_mut is safe. You can only use Pin::get_mut on types that implement Unpin , which means that they don’t have any self referential shenanigans that would make it unsafe to move them around.

I meant Pin::get_unchecked_mut but got it wrong. I fixed that in my post.

There is already a Pin::new , but it requires that <P as Deref>::Target: Unpin . I am not sure why it requires Unpin though.

CAD97 explains this in their answer.

First, when discussing Pin , it should always be Pin<P>.

Agreed. I did understand the difference between the pointer P that Pin wraps and the "pointed to value T" (aka <P as Deref>::Target in the docs), but that got lost in my explanations. I can see how this makes these explanations confusing, because I used "Pin<T> where T is !Unpin" as a means to say: "Pin<P> where <P as Deref>::Target is !Unpin", and “Pin<&mut T>” as a means to say "Pin<P> where P=&mut T for some T". Duly noted for further discussions.

Second and more importantly, you have Pin 's guarantee subtly wrong. Pin 's guarantee isn’t that T isn’t moved while the Pin exists, but that T will never be moved once the Pin is constructed (unless it doesn’t matter and thus T: Unpin ). […] You have to guarantee that the T will never be safe to move (in code beyond the “unsafety scope”) from here on out, forever .

Thank you for this explanation! This clears up my misunderstanding. Where is this precise guarantee stated in the pin module documentation? The closest I can find is the Safety § for get_unchecked_mut that states:

[…] You must guarantee that you will never move the data out of the mutable reference you receive when you call this function, so that the invariants on the Pin type can be upheld.

but it is limited to the returned mutable reference (in particular, it says nothing of the preexisting mutable reference of my playground example).

So to restate the Pin guarantee you expressed, once I create a Pin<P>, the value pointed by my P instance is expected to never move again (unless the pointed to is Unpin of course)?

If this is exactly so, then it looks to me like the Pin type is very limiting, wrt what we can do with it from safe code. Basically I was thinking of Pin as some kind of &mut T with limitations, that would apply for there entire’s life of the Pin but not beyond. Something like the following:

// impl block when P = &mut T for some T
// this Pin only uphelds the guarantee that T is not moved if Pin is observed, for the duration that it is observable.
impl<'pin, T> Pin<&'pin mut T> {
    /// safe to create, we wrapped our only &mut in existence into the Pin
    fn new(x: &'pin mut T) -> Self; 
    
    /// Difference to the existing `get_mut`
    /// 1. borrows the Pin rather than consuming it. 
    /// 2. returns a "wrapper" around &mut T rather than a naked &mut T.
    /// The wrapper impl DerefMut and is only there as a way to differenciate refs obtained from this method from refs obtained from the next. Using DerefMut through this wrapper is semantically unsafe, but since there is no way to ensure that from the wrapper, it is  the `get_mut` method itself that must be `unsafe`.
    unsafe fn get_mut<'a, 'lock : 'a>(&'a mut self) -> PinnedMut<'a, T>;

    /// "unpins" T, removing the guarantee that it won't move, and giving back a naked mut ref as a result. It is safe to move out of the returned mut ref.
    /// For this to be safe, Pin should provide, like &mut, no means of "copying" it. 
   /// There's currently a loophole due to Pin::as_mut that borrows a `Pin` and returns a new Pin that can safely be consumed...
    fn into_mut(self) -> &'lock mut T;
}

From what you’re saying I now understand that Pin is rather a line of demarcation, after which the pointed to value will never move again. If that’s so, it seems to me that the Pin type provides too strong of a guarantee, which limits its uses. Such “strong guarantee” available forever from the moment you create a Pin strikes me as inconsistent with the rest of Rust’s affine type system. Was an API similar to my above code considered (with a weaker guarantee on Pin, but also less unsafety on its usage), and if so, why was it rejected? I remember reading lots of blog articles on the design of Pin, and that it answers a problem very strongly constrained, hence the strong guarantee is maybe needed and surely there’s something I’m missing again?

( unsafe code that still knows what it’s doing to a concrete T is theoretically allowed to move the value, I suppose.)

I’m not sure I understand this statement. How is that unsafe code “knowing what it is doing” if it is violating the safety guarantees of Pin? Won’t that also ruin the guarantees of Pin for safe users, hence defeating the whole point of unsafe?

[Nemo157]

The problem with Pin's guarantees only lasting until the Pin is dropped is that there is no way for the pinned value to know when the Pin is dropped. As an example one of the earliest things I tried doing with Pin was to use an internal buffer for DMA data, here’s a playground showing a sketch of it. If the guarantee didn’t extend until Drop was called then this would be unsound, the value could be moved after the Pin was dropped, then it would pass a different pointer to unlink and not de-register the memory it had previously registered. That memory could then be used for another variable and written over by the DMA at some point in the future.

[zrk]

Interesting use case!

To use a Pin whose guarantee is lasting only until it is dropped, would it work to just make the Pin responsible for the guarantee? ie, embed the pin in a struct that would impl the drop?

Here’s a playground showing an implementation of your type using such Pin implementation. Would that be sound for you? AFAIK there is no way to change the address of the buffer once “linked” in such implementation?

[Nemo157]

I think that runs afoul of the general “scoped destructors are not guaranteed to run”, you could mem::forget the LinkedFoo and then deallocate the Foo without unlinking it.

Whereas Pin’s current guarantees include that the memory remains pinned until after Drop::drop is called, so doing something similar with Box::leak would just leave the memory allocated and linked forever (and it’s not possible to produce a similar setup where the value is on the stack and the destructor does not get called, since it becomes pinned to the stack and cannot be moved into mem::forget).

[zrk]

Does the current Pin forbids us from doing something like that? It looks to me like Foo is never moved after the pin is constructed, yet its destructor is never called.

In my experience, code that relies on destructors being run for safety need to have an unsafe fn new method (or maybe the unsafe comes from the link/foo method itself in our instance?), and use a macro to ensure no trick like the ManuallyDrop can be played. In my example implementation, that would mean I would probably have to further wrap my LinkedFoo in a Pin and give only access to that Pin via a macro in order to ensure that LinkedFoo cannot be mem::forget or put in a ManuallyDrop. We probably ought to document this pattern somewhere.

[CAD97]

The reason Pin is “in perpetuity” is that self referential structures are always still self referential. Another way to consider it is that once you have Pin<P<T>> it’s not safe to access the storage slot of T any more; you have to access it through pin-projected references.

The safety isn’t around drop being called on the same memory location, but that the structure is never accessed in a way that could move it from that memory location while it may still be required to be immobile. The example in the above post looks perfectly sound to me. (In fact, that shadowing trick is about the only way stack pinning is sound.)

[zrk]

I gave your answer a lot of thought, and I think you’re right. I now think that the Pin API, as implemented in the std, is the most suited to the problem at hands. In particular the guarantee of my proposed implementation is too weak to be maximally useful.

Another way to consider it is that once you have Pin<P<T>> it’s not safe to access the storage slot of T any more

I think this helped me a lot towards understanding the current API’s design.

I now think that a safe new from a &mut reference is not desirable, so I feel like this topic can be closed unless somebody wants to add something to the conversation.

Thank you for your patience!

[withoutboats]