Pin<'a, T> is described in RFC 2348 as a wrapped &'a mut T. However, I think it would be better as a wrapped &'a move T, which doesn’t currently exist in Rust, but is a reference type that owns the data it points to and will drop it when dropped.
I think there is a strong case for adding &move T to Rust, possibly with different syntax, but would like to make that case separately. Here is why I think it is better if Pin<T> owns the data it points to:
It makes Pin<T> and PinBox<T> consistent, since they would both own the T that they point to
It would guarantee that the T would never move before being dropped, since the Pin<T> owns the T and drops it for you
If &move T is added at a later date, then there could be a safe API for converting an &'a move T to a Pin<'a, T>
How would reborrowing work with an &move T? Specifically with the below code there are 3 Pin<F>s alive during the baz call, if a Pin uses an &move T internally, and a &move T drops its T when dropped, does that mean the concrete foo is dropped after the first call to the method on it and the second call is using “non-existing” memory?
I’m not sure reborrowing really works with &move-references, since they own the data they point to rather than borrowing it. But to mutably borrow a Pin<T> you would use &mut Pin<T>
Ok, so this basically makes Pin<T> an owned proof that its internal value has been pinned, rather than a reference + proof as it currently is, so all APIs would change to taking &mut Pin<T>?
trait Foo {
fn foo(self: &mut Pin<T>);
}
One question with using that with PinBox: How do you get a &mut Pin<T> from an &mut PinBox<T>? The PinBox is the owner of the data so you can’t take an &move T from it to make the Pin<T>.
A little bit of thinking suggests maybe changing the PinBox API something like:
but… that requires the ability to move part of a value behind an &move reference out to a new &move reference. It also means you can’t call a function on the pinned T in the box, then return the box to your caller as you currently can.
I think I’d need to see a quick sketch of the main Pin and PinBox APIs and how they’d be used to understand what this gives us compared to the current formulation.
I guess you’re right, it sounds like there is good reason for the borrowing version of Pin that we currently have. If &move references are added, it would probably make sense to also add an owning version of Pin that wraps an &move reference.
Immovable types are like dynamically-sized types in that there is currently no way to pass ownership of them without heap allocation, but unlike dynamically-sized types, it seems the Unsized Rvalues RFC doesn’t solve that problem for immoveable types, and that makes &move and OwnedPin more attractive.
Probably other use case for &move T can be self-consuming methods in object safe traits which ideally should work with Self and trait objects which exclusively own data.
This doesn't seem right. You can presumably move out of an &move T, while that's specifically forbidden for a Pin<T>. And a Pin<T> only represents borrowed access; once it drops, the source—such as a PinBox or another Pin—may regain access. An &move T causes access to move permanently with it.
Certainly an &move T could be useful for pinning things not allocated in a PinBox<T>. If you encapsulate it (let's say in a PinMove), this guarantees you control all future access to the value, which means you can safely lend out access via a Pin<T>. This would essentially replace the role of the ouroboros borrow† that's been proposed to make stack pinning work.
† a borrow that eats itself at a function boundary because the borrow checker can't guarantee that the borrow doesn't escape into itself. Consider an &'a mut (i32, &'a i32), where the first field could potentially be borrowed by the second field.
So trying to be a bit more thoughtful, Pin can’t own its referent logically because that would invalidate the &'a mut PinBox<T> -> Pin<'a, T> conversion, which is necessary for PinBox to be worth anything at all. However, I think any API for constructing Pin<'a, T> from the stack will necessarily logically own the T it points to, even though you won’t be able to move out of it. If it didn’t own it, you wouldn’t be able to prevent the user from moving it again on the stack.
Which matches up with the approach taken in your blog post which I was referring to as an ouroboros borrow (ouroborrow?) . An interesting difference is that the &move approach separates ownership from storage, so the owner (PinMove) can still be moved around and accessed, with only the storage being immovable/inaccessible. With the ouroboros approach, the owner (Slot; see Appendix: Ouroboros approach) has to become immovable/inaccessible because it also stores the pinned value.
Ownership might also be the solution to @cramertj’s invariant—i.e. Drop::drop is called before the storage is freed. With a PinBox<T> you know that T isn’t owned by a ManuallyDrop, so you can safely perform setup that must be undone before freeing by Drop::drop. The stack equivalent could be to perform that setup where the ownership is still known. If you can intercept the &'a mut Slot<'a, T> before it becomes a Pin<'a, T>, you know the owner isn’t ManuallyDrop and can safely perform setup. With PinMove, you would also know the owner, but because you retain access to the PinMove throughout the pinning, there’s some more flexibility.
Appendix: Ouroboros approach
I’m arbitrarily renaming the Pin in your blog post to Slot to remove the name conflict with the current pinning design. I at one point suggested Hasp instead, but I have a feeling that one won’t catch on .
Inspired by this thread and the recent DynSized RFC, I had an idea for how to replace Pin with a forwards-compatible, library-only subset of !Move. Which is actually just !DynSized.
In ‘short’:
Pin<'a, T> is replaced by &'a mut T. So Future, Generator, etc. can just take &mut self again! Here I’ll talk about Generator for convenience, but the same applies to Future.
To enforce immovability, users never get access to owned generators directly; thus, instead of generator functions returning impl Generator, they would return something like impl Anchor<Inner=impl Generator>, for some trait
trait Anchor {
type Inner: ?Sized;
// PinBox and friends would use this method to implement DerefMut;
// it's unsafe because you should only call it if you can guarantee
// that `self` will never move again.
unsafe fn get_mut(&mut self) -> &mut Self::Inner;
}
Okay, the hard part. Why doesn’t the above ‘just work’?
Because of mem::swap and friends. Given &mut MyGenerator you can move it or (in some cases) extract an owned MyGenerator, which breaks the assumption of immovability.
So what if we mark MyGenerator as !Sized? (Notwithstanding the semantic strangeness of making something !Sized that doesn’t actually have an unknown size.) That would block mem::swap and similar.
But there’s one dangerous function that doesn’t have a Sized bound: mem::size_of_val. In particular, the Box-to-Rc conversion works even for unsized types and assumes it’s kosher to move them by memcpying mem::size_of_val(foo) bytes.
Although you couldn’t actually use that conversion to break the above scheme, since you’d never be allowed to get a Box<MyGenerator>, its presence in the standard library is a license for unsafe code to use the same technique in broader scenarios.
But this is the same problem meant to be solved by !DynSized!
The above-linked RFC proposes having size_of_val either panic or return 0 for a !DynSized type, combined with a lint; an alternative mentioned is to make DynSized a new default bound and add ?DynSized.
Meant to be used with extern type, i.e. FFI opaque pointers, which itself is already accepted and implemented (but not stable).
But it would work just as well for immovable types. The desired semantics are effectively the same at least from a generic code perspective.
But even ignoring how crazy that sounds, how is it a library-only solution if DynSized doesn’t even exist yet?
Well, for now we can simulate it with a silly hack. Generator functions would expand to something like:
// The anchor contains the actual state...
struct MyGeneratorAnchor {
// local variables go here...
x: i32,
}
// The generator type is just an extern type!
// In other words, &MyGenerator is an 'opaque pointer' that secretly points
// to MyGeneratorAnchor.
extern { type MyGenerator; }
impl Anchor for MyGeneratorAnchor {
type Inner = MyGenerator;
unsafe fn get_mut(&mut self) -> &mut Self::Inner {
// create the opaque pointer by casting to &mut MyGenerator:
&mut *(self as *mut MyGeneratorAnchor as *mut MyGenerator)
}
}
impl Generator for MyGenerator {
fn resume(&mut self) -> whatever {
// undo the above cast:
let anchor: &mut MyGeneratorAnchor = unsafe { &mut *(self as *mut MyGenerator as *mut MyGeneratorAnchor) };
// … now we can use local variables …
}
}
So, how does this solve the problem?
MyGenerator is !Sized (extern type already has that effect), so anything that tries to use the standard mem::swap, mem::size_of, etc. just won’t compile.
mem::size_of_val(foo: &MyGenerator) will work but return 0 (because it’s an extern type). So if some tricky unsafe code tries to, say, swap two &mut MyGenerator instances by memcpying bytes around, that’ll just silently do nothing, leaving the generators’ actual data intact. That’s not great, in the sense that it’s a surprising result, but it’s not unsound either. And again, this is the exact same problem that applies to ‘real’ FFI opaque pointers.
…There might be some even weirder things unsafe code might try to do that would be unsound – though it’s hard to say it’d be justified in doing so – but that would be equally unsound with extern type used for FFI! (For example, a version of take_mut that takes ?Sized and returns the existing object copied to a box.)
In the hopefully near future, the DynSized RFC or something similar will land, providing for a lint or error when someone tries to use size_of_val here.
In the meantime, since essentially no generic code actually does that, you’ll be able to use the generic ecosystem as usual with &MyGenerator and &mut MyGenerator – with no need for everything to add an extra variant for Pin. ?Sized is enough.
It doesn’t seem worth it to me. The design of DynSized is influenced by a couple things: backwards compatibility and being relevant to a low-level feature. This creates a mismatch in design goals compared to an entirely new higher-level usability feature (Pin supports generators and other self-referential types). People will typically expect to run into something a bit hackish when doing FFI, but not for a headline language feature.
Well, for one thing, backwards compatibility is important to the design of any new Rust feature. I don’t see how that differs much between this and extern type. After all, just as Pin<'a, T> is a wrapper for &'a mut T for weird T, FFI users could make MyFFIReference<'a> wrapper structs; but extern type provides extra sugar to allow native references to be used safely, a goal that I think applies here as well.
And keep in mind that this would be a replacement for Pin, which is itself hackish – I’d argue more so. Since Pin<T> is not a native mutable reference:
You can’t reborrow it for a shorter lifetime;
The syntax is clunkier, and it’s non-obvious from the name that Pin<'a, T> acts like &'a mut T;
It doesn’t work with any existing generic code that expects &mut T.
Also:
You can’t borrow a field as a Pin, i.e. given struct Struct { field: MyGenerator }, you can’t go from Pin<Struct> to Pin<Field>…
…except my proposal wouldn’t allow that either (with both versions it’s possible to achieve the same effect, but only with further hacks). However, my proposal is forwards-compatible with some form of native support for immovable structs – a feature I’d love to see added in the future – which could make this ‘just work’. That is, you could write something like
but that would be a lot more awkward. (maybe &pin?)
Anyway, the main source of hackiness in my proposal – well, that doesn’t also apply to Pin – is the idea that size_of_val(&MyGenerator) would compile but return 0 (or panic in the future). This matches the latest DynSized RFC’s design (link again for reference) – but even that RFC suggests a “Milestone 3” where size_of_val would eventually gain a proper DynSized bound, resolving the issue.
It does question whether the epoch system will permit such a change. As an alternative, there’s nothing about that RFC that would be incompatible with eventually moving to a ?DynSized-based approach (where DynSized is added everywhere as an implicit bound). Or perhaps having it be implicit but only for old-epoch code.
Either way, I think that the vast majority of generic parameters currently marked ?Sized can work with non-DynSized types. If a DynSized bound were made implicit, almost everything would want to migrate from ?Sized to ?DynSized; if it’s kept explicit, very little code would need to change from ?Sized to ?Sized + DynSized.
But again, extern type already exists (although it’s not stable), and has the same requirements for correctness as what I’m proposing. That is, to properly support it, there’s no way around either adding some form of DynSized, or adding an unsafe code guideline strictly limiting the use of size_of_val – either of which would also suffice for generators. It’s true that having !DynSized be just for FFI would let us sort of “shove it under the rug”, whereas having it also be for generators would force it to be more prominent. But I’m not convinced by that argument.
For one thing, safe FFI wrappers ought to be, well, safe, and at least potentially (depending on the design of the foreign library) no less hacky than any other Rust API. If DynSized-related issues are unacceptable for normal Rust code (whether compile-time breakage or runtime panics), they’re also unacceptable for FFI wrappers, and we should just get rid of extern type before it’s stabilized.
For another, I personally think that full-fledged support for immovable types will be an important part of Rust’s future – or ought to be. So the only question is whether to stick with Pin forever, or eventually have some form of !Move. But most code that could work with T: !Move can work just as well with T: !DynSized, and that’s largely also the same as the code that currently takes T: !Sized – it’s mostly a question of whether some code only handles references to T (in which case the most general bound, !DynSized, should work), or whether it handles it by value (in which case you currently need Sized anyway). There are only a few exceptions, like the Box-to-Rc thing.
Thus, it makes perfect sense for the ultimate design of Move to only be a slight addition on top of DynSized, a subtrait that most code won’t want to explicitly bound on. If Move were somehow orthogonal to DynSized, that would just feel like two redundant language features. So it’s logical, not merely a hack, to base an initial version of immovable types (i.e. generators/async fns) on top of !DynSized. In the future, they could switch to impl’ing DynSized but not Move, to properly express that they actually do have a size, and that would be a backward-compatible change just like adding an impl for any other trait. Or we could just stick with DynSized = Move forever, since as I said, there’s very little justification for generic code to query the size of some generic T if it’s not allowed to memcpy based on that size.
What I meant for backwards compatibility was that extern types are already in the wild, whereas immovable types aren't. Subordinating immovable types to the design considerations of extern types forces compromises into the design of immovable types which don't need to exist.
Viewing Pin<T> as a wrapper for &mut T for weird T begs the question somewhat, since you're arguing for it to be equivalent to &mut T where T: !DynSized. Also, to the extent that this describes the implementation, you're violating the abstraction barrier. Pin<'a, T> can be viewed in its own right as a unique reference type with lifetime 'a which, for T: !Unpin, can only exist where &mut / move access has been permanently disabled for the referent.
For one thing, safe FFI wrappers ought to be, well, safe, and at least potentially (depending on the design of the foreign library) no less hacky than any other Rust API. If DynSized-related issues are unacceptable for normal Rust code (whether compile-time breakage or runtime panics), they’re also unacceptable for FFI wrappers, and we should just get rid of extern type before it’s stabilized.
In principle, yes, but extern types are a bit of a necessary evil unless/until the soundness issues can be fully addressed. The opportunity exists to have immovable types enter the world entirely absent these issues.
Box-to-Rc comes to mind. I'm pretty sure you'd end up with an uninitialized T at the end of it. I'm not convinced by the argument that immovable types will always be sufficiently abstract that you can't put them in a Box. Maybe you can come up with a scheme that works for generators, but that's far short of a generalized feature, and doesn't address what external library code might (validly?) do.
IfDynSized is accepted, and if it moves sufficiently close to closing the soundness holes (for some meaning of "sufficiently close"), then yes, I can see this approach being valid. I don't think an unmitigated zero-size hack should make it into stable/production code.
If the idea is for this proposal to be fully backward-compatible with the existing Pin design, then this doesn't work. Being !DynSized is inseparable from the type, whereas the T in a Pin<T> can have had a fulfilling life as a movable value before getting pinned. Also, unless you want a transition period in which !Unpin types are warned to implement !DynSized with ultimate breakage, Pin<T> needs to be able to maintain its invariants without any changes needed to T. For these reasons, Pin<T> would probably have to be equivalent to something like &mut Pinned<T> where Pinned<T>: !DynSized. (Edit: And this doesn’t address any potential method conflicts, Deref impl conflicts, ...).
The current design of Pin currently has that for the &T case. But yeah, having it just work for &mut T where T: ?Sized is nice, though I'm not sure how common that is compared to &T where T: ?Sized. I'm not entirely convinced about the need for distinguishing whether the referent is movable vs immovable for &T.
Fair enough. But – first of all, I don't think this question really affects any of the points I've made. I wrote:
After all, just as Pin<'a, T> is a wrapper for &'a mut T for weird T, FFI users could make MyFFIReference<'a> wrapper structs; but extern type provides extra sugar to allow native references to be used safely, a goal that I think applies here as well.
I guess that if Pin is seen as a fundamental reference type, the guidance could be that instead of making your own MyFFIReference<'a> wrapper, you should design your APIs around Pin<'a, MyFFIType>. But then what do you use for non-unique references? &MyFFIType doesn't work because of the same issue with size_of_val. You still need DynSized. On the other hand, if you purely used, e.g., MyFFIReference<'a> and MyFFIReferenceMut<'a>, then you wouldn't need either DynSized or extern type, but the API would be less ergonomic, which is the same issue that applies to Pin.
In any case, my problem with viewing Pin as a fundamental reference type is that at least for general self-referential structs, there are also use cases for reference types that correspond to & and, potentially, &move (the original proposal in this thread), while guaranteeing immovability. So we'd end up with 6 reference types instead of 3 (or without &move, 4 instead of 2).
For example, this is unsound using regular &:
struct S {
foo: [i32; 32],
// points to one of foo's elements:
bar: Cell<Option<&'foo i32>>,
}
impl S {
fn set_bar(&self) {
self.bar.set(Some(&self.foo[0]));
}
}
But it would be sound with an immovable version of &.
(&Pin<T> could work as a substitute, but has limitations due to being a double pointer, e.g. you can't go from &Pin<Struct> to &Pin<SomeField>.)
[edit: See also @RalfJung's recent post Safe Intrusive Collections with Pinning, which explores a somewhat different direction than what I'm proposing, but also concludes that an immutable pin is useful. I think the invariant he wants can also be achieved with my Pin-less approach.]
I missed that, but let's correct the point to: "you can't implicitly reborrow it for a shorter lifetime".
In principle, yes, but extern types are a bit of a necessary evil unless/until the soundness issues can be fully addressed. The opportunity exists to have immovable types enter the world entirely absent these issues.
I think that's short-term thinking. To the extent that soundness issues exist, they're an urgent problem and need to be solved before extern type can be stabilized. (Or else there's not much benefit to having it when unit structs already satisfy the unsound-but-mostly-works use case.) However, I don't think they're a big deal; in particular, I'm not sure whether any code actually exists that uses size_of_val to copy bytes from a non-owned type. And the DynSized RFC proposes having size_of_val panic (instead of returning 0), which will eliminate any such unsoundness, at the cost of surprising behavior.
Box-to-Rc comes to mind. I’m pretty sure you’d end up with an uninitialized T at the end of it. I’m not convinced by the argument that immovable types will always be sufficiently abstract that you can’t put them in a Box. Maybe you can come up with a scheme that works for generators, but that’s far short of a generalized feature, and doesn’t address what external library code might (validly?) do.
Regarding Box-to-Rc specifically: in the short term, exposing a Box<Immovable> to safe code would be unsound because you could move out of it.
In the medium term, having size_of_val panic fully removes the potential for unsoundness, again at the cost of surprising behavior.
In the long term, the goal is to have the standard library include appropriate DynSized bounds so the compiler can reject such code at compile time. The DynSized RFC proposes one way of doing that while minimizing disruption.
Actually, I'd only skimmed that RFC, and it turns out I misunderstood what it was proposing . Now that I understand, I like it much better! And think it's clearly better than !DynSized. I thought it was proposing a lint at monomorphization time if size_of_val etc. is instantiated with !DynSized types. But it's actually proposing that for each caller, the compiler try to prove, in a generic context, that T: DynSized holds, and lint if it can't. So to avoid the lint, callers must change their bounds to add either T: DynSized or #[assume_dyn_sized], either of which propagates the requirement outward. Thus, as long as your code compiles without producing the lint, you have a strong guarantee it's fully DynSized-safe, even in the fact of clients instantiating your generic code with arbitrary parameters.
You probably already knew that, but I didn't, and after thinking about it, I'm now much more confident that DynSized can be adopted quickly and painlessly. I also commented on the RFC with a suggestion: instead of adding a special-case #[assume_dyn_sized], we can get the same effect using specialization, if the compiler is changed to support #[deprecated] on impls.
Moving on:
If the idea is for this proposal to be fully backward-compatible with the existing Pin design, then this doesn’t work.
No, it's not meant to be backwards-compatible with Pin. Even if it's a bit late, there's still some time to kill Pin before it's stabilized.
The current design of Pin currently has that for the &T case. But yeah, having it just work for &mut T where T: ?Sized is nice, though I’m not sure how common that is compared to &T where T: ?Sized. I’m not entirely convinced about the need for distinguishing whether the referent is movable vs immovable for &T.
Above I showed a use case for immovable &T – basically any use of interior mutability with self-references. The current design… oh, I missed the unconditional Deref impl on Pin even for T: !Unpin. So you get to work with native &T, at the cost that immovable types based on Pin can never have interior mutability. Not great, IMO...
To be honest, there’s a lot about your proposal that confuses me @comex. Here are some points:
What does calling a generator return? It seems like it returns a pointer to the generator, but then where is the generator allocated? On the heap? That’s exactly what we don’t want - to heap allocate generators every time. (If we were okay with that, this would all be fairly trivial actually: we’d just implement Generator for PinBox<{anon type}> and call it a day).
Your commentary about the relationship between DynSized and Unpin doesn’t seem correct for me. We know the size of a generator - statically in fact. This is very important: DynSized is required to be able to deallocate something, because the dealloc API takes a Layout argument. If we were to say that the anonymous generator were !DynSized, we’d never be able to deallocate them.
I am similarly confused. To add to @withoutboats list:
Pin is forwards compatible with introducing &pin into the language some day. That would provide both implicit reborrowing and implicit field access (which currently both exist, bot only explicitly and, for field access, unsafely: Pin::map).
I also have to say it sounds extremely hackish to me to tie pinning and sizedness together. There can be sized types that require pinning (like most generators), and there can be unsized types that do not (like all the unsized types we have currently). You are right that a !DynSized type is implicitly pinned, but requiring all pinned types to be !DynSized seems overkill. For example, I might want to take two futures and put them into a pair, but only the last element of a struct can be !Sized -- so you'd restrict composition of futures, for no good reason.
Also, forgive me if you have commented on that already (in that case I missed it), but did you comment on whether this is a "local extension"? The fact that Pin is automatically correct for any type that does not care about pinning is a big win. In contrast, your comment about Rc and Box seems to indicate that your proposal is incompatible with some existing code.
EDIT: Oh, also I think your proposal is incompatible with code I consider legal. I would think one can have a strange unsized variant of swap with a type like
Compare the (dynamic) sizes of *x and *y to make sure they are equal; panic if they are not.
swap the fat pointer information between *x and *y
mem::swap the memory at **x and **y (it's the same size, after all)
Is this crazy? Yes. But I think it should be legal. Not because it is useful, but because it demonstrates that unsized types really are just that---unsized---and not have some other move-related magic to them that nobody expects there.
Unsizing and pinning really are two orthogonal features. We should keep them separate.
. An interesting difference is that the 
. Now that I understand, I like it much better! And think it's clearly better than 