Pin projection with lots of language support

language design
1

Most of the design work and discussion on pin projection has focused on implementing the feature with minimal specialized language support, using macros or non-specialized language features to accomplish as much as possible. What if instead, Rust went all in on making Pin a core language type? I explore such a design below.

Places today

Before discussing how to extend Rust's places to better support Pin, we need to review how they work today.

Properties of places

In today's surface Rust, a place is defined by the following attributes:

  • The type of its pointee
  • Its memory address
  • Its lifetime
  • The capabilities it possesses described below. We will call the possible capabilities mutable, movable, and unpinned; the next section describes each.

Operations on places

Different operations on places require different combinations of the aforementioned capabilities:

  • addr_of!(place) and field projection (via place.field or match destructuring): no capability requirements, all places can do it.
  • addr_of_mut!(place): requires mutable.
  • place-to-value coercion: requires movable.
  • &place (or ref place in a match): requires unpinned, and the lifetime of the resulting reference cannot exceed that of the place itself.
  • &mut place (or ref mut place in a match): requires unpinned AND mutable, and the lifetime of the resulting reference cannot exceed that of the place itself.

Field projection (via place.field or match destructuring) produces a new place with different capabilities, based on the following rules:

  • A projected place is mutable iff its parent place was mutable.
  • A projected place is movable iff (its parent place was movable AND no implementation of Drop exists for the type of the parent place) OR (the type of the projected place implements Copy).
  • A projected place is unpinned iff its parent was unpinned AND it is not a potentially-unaligned member of a #[repr(packed)] ADT.

Examples of places

Here are some examples of places with different combinations of capabilities:

use core::ptr::{addr_of, addr_of_mut};

struct NotCopy(u16);

struct Wrapper(u16, NotCopy);
#[repr(packed)]
struct PackedWrapper(u16, NotCopy);

fn foo(w: &Wrapper, mw: &mut Wrapper, pw: &PackedWrapper, mpw: &mut PackedWrapper) {
    // `w.0` is by-value and unpinned
    addr_of!(w.0);
    w.0;
    &w.0;

    // `w.1` is unpinned
    addr_of!(w.1);
    &w.1;

    // `mw.0` is by-value, unpinned, and mutable
    addr_of!(mw.0);
    addr_of_mut!(mw.0);
    mw.0;
    &mw.0;
    &mut mw.0;

    // `mw.1` is unpinned and mutable
    addr_of!(mw.1);
    addr_of_mut!(mw.1);
    &mw.1;
    &mut mw.1;

    // `pw.0` is by-value
    addr_of!(pw.0);
    pw.0;

    // `pw.1` has no capabilities
    addr_of!(pw.1);

    // `mpw.0` is by-value and mutable
    addr_of!(mpw.0);
    addr_of_mut!(mpw.0);
    mpw.0;

    // `mpw.1` is mutable
    addr_of!(mpw.1);
    addr_of_mut!(mpw.1);
}

New additions for pin projection

To support pin projections, a new place capability, pinned, is added. (This capability is not mutually exclusive with unpinned).

Producing a pinned place

A pinned place can be produced in the following ways:

  • Any unpinned place whose type implements Unpin is also pinned (and any pinned place whose type implements Unpin is also unpinned).
  • Any unsafe place (result of a raw pointer dereference, or static mut reference) is pinned (as well as unpinned).
  • Dereferencing a Pin<P> where P: Deref<Target = T> produces a pinned place of type T. (If P: DerefMut, the place is also mutable. If T: Unpin, the place is also unpinned.)
  • Projection from a pinned place via a #[pin] field (explained later) produces a pinned place.

New operations on pinned places

  • &place: requires unpinned OR pinned, and the lifetime of the resulting reference cannot exceed that of the place itself.
  • &pin place (or ref pin place in a match): this new operator produces a Pin<&TypeOfPlace>. It requires pinned—unless the resulting reference is to have a 'static lifetime, in which case unpinned is also acceptable. The lifetime of the resulting reference cannot exceed that of the place itself.
  • &pin mut place (or ref pin mut place in a match): this new operator produces a Pin<&mut TypeOfPlace>. It requires pinned AND mutable—unless the resulting reference is to have a 'static lifetime, in which case unpinned AND mutable is also acceptable. The lifetime of the resulting reference cannot exceed that of the place itself.

Projecting from a pinned place

Any field of an ADT can be annotated with at most one of the attributes #[pin] or #[unpin], unless it is a potentially unaligned member of a #[repr(packed)] ADT.

The projection rules for pinned and unpinned are as follows:

  • A projected place is unpinned iff (its parent was unpinned OR (its parent was pinned AND (the field is annotated with #[unpin] OR the field's type implements Unpin))) AND the field is not a potentially-unaligned member of a #[repr(packed)] ADT.
  • A projected place is pinned iff ((its parent was unpinned AND the field's type implements Unpin) OR (its parent was pinned AND (the field is annotated with #[pin] OR the field's type implements Unpin))) AND the field is not a potentially-unaligned member of a #[repr(packed)] ADT.

Reborrows

Automatic reborrows are extended to support pinned pointers, wherever the reborrow could be written out explicitly with &pin and the new projections.

Drop

The following changes are made to the Drop trait:

  • Implementations of Drop::drop() can choose to use a signature of either fn drop(&mut self) OR fn drop(self: Pin<&mut Self>).
  • However, if the implementing type is an ADT with at least one field annotated with #[pin], only the fn drop(self: Pin<&mut Self>) signature is permitted.

What it looks like

Here are some examples:

use core::{future::Future, marker::PhantomPinned, pin::Pin, ptr::addr_of_mut, task::Context};
use futures::future;

fn foo(ctx: &mut Context<'_>) {
    let mut r = future::ready(42_u8);
    // Automatic `&pin mut` borrow:
    // `u8: Unpin`, so place `r` is pinned
    r.poll(ctx);
}

fn poll_twice(f: Pin<&mut impl Future>, ctx: &mut Context<'_>) {
    // f is reborrowed, not moved
    f.poll(ctx);
    f.poll(ctx);
}

struct Projections {
    #[pin]
    pinned: PhantomPinned,
    #[unpin]
    unpinned: PhantomPinned,
    both: u8,
    neither: PhantomPinned,
}

impl Projections {
    // or perhaps `&pin mut self`?
    fn bar(self: Pin<&mut Self>) {
        let _: Pin<&mut PhantomPinned> = &pin mut self.pinned;
        let _: &mut PhantomPinned = &mut self.unpinned;
        let _: &mut u8 = &mut self.both;
        // No need fot `&pin mut`, implicit reborrow as `Pin`
        let _: Pin<&mut u8> = &mut self.both;
        let _: *mut PhantomPinned = addr_of_mut!(self.neither);
    }
}

impl Drop for Projections {
    fn drop(self: Pin<&mut Self>) {
        /* ... */
    }
}
5 Likes
2

I think this is generally a good idea: my only concerns are backward compatibility (isn't Pin<&T> unconditionally Deref<Target = T>?) and needing new keywords could be problematic for old editions (through 2021 and probably 2024 at this point) depending on if &pin <expr> is always unambiguous

3

I believe my proposal should not break this, as

1 Like
4

I'm assuming that includes calling &self methods and deref coercions.

1 Like
5

Trivially not: &pin (expr)

A pin keyword will be quite difficult to justify with pin being an identifier in use by std. A "weak" keyword that's only reserved in specific positions would work but is more complexity than a true keyword. Existing weak keywords essentially only appear where bare identifiers can't.

2 Likes
6

Note that last time we tried to do a weak keyword in expressions it didn't work.

They work well in items, but unless it's conjoined with another keyword (like if you needed const pin { ... } or something) contextual in expressions should probably be avoided.

1 Like
7

I've believed for years that Pin is the wrong abstraction, for two related reasons. One, it doesn't allow pinned types to implement traits whose methods take &mut self. Two, having the concept of both pinned and unpinned references to every type, when most types only ever want one or the other, adds a great deal of incidental complexity.

This proposal runs into the &mut self issue for Drop::drop, and solves it by adding magic overloading behavior; but that's not an option for any normal trait, which instead must be split into pinned and non-pinned versions.

Thus I prefer the !Move proposals that are still floating around. This proposal seems effectively mutually exclusive with !Move, since adding both would mean having language support for two different notions of immovability.

But if !Move doesn't happen, then this proposal would certainly be beneficial.

6 Likes
8

One concern I've always had with !Move proposals is that in many cases, "immovable" values can—and often must—be moved at some point. For example, a self-referential future may be moved into an executor spawn function before being polled. The Pin model handles this quite elegantly.

9

one other proposal I've seen before is to have a Pinned<T> type where &mut Pinned<T> is 100% equivalent and interconvertible with Pin<&mut T>. If pin is builtin, we could spell Pinned<T> as pin T and have &pin mut T be the same type as &mut pin T

10

what about reserving pin except for in a macro's name, a method's name (so a.pin() is valid but pin() is reserved), associated function/method names, mod names, a path with multiple segments (so pin::Pin, Box::pin are valid paths, but pin by itself is not unless it's a macro, mod, function definition, or method definition/invocation name).

afaict this covers all current stable uses in std

11

What stops you calling mem::swap on &mut Pinned<T>?

12

Pinned<T> is an extern type in that demo...so it isn't Sized and size_of_val returns 0 or panics.

13

That would still be ambiguous. &pin!(3) could be either a shared reference to the result of the pin! macro, or an &pin reference to !(3). I see no easy solution here sadly

14

could we not just always resolve that to the pin! macro and require parenthesis for ! -- &pin(!(3))?

15

One concern I've always had with !Move proposals is that in many cases, "immovable" values can —and often must —be moved at some point. For example, a self-referential future may be moved into an executor spawn function before being polled. The Pin model handles this quite elegantly.

I don't want to derail the thread, but generally speaking, !Move should be paired with some form of in-place construction. That could be in the form of "placement by return" or something more explicit. In-place construction is a highly-requested feature even for movable structs because of its performance benefits. But with !Move, such a feature would be necessary in order to construct things that should never move at all. This should also generally be sufficient for futures.

(Or if in-place Future construction isn't enough and you really need to move them, then the design could be combined with the recent "async is IntoFuture, not Future" proposal: the IntoFuture object could be movable while the Future object is not. Admittedly, backwards compatibility would make this way, way more complicated. Broadly speaking, backwards compatibility with the existing async ecosystem is the hardest part of any !Move proposal; I'd like to believe it's solvable, but it won't be pretty.)

5 Likes
16

or, actually, always resolve pin!(...) to the pin macro only in old editions, new editions would have a pin operator that takes a place and returns a pinned place.

17

This feels like making Sized a subset of Move. It avoids the ecosystem split that Move would have only because it makes it impossible to have Sized + !Move.

18

that's only how they did it without language support, if we have language support, a native pin T type could likely do it better without having to rely on an extern type and Sized shenanigans.

19

If we can manage to simplify pinning and provide linear types with the same change, that'll be wonderful.

1 Like
20

Edition 2024 is to be finalized by May, so it might worth trying to reserve a keyword for this. Otherwise we must wait for another three years.