First experience and thoughts about `Pin` API

[earthengine]

In fact, pin_project 's “derive” actually strips the pin for you.

I was looked into pin_project. Yes it is ergonomic and easy to use. However in terms of safety, it is not much better than unsafe code blocks above. We still have to make some promise that the compiler cannot check, although it is not using the unsafe keyword explicitly.

[CAD97]

And this is an inherently hard and unsafe task. It’s like writing Vec; it’s not something that needs to be safe.

Working directly with Pin is probably going to always require you (or a “derive” you use) to use unsafe. Pin projection is unsafe. And one of the best things about Pin's stabilized design is that it is purely a library type.

Using pin_project encapsulates the unsafety for you (though personally I’d prefer opting in to stripping the pin rather than keeping it on fields for safer-by-default). It’s the same as using Vec and the entire Rust philosophy: it’s possible to write machine-checked safe code on top of human-promised safe code (that does generally unsafe things).

[earthengine]

I feel like Vec's story is quite different here. With Vec, if only using the safe APIs, you get panics when you get things wrong (for example, out of index etc). The safety promise of Vec is: your error is either checked at compile time, or at runtime with panics. You cannot cause UB whatever you did in safe code. This is not as good as full compiler checks, but it is the best thing next to it.

With Pin, if we didn’t keep the promise, it will be UB: the programs works like there is no issues, but in some cases it just behaving wierd.

Can we have the samething like in Vec? So when you moved a U under a Pin<&mut U>> returned from get_mut_runtime_checked() incidently, the compiler panics.

[CAD97]

The point I’m trying to make isn’t that Pin<_> is Vec<_> in terms of safety abstractions, if anything, it’s that Pin<_> is ptr::NonNull<_>. That is, it’s an unsafe-to-use abstraction that you build safe abstractions on top of.

And Pin is the same as Vec in respect to safe code. Anything you do safely to any type in Rust is free from UB. This is the guarantee of Rust. This holds for Pin. It’s just that because the guarantee that it provides – that a type instance won’t be moved by untrusted code – isn’t guaranteed by the compiler that Pin<P<_>> to &_ is safe but to &mut _ is unsafe. Pin is the guarantee of immobility.

Basically, the point I’m trying to make is that you shouldn’t need to use Pin directly the same way you shouldn’t need to use ptr::NonNull directly. Other layers of the abstraction have been written already that you can use safely. join has to be written once, and async/await! hides the details of immobility required to drive a Future.

Pin projection would definitely be nice to have safely. But it doesn’t have to be a feature of the language or even the stdlib. It would help a small number of libraries (not applications, probably, which would just use async/await! and libraries built on Future) for a very noticeable cost (defining how exactly pin projection works) when we have today a macro that solves the issues in a resonably-safe-to-use manner (that is, without writing unsafe yourself, I don’t think you can use pin_project's attribute macros to cause unsafety – correct me if I’m wrong here).

[comex]

I think there may be some confusion about how Pin works. Going from Pin<&mut Struct> to Pin<&mut Field> is safe: it upholds the Pin invariant. There is not even a need for a runtime check. But there are two caveats:

• The language does not provide a way to actually perform that operation without first obtaining a real pointer to the struct, hence the need for an unsafe implementation.
• It’s only safe as there is not also a way for safe code to go from Pin<&mut Struct> to plain &mut Field (i.e. it’s either-or). This means you have to limit Drop impls can do, which I guess pin_project takes care of.

I somewhat disagree with @CAD97, in that I think built-in pin support probably would help applications significantly. I admittedly don’t have much experience with Rust async/await, but even if it often obviates the need to manually handle Futures, it doesn’t always – nor should it, since not everything you might want to do with them can be expressed with async/await. But that’s somewhat beside the point.

[earthengine]

This means you have to limit Drop impls can do, which I guess pin_project takes care of.

Actually pin_project didn’t take care of it. It just simply require the user to uphold this, so this why I said it is not much better than use explicit unsafe code.

since not everything you might want to do with them can be expressed with async/await. But that’s somewhat beside the point.

join has to be written once, and async / await! hides the details of immobility required to drive a Future .

Just giving another example that require manually handle Futures. Here is the alt function: given 2 Future, poll them parallelly, and prepare a result if at least one of them is ready:

pub enum EitherOr<T1,T2> {
    This(T1),
    That(T2),
    Both(T1,T2),
}
pub fn alt<T1, T2>(
    f1: impl Future<Output=T1>,
    f2: impl Future<Output=T2>
) -> impl Future<Output=EitherOr<T1,T2>> {
    struct AltFuture<F1,F2>(F1,F2);
    impl<T,F1,F2> Future for AltFuture<F1,F2>
    where
        F1: Future<Output=T>,
        F2: Future<Output=T>,
    {
        type Output=EitherOr<T1,T2>;
        fn poll(self: Pin<&mut Self>, lw: &LocalWaker) -> Poll<Self::Output> {
            let this = unsafe {
                let this = self.get_unchecked_mut();
                (
                    Pin::new_unchecked(&mut this.0),
                    Pin::new_unchecked(&mut this.1),
                )
            };
            match (this.0.poll(lw), this.1.poll(lw)) {
                (Ready(v1), Ready(v2)) => EitherOr::Both(v1, v2),
                (Ready(v1), _) => EitherOr::This(v1),
                (_, Ready(v2)) => EitherOr::That(v2),
                (_, _) => Pending,
            }
        }
    }
    AltFuture(f1, f2)
}

(Note this is more flexible than the futures::Future::select: it does not require the futures to have the same type. And more importantly, it is less biased: in case that both futures being Ready, it don’t have to pick any arbitrary one.)

[CAD97]

(futures:0.1)::Future::select doesn’t have a restriction on input type? And the current state is a n-ary macro fn: (futures-preview:0.3-alpha.12)::select!. That macro does require a unified output mapped type, but you can still do an either-or enum mapping yourself, and it allows recovering the non-exhausted inputs, unlike your implementation.

If I read the implementation of select correctly, it doesn’t have a bias either: it’s random in what order the futures get polled, and once a selection has been made, no more polls are done.

[earthengine]

Oh, I mean futures::Future::select is too much restrictful, my alt don’t have restrictions, as the input futures can have different types.

If I read the implementation of select correctly, it doesn’t have a bias either: it’s random in what order the futures get polled, and once a selection has been made, no more polls are done.

I am curious on this as it is either impossible for the signature given, or it have to pay the cost of a random number generator.

Also a “random” bias is the worse case of bias as it is nondeterministic.

To resolve the unified result, you have to pick a Future to poll, and if you don’t poll all Futures at the same poll cycle, you are bias to the one you just pick. If you do poll both then you have a chance to get both results in hand, in that case you either have to combine them together (if you have a monoid, but that means you have much bigger restriction on the return type), or you have to drop (or restore to unresolved) one of them. Either use a random generator to randomly pick one to be drop or restored, or bias to one of them.

So my solution is to return them both in a new variant in EitherOr, this is my way to avoid bias.

Of cause, my design is also easy to extend to alow recovery of the unresolved future and use macro to allow n-ary operations.

[CAD97]

Neither futures 0.1 or 0.3-alpha.12 have any restrictions on the input type of select other than that it is a future and that you map all the outputs to a unified type. The only restriction that they apply that you don’t is on the output type of the future, directly in 0.1 and indirectly in 0.3. The inputs can be unrelated types though you seem to be implying they can’t.

And yes, futures 0.3 select uses rand::thread_rng to shuffle the array of potential polling functions to determine the order it polls in. All futures are polled in that random order if none complete, and short circuits at the first completed one. Nothing is lost.

But we’re off topic here, as this isn’t talking about Pin anymore. Sure, maybe the select! implementation isn’t exactly what you’d prefer. But implementing your own alt only requires a small safe cost of #[pin_project] in terms of pin projection.

(I’d very much love to hate to see the “OptionHList” mess required for an n-ary alt that polls all the futures and gives back all the successes.)

[earthengine]

But we’re off topic here, as this isn’t talking about Pin anymore. Sure, maybe the select! implementation isn’t exactly what you’d prefer. But implementing your own alt only requires a small safe cost of #[pin_project] in terms of pin projection.

Ok let’s get back to the topic. A small safe cost is still not “zero cost”. Those alt or select or join have to be in the std or at least in a crate, as well as all other convenience methods that might be useful. Then we can see people eliminate Pin in their safe code.

(I would be interested in writing such a crate, just advise me other useful staff people would need, I probably would not write and_then or other common things as they are trivial with async/await, we have to think again before accepting a function that was useful in the old ways)

(I’d very much love to hate to see the “ OptionHList ” mess required for an n-ary alt that polls all the futures and gives back all the successes.)

(continue the off-topic)

My original design was returning a (Option<T1>,Option<T2>), but its “(None, None)” case is not reachable, and so that I switched to EitherOr. For n-ary alt, I would definitely use n-tuples of Options if we don’t need recovery and Eithers if we do, instead of defining new enums, as the all-none case is negligible.

(Also, there is a strong reason to not featuring recovery - we are in a Pined struct, and its content was garanteed not to move until drop. This means, even we ignore those futures that didn’t resolve, they are still available somewhere in the stack, the user still can access them and do whatever they want. This is another reason why the std::future helpers need to be re-designed.)

[newpavlov]

A bit off-topic question: is conversion from &mut T to Pin<T> done automatically for T: Unpin? In other words, I still don’t quite get how generators with fn resume(self: Pin<Self>) will work. Do I have to pin one explicitly before using it even if it’s not self-referential, or will it be done automatically somehow behind scenes?

[Nemo157]

There is Pin::new for this conversion, e.g. see that this test is able to avoid using any unsafe code when interacting with movable generators.

[newpavlov]

So if we’ll get for loop integration, we will have to write code like this:

struct Foo { .. }
impl Generator for Foo { .. }

let result = for val in Pin::new(Foo::new(bar)) { .. }
// or
let result = for val in Foo::new(bar).pin() { .. }

Even though this custom generator does not do any self-referencing? Does not look that nice.

A good example where generators can result in a nicer API is chunks_exact variant which returns leftover slice instead of simply omitting it:

// `chunk` has type `&[T; 16]` and `leftover` has type `&[T]` with length < 16
let leftover = for chunk in Pin::new(slice.const_chunks::<16>()) { .. }

Looks quite clunky to me…

[Nemo157]

@newpavlov I don’t think so, I’m pretty sure that would be done via an impl<G> IntoIterator for G where G: Generator + Unpin which could be used like

let gen = || { yield 1; yield 2 };
for val in gen {
} 

let self_ref_gen = static || { yield 1; yield 2};
pin_mut!(self_ref_gen);
// or `let self_ref_gen = Box::pin(self_ref_gen);` to use the heap
for val in self_ref_gen {
}

I actually have an experimental crate near this area that I can probably use to verify this all works over the weekend.

[newpavlov]

While implementing IntoIterator for Generator will be a path of least friction, it will discard return value, which I believe is a really bad design decision. Viewing Iterator<T> as Generator<T, ()> is very natural, and doing it the other way around feels simply wrong. Also the proposal quite neatly solves question of for loop return value and issues around Iterator<Result<T, E>>, which can be replaced with Generator<T, Result<(), E>>.

[Nemo157]

I don’t know how you expect to receive the return value from a generator that is being used in a for loop, it must be consumed by the Iterator::next implementation and discarded since it can only return None. Unless you’re referring to actually change the for loop expansion to use something other than Iterator, in which case you can just do whatever is necessary to make it work nicely. If you know how long the iterator is beforehand I guess you can use take to stop before you consume the return value, but that seems unlikely in most cases.

I should have noted that as well as that IntoIterator it would probably use impl<G> Iterator for G where G: Generator + Unpin as well which (along with some extension methods) could allow for pretty complex uses without much overhead in the logic like:

trait GeneratorExt: Generator {
    fn by_ref(&mut self) -> &mut Self;

    fn map_complete<U, F>(self, f: F) -> impl Generator<Yield = Self::Yield, Return = U>
    where
        F: FnOnce(Self::Return) -> U;

    fn resume_unpin(&mut self) -> GeneratorState<Self::Yield, Self::Return>
    where
        Self: Unpin;
}

let gen = || { yield 1; yield 2; return "foo"; };

for one in gen.by_mut().map_return(drop).take(1) {}
if let Some(two) = gen.next() {}
if let GeneratorState::Complete(final) = gen.resume_unpin() {}

[newpavlov]

Please, read the linked proposal (and ideally discussion). In addition to motivation and explanations why I consider implementing iterator traits for generators a wrong move (but you still can use a wrapper which will transform Generator into Iterator by discarding return value), it also includes a rough description how for loop expansion can be changed in a backwards compatible way (though it relies on negative trait bounds).

[Nemo157]

Ok, sorry for assuming that it would be based around converting Generator -> Iterator. If you’re changing the for loop expansion then it should be possible to make it so that pinning is integrated into it better anyway, I doubt it would be necessary to manually wrap the Generator in a pinned reference (presumably by just swapping generator.resume() with Pin::new(&mut generator).resume() in the expansion so that users must supply a IntoGenerator<Generator: Unpin> value).

[newpavlov]

Hm, you are right, pinning as part of expansion could indeed work. Hopefully manual resume calls on non-self-referential generators will be rare enough for unwieldness of Pin::new(&mut generator).resume() to matter.

[earthengine]

The new std::future::Future common combinators are mush more complicated than I thought. Here is another one join_on_ok will run two Result returning Futures in parallel, and stop if either of then results in Err, or both of them results in Ok.

With the EitherOr defined before, I wrote

pub fn join_on_ok<T1, T2, E1, E2>(
    f1: impl Future<Output = Result<T1, E1>>,
    f2: impl Future<Output = Result<T2, E2>>,
) -> impl Future<Output = EitherOr<Result<T1,E1>,Result<T2,E2>>> {
    struct JoinFuture<T1, T2, F1, F2>(F1, F2, Option<T1>, Option<T2>);
    impl<T1, T2, E1, E2, F1, F2> Future for JoinFuture<T1, T2, F1, F2>
    where
        F1: Future<Output = Result<T1, E1>>,
        F2: Future<Output = Result<T2, E2>>,
    {
        type Output = EitherOr<Result<T1,E1>,Result<T2,E2>>;
        fn poll(self: Pin<&mut Self>, lw: &LocalWaker) -> Poll<Self::Output> {
            let this = unsafe {
                let this = self.get_unchecked_mut();
                (
                    Pin::new_unchecked(&mut this.0),
                    Pin::new_unchecked(&mut this.1),
                    &mut this.2,
                    &mut this.3,
                )
            };
            match (&this.2, &this.3) {
                (Some(_), Some(_)) => unreachable!(),
                (Some(_), _) => match this.1.poll(lw) {
                    Ready(Ok(v2)) => Ready(EitherOr::Both(Ok(this.2.take().unwrap()), Ok(v2))),
                    Ready(Err(e2)) => Ready(EitherOr::Both(Ok(this.2.take().unwrap()), Err(e2))),
                    _ => Pending,
                },
                (_, Some(_)) => match this.0.poll(lw) {
                    Ready(Ok(v1)) => Ready(EitherOr::Both(Ok(v1), Ok(this.3.take().unwrap()))),
                    Ready(Err(e1)) => Ready(EitherOr::Both(Err(e1), Ok(this.3.take().unwrap()))),
                    _ => Pending,
                },
                _ => match (this.0.poll(lw), this.1.poll(lw)) {
                    (Ready(Ok(v1)), Ready(Ok(v2))) => Ready(EitherOr::Both(Ok(v1), Ok(v2))),
                    (Ready(Err(e1)), Ready(Ok(v2))) => Ready(EitherOr::Both(Err(e1), Ok(v2))),
                    (Ready(Ok(v1)), Ready(Err(e2))) => Ready(EitherOr::Both(Ok(v1), Err(e2))),
                    (Ready(Err(e1)), Ready(Err(e2))) => Ready(EitherOr::Both(Err(e1), Err(e2))),
                    (Ready(Ok(v1)), Pending) => {
                        *this.2 = Some(v1);
                        Pending
                    },
                    (Ready(Err(e1)), _) => Ready(EitherOr::This(Err(e1))),
                    (Pending, Ready(Ok(v2))) => {
                        *this.3 = Some(v2);
                        Pending
                    },
                    (_, Ready(Err(e2))) => Ready(EitherOr::That(Err(e2))),
                    _ => Pending,
                },
            }
        }
    }
    JoinFuture(f1, f2, None, None)
}

This combinator is natually appear when I have to proxy connection. In such a case I hold 2 seperate data transfering futures in the form of source -> proxy -> target, and if either futures are stopped I need to stop the other one as well, or they can both close peacefully.

Right now, I identified 4 combinators that cannot be written in await! form, and requires manual implementing:

1. join: run two Futures in parallel, stop if both stops.
2. alt: run two Futures in parrel, stop if either or both of them stops.
3. select, a varition of alt, where will be bias to one of the Futures, such that other Futures wil not be polled in the same circle.
4. join_on_ok: a variation of join such that the outputs are Results, and the stop criteria is either of the futures fail or both success.