Moves from `self` during the drop hook

[aidancully]

Some of the discussion on my linear types RFC seems locked on the semantics of the drop hook. I am hoping the discussion can move here, since this seems to be a preferred forum for fleshing out design details. Also, any change to the Drop interface should hopefully be incorporated before 1.0, so I’d hope it would be considered soon-ish.

Basically, I’d like to see it be possible to move data out of self during the drop hook. I believe there is nothing unsafe about moving data out of self during drop, and there are occasionally large benefits to doing so. One such benefit is that it is a key component of the linear types RFC, but there are others. Consider this proof-of-concept showing how allowing partial moves from self during drop could be used to enforce that a memory block allocated from a pool be returned to the same pool from which it was allocated. This type of pattern is greatly simplified by allowing partial moves from self during drop.

While there are (I believe) significant benefits to allowing partial moves from drop, we’d still need to find the right mechanism to make the best design possible, given the design goals of the language. There are a couple of ideas floating around.

First, the mechanisms (ignore the names, for now, I’ll talk about them below):

1. (From my proposal) Define a new DropPtr type (the name should be bike-shedded), which is the pointer-equivalent of a compound type that does not have a Drop property, and change the drop hook to take a self argument of this type. Since partial moves are allowed from a compound-type that does not implement Drop, they should also be allowed from DropPtr. Dropping a DropPtr behaves identically to dropping a compound variable that did not implement Drop: any partially-moved fields are not dropped, while any fields that remain in the container get dropped as normal. The referent of the DropPtr can be moved, so long as no fields have been partially dropped. (TBD: does this work? It could be used to enable recursion in the drop hook, but might take extra work to do so… and maybe recursion in the drop hook can be desirable, sometimes?)
2. Change drop to take an &move self pointer, or even to take self by value. Require that self be destructed in the body of the drop hook to prevent recursively invoking the drop hook from within the drop hook. (@Ericson2314, please correct me if I misunderstand this part of the proposal, I fear I’m unable to do it justice.)

Then there are the parts of the proposals centering around API names:

1. Regarding proposal #1 above, should DropPtr be called DropFields, or NoDropHookMovePtr, or what? I am having a hard time identifying the “best” name for the concept.
2. The current Drop trait could be renamed to Destroy, and similar for the drop method in the trait. That frees up Drop to mean “can be implicitly dropped” (which would be nice for linear-types support, since that would make linear types !Drop, and also has a nice relationship with Copy and Move, which are other traits where the compiler itself would treat variables of these types differently), and probably better captures the idea that these types have destructors than Drop does.

Thoughts?

[aidancully]

I’m hoping this can get some attention from someone with some decision-making power in the language definition… Let me put my cards on the table:

• I would really like to see the ability to move values out of the drop pointer at some point.
• I think the way that would best fit in Rust involves changing the signature of the drop hook. (I’d love to be proven wrong on this point.)
• If so, then supporting moves from self during drop in the future will create a backwards-compatibility issue today.

The eventual design I’d like to see is:

• Define a DropPtr<T> type, which will behave like &move T would, but acting as though the referent does not implement Drop (so that partial- and full-moves from the referent are allowed). However, memory for the referent will be cleaned up after the pointer goes out of scope. Moves require copying.
• Change the argument to Drop::drop to be of DropPtr<T> type.
• Drop glue for a variable of type T essentially casts the address of the variable to a DropPtr<T>, invokes the drop hook, then cleans up the memory used by the variable.
• Interaction with dropping a future &move T will be similar, but fleshing this part out further will require a specification of how &move T works, and I haven’t seen anything very concrete, yet.

But I wouldn’t try to get there all at once. First, I’d try to do the time-critical part of solving the forwards-compatibility problem with a future full DropPtr<T> implementation. (Generally, I’m trying to stay aware of the language’s push to 1.0, and am really trying not to ask for anything to be done that adds to that queue… but the only current-Rust-compatible alternatives I’ve come up with feel bolted on, or otherwise inelegant, which I assume is also something the maintainers want to avoid, hence this discussion.) Here’s how I’d do that:

• Define the DropPtr<T> type as the drop_ptr language-item.
• Modify the drop hook to take DropPtr<Self> instead of &mut self, and the drop glue to make sure an argument of DropPtr<T> is passed instead of &mut self.
• DropPtr<T> will be castable to &mut self, and will generally behave as &mut self within a function.
• DropPtr<T> cannot be moved, it will only be used by the drop hook.

This is, I think, the bare minimum to solve the forwards-compatibility issue with a future DropPtr<T> implementation. While it would introduce some ugliness to the language (it defines a non-first-class pointer-type), most (and I think all) of the ugliness is temporary, since it should only be temporary that DropPtr<T> would be a second-class construct.

I am also sympathetic to renaming Drop to Destroy, and to changing the meaning of Drop to mean “can be implicitly dropped” – if I were to start Rust over again from scratch, I’d probably make that change. But I think the language is so close to 1.0 that I can’t personally push very hard on that point. @Ericson2314, I’ll leave that argument to you.

I’ll try to put this plan into a standalone RFC. I hope it can get some attention from someone in a position to give some indication if this idea might be desirable to the project maintainers…

[eddyb]

I didn’t have time to write this up anywhere, but I came up with a simpler system a few days ago:

#[lang="interior"]
struct Interior<T>(T);

impl<T> Interior<T> {
    unsafe fn of(x: T) -> Interior<T> {
        Interior(x)
    }
}

#[lang="drop"]
trait Drop {
    fn drop(self: Interior<Self>);
}

Interior<T> would provide field access and pattern-matching as if it were T, but otherwise act like a different type entirely, e.g. it has none of the methods implemented on T.

It wouldn’t have Drop implemented on it, so dropping Interior<T> will only destroy the fields, but won’t call T's Drop implementation, solving the infinite drop recursion issue.

We can also remove the restriction on calling Drop::drop directly, since it needs Interior<T> and that’s unsafe to get from T.

Personally, I think DropPtr is an unnecessary complication and I won’t support it.

Linear types don’t need the ability, but I am sympathetic to finally fixing Drop, if we don’t make it worse.

cc @nikomatsakis

[aidancully]

It looks like you have the drop-argument passed-by-value? Wouldn’t that cause a performance regression, if trying to drop a large structure? Seems like what you’d want is:

trait Drop {
    fn drop(self: &move Interior<Self>);
}

But then that would require &move pointers pre-1.0… One of the goals of DropPtr<T> was to solve the forwards-compatibility problem with eventually supporting move-from-self-during-drop, while not actually requiring a large change pre-1.0. &move pointers feel like a large change that would otherwise not be required, yet… Also, I don’t yet see the essential difference between &move Interior<T> and the full design of DropPtr<T>, and I’d probably be comfortable with a future version of the language having DropPtr<T> be a type-alias for &move Interior<T>, at least if you’re amenable to as and as_mut methods in Interior:

impl<T> Interior<T> {
  ...
  fn as<'a>(&'a self) -> &'a T { ... }
  fn as_mut<'a>(&'a mut self) -> &'a mut T { ... }
}

Still, I care more about the feature than the specific technique that allows it. If Interior<T> gets blessed, that’s fine for me.

[eddyb]

No, by value wouldn’t be a performance regression, passing by value is as optimal as you can get, at least when transferring ownership - for values larger than a pointer it should be identical to what we have today, for something smaller it can be more efficient without optimizations chewing through double indirection (dropping Box<T> involves calling a function which takes T** to a stack slot containing the Box<T>).

AFAICT, as_mut wouldn’t be safe, as you could replace the contents with a dummy and do some forgetting that way.

[rkjnsn]

I’d like to see immovable types implemented some day, and having drop work by-value would seem to prevent that.

[aidancully]

After IRC discussion, I think we now agree that as_mut is not a safety issue. It would also be possible to implement what I called as and as_mut in the Deref and DerefMut traits, respectively. I think we’ve basically got a working idea. I still sort-of prefer DropPtr to Internal, since the stop-gap version is a lighter lift for a 1.0 release, and it stabilizes the syntax in a forward-compatible way that allows a better design more time to evolve even after 1.0. (I don’t think there’s any behavioral difference between my DropPtr proposal and &move Internal.) But I don’t lean heavily, and will be happy with any solution that gets the project’s blessing.

[Ericson2314]

I was a bit concerned when the linear types RFC was closed.

Some form of consuming drop is wholly better than the status quo, and I would really like to see that happen before it is to late. Don’t care super much about @eddy’s plan vs my own. They both consume and I see that as the important part. [When I first heard it on IRC, I thought @eddyb was proposing that Interior<T> have T's fields without T—this misunderstanding of mine led me to deem his plan less elegant.]

I worry not somehow addressing asymmetry between Drop and Clone+Copy now will muddle the language until 2.0, but at least linearity can still be made the core-language default down the road with an opt-out trait like Size.

[aidancully]

@eddyb, what about calling DropPtr InteriorPtr?

[eddyb]

Even though I’m not sure what you want is feasible in Rust, the only thing Interior<T> would prevent would be calling Drop::drop directly - and you can’t even do that today, at all. The compiler would still be able to call the method just fine.

[rkjnsn]

Are you saying the compiler would be able to call the method without moving the type or copying any bits? Even if so, would there be any way for the drop method to get the address of the object?

One specific use case I have in mind is something like Boost.Interprocess, which lets you store objects in shared memory that refer to one another (which may be mapped to different addresses in different processes). It does this by providing a smart pointer that stores an offset instead of an absolute address. Objects in shared memory can use this pointer type to refer to one another without worrying about where they are mapped. Something like an owning offset pointer couldn’t be implemented in Rust if drop gets called by value, because the offset pointer would have no way to locate its pointee.

The way I see this working is to have a Move trait, implemented by all structs by default, but of which a struct could explicitly opt out. Move would also be a default bound, of which a generic could opt out using ?Move. Such a type could never be used by value. This would require a way to construct an instance directly into a given memory location (&out?), and that drop be called with a reference. Additionally, it my be useful to have a Relocate trait, analogous to Clone, that would provide a relocate method to relocate an immovable type. This could do any necessary fix-ups such as updating the offset of an offset pointer. This would additionally require a consuming reference parameter. (&in? &move?). While something like this obviously can’t be implemented before 1.0, it is potentially very useful, and I would really hate to see it be made impossible.

[pnkfelix]

I actually kind of like @eddyb’s Interior<T> proposal.

Furthermore, one might also make a variant of std::mem::forget, e.g. std::mem::forget_outer, that returns Interior<T>. This would solve a problem I have had recently where, AFAICT, it is not possible to both forget a value and also retain (by move) some of its contents; at least, not without throwing some cells-of-options and doing swap-dances.

(I had originally thought that was could simply change the signature of std::mem::forget<T>(t: T) to return an Interior<T>, but that would change the semantics of std::mem::forget in a big way.)

[eddyb]

I thought of Interior::of for this but I like how forget_outer conveys the semantics involved better.

[aidancully]

This isn’t a fully-formed opinion, by any stretch, but I find I’m more sympathetic today to the idea of having multiple Drop traits:

// strictly worse than other options, but could be preserved
// for backwards-compatibility. should become deprecated.
#[lang="drop_mut"]
trait DropMut {
  fn drop(&mut self);
}
// preferred new implementation.
#[lang="drop_interior"]
trait DropInterior {
  fn drop(self: Interior<Self>);
}
// alternative new implementation. used when you really want to
// ensure that the source hasn't moved.
#[lang="drop_move_interior"]
trait DropMoveInterior {
  fn drop(self: &move Interior<Self>);
}

The compiler would restrict a type to only implement one of these (you couldn’t implement both DropInterior and DropMoveInterior for your type). I had been worried about composability with the compiler supporting different drop schemes at once, but I think things should just work: for DropMut, the drop-glue for members would be inserted outside the drop hook, while for DropInterior, the drop-glue for unmoved members is inserted inside the drop hook. In no case would the memory be cleaned up too early, or would there be other than a single place where the drop glue for a member-field to be inserted. This should end up just working out.

That said, even though this may not be the forward-compatibility issue I was worried about, I still think current Drop should be changed pre-1.0. (Or perhaps current Drop should have the stable attribute revoked? That would be very annoying without a replacement available, but may be more accurate if the concerns raised in this thread are taken seriously, while not blocking the release of 1.0.) Otherwise, the compiler would have to carry around the DropMut baggage until Rust 2.0.

[Ericson2314]

swap-dances

That made me chuckle.

@rkjnsn &move for drop would be what you want. [&move and &out are duals, so I like your &in name]. One use-case today is locks, which are currently boxed under the hood so the underlying box is not moved. Limited CTFE + immovable types + no funny static drop restriction (which I think follows from consuming drop) is the triple-threat need to remove the need for Static- types. I’m quite sympathetic, but since it requires a new type of pointer I doubt this can be done before 1.0, whereas the other options can.

@eddyb @pnkfelix I guess am wondering what makes adding the magic field access to Interior<T> preferable to a lint against recursive drop with drop consuming T, insofar as there will be a little ad-hoc magic either way?

@aidancully I fear mixing non-consuming and consuming drop will lead to problems and/or confusion. But the fact that &move is probably the best if we ever support non-movable types leads me to think there is no hope of being indefinitely forward-compatible today anyways.

[rkjnsn]

since it requires a new type of pointer I doubt this can be done before 1.0

Right, I realize a general consuming reference parameter type is likely out of scope for 1.0. I’m mainly arguing against making drop consume by value. I like the idea of Interior in general, so it seems like &in Interior<Self> would be ideal in the future. In the mean time, I think @aidancully’s suggestion of a special-case InteriorPtr is the way to go. This way, &in (or &move) can be introduced and used for drop in the future without breaking backward compatibility (InteriorPtr would become a deprecated alternative spelling).

[aidancully]

@aidancully I fear mixing non-consuming and consuming drop will lead to problems and/or confusion. But the fact that &move is probably the best if we ever support non-movable types leads me to think there is no hope of being indefinitely forward-compatible today anyways.

I think InternalPtr achieves forward-compatibility. It gives the least powerful option available (&mut) a forward-looking name (InternalPtr), while restricting the use of this name to the only place where there is a forward-compatibility problem (Drop::drop), so that it’s impossible for user-code to rely on the desired eventual semantics of this new pointer type before they are implemented. This is compatible with current semantics, and is expandable to &move Internal in the future. It’s definitely ugly, but I think it’s perfectly forward-compatible…

[pnkfelix]

@Ericson2314 With either option, you need some sort of magic. I thnk you are under-estimating the amount of magic necessary to support drop-consuming T properly.

You are either going to impose rules like "you cannot move out of self in the context of fn drop" (which to me seems much the same as Interior<T>, or you are going to need some sort of ad-hoc analysis to determine which instances of T should not have their destructor invoked.

As a quick example scenario, consider a destructor for T that can create other T’s:

#[derive(Debug)]
struct Tick { countdown: u32 }

impl Drop for Tick {
    fn drop(&mut self) {
        let t;
        let s = self;
        println!("dropping {:?}", s) {
        if s.countdown > 0 {
            t = Tick { countdown: s.countdown - 1 };
            println!("created {:?}", t);
        }
        println!("done dropping {:?}", s);
    }
}

fn main() {
    let _spoon = Tick { countdown: 3 };
}

In the above code, if we solely changed the signature to fn drop(self) and made no other changes, then you would need an analysis to distinguish cases like t whose destructor should be invoked, versus cases like s where doing so would cause infinite regress. (The easiest way to eliminate the above scenario is to disallow moves out of self within fn drop, but again, what’s the difference between that rule and adding Interior<T> as an explicit form of it?

[eddyb]

Are we willing to guarantee that drops won’t ever copy any memory around? Lots of things use ptr::read right now, and even if they didn’t, you still have things like mem::replace all over the place (think Option::take).

@pnkfelix It seems to me that adding “immovable types” isn’t backwards compatible and just Interior<T> is a much simpler solution than some form of &move Interior<T> (which requires actually designing a sufficient &move-like system) and cleaner than a special type which behaves like the latter.

I wouldn’t mind a special pointer-like type it very much if we have to do it for future-proofing, but I’d like to avoid it.

[aidancully]

I had a small realization (that may have been obvious to others here): The type-definition defines the Interior type, while implementing Drop creates an aliasing, zero-overhead, decorator of the Interior type. In other words, if we removed the “aliasing” function from Drop, we’d get something like the following:

struct DropType<T>(T);
struct Foo;
impl Drop for DropType<Foo> {
  fn drop(self: &mut Foo) {
  }
}
let x = Foo; // equivalent to `forget_outer(Foo)`
let x = DropType(Foo); // equivalent to current `Foo` construction.

Obviously that isn’t something we’d want to read or write in normal Rust code, the aliasing is very important for ergonomics and safety. But if we consider implementing Drop to define an aliasing decorator for the Internal type, the following sorts of things become possible (again, not code we’d actually write, just trying to demonstrate the idea):

struct DropValueType<T>(T);
trait DropValue<T> {
  fn drop_value(self);
}
impl<T: DropMut> DropValue for DropValueType<T> {
  #[inline]
  fn drop_value(self) {
    self.0.drop();
    forget(self);
  }
}
struct DropMoveType<T>(T);
trait DropMove<T> {
  fn drop_move(&move self);
}
impl<T: DropValue> DropMove for DropMoveType<T> {
  #[inline]
  fn drop_move(&move self) {
    self.0.drop_value();
    forget(self);
  }
}

With aliasing, it’d be considered that implemeting Drop for a type creates an aliasing Drop decorator, which then gets decorated by an aliasing DropValue type, which then gets decorated by an aliasing DropMove type. And now the drop-glue insertion only needs to consider the DropMove case - the rest is handled by the generic implementation of drop_move forwarding to drop_value, and drop_value forwarding to drop. I think this presentation shows that composability is handled just fine if multiple drop-traits are eventually supported… Using multiple drop-traits will also avoid code-churn for a large body of existing code, allow forward-compatibility, and allow drop-by-value optimization as well as immovable types (if and when they’re added).

That said, even though I don’t think there’s a forward-compatibility problem, I still think current Drop should be removed in favor of @eddyb’s Drop, if that can be made to fit within the project’s release time-line. Current Drop is strictly worse, and it’d be a shame to keep it forever. If immovable types are added in the future, they can be supported with a new DropMove trait, and that could be implemented using the mechanism I’ve described here.