`*move` raw pointers

language design
1

Over the years, there has been ample discussion on the concept of a "&move"/"&own"/"move reference"/"owning reference". However, its raw pointer cousin has been comparatively neglected. This topic aims to correct that shortfall.

For & (reference to value you don't own and can't modify), we have *const. For &mut (reference to value you don't own but can modify), we have *mut. But for Box, Vec, Arc, and all the other safe-code references to values you own… we have, erm, NonNull? Except that's not a real pointer type, so you have to call .as_ptr() before using it (and no convenient coercions!). To construct it, you need new().unwrap() (the concerns of "pointer to value you own" and "pointer guaranteed to be non-null" are sadly not separated).

In practice, when writing unsafe code to manage owned memory, one must use an awkward mix of NonNull (correctly represents semantic intent, but inconvenient to use in practice), *const (a real raw pointer type, and with the correct variance, but you can't write to it, and it conveys the wrong semantic intent), and *mut (a real raw pointer you can write to, but with the wrong variance, and semantic intent not fully correct). Unsafe code is tricky enough without this extra hassle; can we do better?

*move (*own?) would be a raw pointer type with the following characteristics:

  • Covariant
  • Allows writes
  • Implicitly coerces to *mut and *const
  • Constructed from a place via addr_of_move!(place), which requires the place to be something you can move out of (so addr_of_move!(*(&mut foo)) doesn't work for example). This macro would in fact logically perform such a move; the place would no longer be considered accessible by the borrow checker, and drop_in_place would not be called on it.
    • No construction via coercion from & or &mut
    • As with the other pointer types, can also be constructed via cast from a pointer with different mutability
  • Basis for NonNull
  • The raw pointer type of choice for dealing with owned memory
    • Backward compatibility will be an issue, the standard library is full of APIs like Box::into_raw() that use *mut as a substitute for *move.

In my Notes on partial borrows a few months ago, I observed that if &mut T were to be made a subtype of &T (as it is logically), Rust's variance rules would need to handle such mutability-based subtyping differently from lifetime-based subtyping. This is because &&mut T is logically a supertype of &&T; &T is contravariant with respect to the mutability of T. However, Box<T> is logically covariant with respect to T's mutability, just as it is with respect to T's lifetime. In a hypothetical Rust with both mutability variance and *move, *const T would be contravariant[1] with respect to Ts mutability, but *move T would be covariant. In my view, this further demonstrates that *move is a distinct concept that deserves its own type.

  1. in practice it would probably be invariant, due to backward compatibility concerns ↩︎

2 Likes
2

I have a lot to say about &move and *mut, but it's getting late, so I'll have to remember to do it tomorrow. But basically,

  • *move is unrelated to NonNull. Nullability and move capability are orthogonal. The proper analogue is the perma-unstable Unique pointer type, which powers Box and Vec.
  • The variance must be the same as *mut, since *move T can be both read and written. Also, &move is a subtype of &mut in the same sense as &mut T <: &T, thus it cannot provide more relaxed guarantees, including variance.
  • *move T is the proper solution to #[may_dangle] --- it is (almost) a universal type having that property.
  • It must have mostly the same magic as &move, since it's the unsafe pointer powering all safe operations with &move.
  • The latter two points are also the reason it must be a separate type and cannot be modelled with *mut and a bit of careful programming. Variance isn't a good enough reason to add a new pointer type.
1 Like
3

Agreed, and I even state this in the post, but

does not follow. From the Rustdoc:

Unlike *mut T, NonNull<T> was chosen to be covariant over T. This makes it possible to use NonNull when building covariant types [...]

Covariance is correct for most safe abstractions, such as Box, Rc, Arc, Vec, and LinkedList. This is the case because they provide a public API that follows the normal shared XOR mutable rules of Rust.

It's NonNull's role as a covariant mutable pointer that *move would replace, not the actual "non-null" aspect of it.

Box<T> can be read from and written to, and is covariant. Being the raw-pointer version of Box et al is the primary motivation for *move.

&move/*move imply ownership, and therefore responsibility for dropping; & and &mut do not. Therefore, no subtyping relationship can be derived between them.

2 Likes
4

For the name, I think *own is a better option, for the reference as well. It conveys more meaning.

I think that new pointer types should be non-null by default, with Option<ptr> as the nullable version. Even in unsafe code, null pointers are the exception, not the norm. The only case null pointers are commonly used in my experience is FFI, and Option<ptr> pretty much always models those cases perfectly. I'm fact, I'd like to see all pointer types transition to this model across an edition.

3 Likes
5

If there were no backward compatibility considerations, I think this would be a no-brainer. But given that they do exist, I wonder how much we can do with pattern types and a non-null pattern.

2 Likes
6

Very much against this. It would mean that if I declare an FFI function which takes a *(const/mut/move) T and someone passes in a null pointer, I get immediate UB. That is much worse than getting UB on the pointer's dereference, which may not even happen in my code. How is an immediate hidden UB better than an UB on an explicit unsafe operation? Besides, nullability isn't even the only pointer validity guarantee, nor the most important one. Neither alignment nor liveness can be properly encoded in the typesystem anyway, nor could they be enforced over FFI even if Rust types allowed it.

Null pointers are also super common in C/C++ code. Plenty of functions take or return null as some sentinel value. Overall this means that pointer types would be unsuitable for FFI, and would have to be exclusively used in the Option<ptr> form, which is a significant hit to the ergonomics, and goes against the (natural, and encouraged) assumption that raw pointers are the same as C pointers.

This is already an annoyance and footgun for function pointers, but at least in that case there are strong reasons, both for safety and target architecture behaviour, why function pointers should be non-null and entirely different from data pointers. With pure data, it would cause more UB than it would avoid.

2 Likes
7

That might just be an argument for disallowing (or at least linting against) the non-null pointer types from being used in FFI (just as with non repr(C) types currently), thus forcing use of an Option around it instead? Granted there is an ergonomic hit as you mention, however.

4 Likes
8

Covariant and mutable aren't good enough reasons for a new pointer type. NonNull already achieves it, without any changes to the language. It's not quite a pointer, but the difference is, frankly, minor. Again, avoiding a couple of short conversions isn't enough to warrant a new fundamental type. Besides, if pattern types ever get stable, we'll be able to declare non-null type coercible to a pointer purely in library code.

That analogy only goes so far. Unlike Box<T>, which is always guaranteed to be heap-allocated, *move T may point anywhere, including the stack. It would also be a core type, thus available even on #[no_std] systems with no heap allocation.

Thus *move T should be treated as a raw-pointer version of &move T, rather than Box<T>. And &move T has the capability, but not obligation to drop its contents. It's a reference, so one should be able to use it like any other reference type: create and drop without using it, include it in method resolution, create traits like IndexMove, DerefMove and the likes. This means that it must not drop its contents unconditionally, only in specific circumstances. This code should work:

let s = String::new();
let _ = &move s;
dbg!(s);

Similarly, it should be possible to use &move T as &mut T, purely for some local mutations.

let mut s = String::new();
let m = &move s;
*m = String::from("hello");
dbg!(s);

Thus &move T should be able to coerce to &mut T, and to be reborrowed as &mut T. This means that it cannot provide weaker requirements than &mut T, and must be invariant w.r.t. T for the same reasons. *move T, being the unsafe unchecked version of &move T, must also be invariant.

This does introduce an issue when using it inside a Box or Vec. My opinion is that there should be a way to exactly specify the variance of types, and the current approach is a mistake. The result of the current restrictions is that types need to introduce hacky workarounds, like using *const T where *mut T is appropriate, or even storing *mut () and casting to *mut T for actual pointer operations, instead of using the proper pointer type and declaring their variance.

With regards to subtyping, I model the relation between safe references in the following way: there are capabilities Read, Write and Move, which describe everything you can do to the pointed data. Read and Write are self-explanatory. By Move I mean the capability to mark data as initialized or uninitialized (so an alternative name could be Init). This means that the usual move operations is either Read+Move, for moves out of a place, or Write+Move, for moves into the place (or even Read+Write+Move, when we are moving into an occupied place and must drop the old value). The references are thus strictly ordered with respect to their capabilities: &move T <: &mut T <: &T, because &move T: Read + Write + Move, &mut T: Read + Write, &T: Read (assuming no UnsafeCell inside). UnsafeCell, as always, complicates everything, because &UnsafeCell<T>: Write, though it isn't Move.

1 Like
9

We could just as well lint any dereferences on raw pointers which aren't explicitly checked to be non-null.

What's the point of having an option in the language which is always wrong? It should be made to Just Work instead.

10

I don't think it should. When you take a move reference to a place, you move out of that place, and whoever receives the &move is responsible for dropping it. Under your approach, what would the following function do?

fn main(){
    let s = String::from("Hello");
    dbg!(other_func(&move s));
    dbg!(s);
}

fn other_func(input:&move String)->usize{
    let ret = input.len();
    println!("The string \"{}\" has a length of {}.",&*&input,ret);
    drop(*input);
    return ret;
}
2 Likes
11

It would not compile.

Under my approach, there is a difference between simply taking a &move reference, and actually moving it somewhere. A &move T passed into the function passes ownership of the value, so that the backing place cannot be read again (but it can be filled with a new value, or deallocated). A &move T which is just passed around in a single function (between variable bindings, moving into a closure or async {} block would act like a move into a function) is ephemeral and doesn't move the value, in both the sense of a data copy and logical ownership.

This is a necessary requirement if we are ever to support something like DerefMove and custom Box implementations. References in Rust are often created implicitly, via deref coercions and method resolution. If all of those implicit references moved the referent, that would be very, very bad.

It would be also impossible to implement Box-like semantics. Currently the following is valid Rust:

let mut b = Box::<(Foo, Bar)>::new(get_foobar());
drop(b.0);
drop(b.1);
b.0 = get_foo();
b.1 = get_bar();

Note that .0 and .1 are not fields on Box<T>, but rather fields on its pointee. The above code only makes sense if the field access works via a deref-coercion mechanism, like it would with &b.0 and &mut b.0, which use Deref/DerefMut. b.0 should be the syntax sugar for

<Box<T> as DerefMove>::deref_move(&move b /* : &move Box<T> */) // : &move T
    .0  // : &move Foo

just like the current deref coercions. Note that this starts with creating an implicit &move Box<T>. If the mere creation of a &move caused ownership transfer, the above code would never work: drop(bar.0) would also drop b, and all following lines would error as attempts to use a dropped value. Even if the coercions &move Box<T> :> &move T could work by some special magic, the could still wouldn't compile, since drop(b.0) passes through &move (Foo, Bar) and thus would drop the referent, thus also implicitly dropping b.1.

The proposed design basically works this way: &move T doesn't transfer ownership in cases where currently the compiler allows passing partially initialized data, and transfers ownership otherwise. This basically means that ownership is kept when &move T is created, passed through local variables and implicitly dropped at the end of scope. Ownership is transferred when &move T is passed as a function argument, including drop(_), or is used to initialize fields of structs, including implicit ones like closures or async {}, or on unwind (but in that case the referent would be dropped anyway, so perhaps the &move shouldn't be separately dropped anyway).

At that point the natural question is "how can deref_move work in that case since it's a function?". In my opinion it requires a new feature with special borrowcheck-transparent functions, but that's beyond the scope of the current discussion. Alternatively, it could be a special #[lang] item, i.e. the only function which would have this borrowck transparency.

12

While I disagree with your point(let _ = $expr and drop($expr) behaving differently would be unexpected), I think we are getting away from the orrigional proposal i.e *move Pointers. This is a much simpler proposal, and I think the only point of disscution left is that if the theoretical addr_of_move!() should move out of the target place or not.

13

But they do! When $expr is a place expression, binding it to _ does nothing. [playground]

    let v = vec![0];
    let _ = v;
    drop(v); // okay

    let v = vec![0];
    drop(v);
    drop(v); // ~err
1 Like
14

Is there a reason for this behavior? It seems inconsistent with both let w = v and let (a, _) = b.

15

It's because using a pattern of _ is choosing to not bind the value. And it is consistent with the behavior of _ in compound patterns like tuples, e.g.

let b = (vec![0], vec![1]);
let (a, _) = b;
dbg!(b.1); // this is allowed, as we haven't moved from this place

With pattern binding modes its perhaps less necessary, but it's surprisingly prevalent without many people actively realizing, e.g. it's why matches!(o, Some(_)) doesn't move out of o.

That pattern matching works like this at all is also because of binding modes, e.g. let ref w = v doesn't move out of v either, just reference it. _ is then just the no-op binding mode. In the absence of non-moving binding modes, it could be decided to have let $pat = $expr; be equivalent to today's let $pat = {$expr}; and always turn places into temporaries (i.e. move from them), but with them, patterns necessarily operate on places rather than just on values.

6 Likes
16

Nit: Binding modes are those RFC 2005 and beyond mechanics of automatically using ref and ref mut patterns sometimes; _ patterns predate binding modes. I think you just meant that there are a variety of binding patterns (moving, ref, ref mut... no-op I guess[1]).

Here's discussion of the actual decision.

Some things have changed, but note how this still doesn't work with patterns that actually bind.

  1. it's really a non-binding pattern ↩︎

1 Like
17

The terminology I've seen has been that ref/ref mut are the binding modes, and RFC 2005 introduced the use of default binding modes, with ref patterns and friends being explicit binding modes. But otherwise yeah, we're on the same page, and it's possible I misremember. I do recall that RFC introducing the term "binding mode" but that doesn't mean ref isn't still an explicit one.

2 Likes
18

Imagine something like *out T / &out T existed, with semantics like:

  • For the expression &out place, place must not be initialized
  • The types *out T / &out T are contravariant to T
  • A call to a function taking such a type may assume place is initialized once the function returns
  • This, a function must definitely write to such a type before returning

Would that mean this should be named as an antonym of such a feature? Eg *in T?

19

AFAIK the problem with this is that it interacts very poorly when they are captured in closures (nothing guarantees that they will be run and thus the &out T will be initialized) and errors (what if you want to either initialize a &out T or return an error?).

Such references are also tricky to handle when unwinding happens, since the function may not have initialized the &out T yet when that happens.

Also, AFAIK this can also be implemented in a library by using branded lifetimes to link together an Out<'brand, T> with an InitializedProof<'brand, T> which must be returned (I remember seeing a crate that implemented this pattern but can't remember its name). This also resolves most of the problems mentioned because the proof that the out reference has been initialized is a simple value, so it is not returned when unwinding happens and can be conditionally returned in Results.

2 Likes
20

(this was more an example of a nearby feature for comparing this suggestion to, not a suggestion itself, strictly, but sure, I'm game!)

This would be more a feature for matching existing FFI semantics and maybe simplifying optimization of some easy cases of unnecessary initialization when refactoring, I would think?

Do you mean if the reference/pointer is captured? Presumably current definite assignment checking with regular captured uninitialized places just complains, so that should be fine to do with uninitialized out types too.

If the function doesn't return, then it doesn't return, so it doesn't need to / can't be assumed to have written the place!

That library design is pretty neat, if it can enforce it on the callee as well as the caller? (I guess Out just has a write that returns the Proof value with no other way to construct one)