(Mega-pre-RFC) Reference specialization types (DSTs, proxy-references)

[drXor]

Besides the compiler-generated impls for trait objects and slices, the primary feature of CoerceUnsized is that it’s a coercion (as the name says), i.e., automatically and implicitly applied. Many people don’t want that for expensive conversions such as Vec<T> -> Box<[T]> (which in general needs to reallocate) or String -> Rc<str> (which always needs to reallocate).

Hmm, that’s a good point, though I was pretty sure CoerceUnsized allowed for casts, rather than coercions (oops, I guess). I feel like this could be resolved with some sort of general version of unsizing? That said, I’m not convinced that RSTs are a clean solution, compared to custom DSTs.

[duanebyer]

I’ve got two motivating cases for RSTs over just DSTs. Both of them are related to proxy-references.

• Example #1 (playground link)
• Example #2 (playground link)

Is there a way to write these examples with just DSTs? I don’t think there is, but maybe I’m underestimating how powerful they are.

[Ixrec]

I think the motivation is skipping a few steps. It’s obvious that there are types which are arguably conceptually analogous to references, but do not physically contain a pointer or reference to the actual sized or dynamically-sized type they conceptually refer to. To keep things tractably concrete, let’s narrow that to “offset-as-reference types” or OaRTs, because that seems to be the most obvious and compelling case (and all of the examples you’ve given so far have been OaRTs). I’m also more or less convinced that if we decided Rust must support OaRTs with all the same magic rules that actual references get, we’d need a mechanism similar to this RST proposal.

What I have not seen an argument for is the premise that there’s a need for OaRTs to share the same magic as built-in references. More generally, it’s not at all clear to me that OaRTs should try to behave as much like references as possible. I would’ve assumed they should behave like the offsets/indexes that they actually are.

So perhaps we should take a few steps back and look at examples of the kind of client/application code you’re trying to make possible with reference analogues/RSTs/etc, not just RST implementation code?

[duanebyer]

I agree that the motivation may not be as fleshed out as it should be. The main reason that OaRTs should have reference magic is:

• OaRTs cannot derive traits properly. Imagine trying to implement common traits such as Deref for a container into its OaRT, or Index/IndexMut. The problem is that these traits have methods that take by reference &self/&T, but for an OaRT, we want to take by value (self/T).
• OaRTs cannot be passed to certain generic functions that take reference arguments. The most obvious examples are ones that take two arguments &T and &mut T. See also, functions that take multiple references with different lifetimes. This also affects trait implementations (see below).

In Rust, imagine I have to deal with a trait like the following:

trait Storage {
    type Item;
    fn get(&self) -> &Item;
    fn get_mut(&mut self) -> &mut Item;
}

How can I implement this for my custom storage type with its OaRTs ItemRef and ItemRefMut without reference magic?

So perhaps we should take a few steps back and look at examples of the kind of client/application code you’re trying to make possible with reference analogues/RSTs/etc, not just RST implementation code?

The purpose of this RST isn’t primarily for client/application code, but for libraries. C++ supports proxy-references (although not very well, and it’s probably bad style to use them). Here’s a C++ octree library that I wrote and have up on Github that you can take a look at. I resorted to using proxy-references because I had a hard restrictions on the data layout of the octree, but I wanted to meet certain C++ collection concepts (similar to implementing a Rust trait). If C++ didn’t have proxy references, I would have been out of luck.

If Rust ever adds collection traits, the same issue is going to arise then. For now things are workable but ugly, but they might not always be workable at all.

[duanebyer]

A DST pointer can be cast to a thin pointer:

How does this work with RSTs?

I guess such a cast would have to be disallowed. Since its looking like RSTs have to be a super-set of DSTs, maybe it’s ok? Something else I will have to think about…

[bugaevc]

Note: I’ve made a similar proposal on the now closed RFC (view it on GitHub desktop, the mobile version hides some comments when they’re many of them), and it was rejected because of complications with other unsized references that the one we’re defining.

[kennytm]

Question 2.

References &'a T and &'a mut T nowadays are covariant in 'a, e.g. the following is always legal:

let a: &'static T = ...;
let b: &'a T = a;

The RSTs definition did not mention how the lifetime will be constrained, making it possible to force &'a T to be invariant:

struct &'a Rst {
    wrapped_ref_1: &'a Type1,
    invariance_marker: PhantomData<*mut &'a ()>,
}

How will the RFC deal with this? Use “pointer equivalence”?

[rkruppe]

I second @Ixrec’s sentiment, while “offsets” are a reasonable and common way to have a handle into a collection, it’s not clear why those should be disguised as bona fide references. You mention conformance with existing interfaces as a requirement, but these “RSTs” are so substantially different from actual references that I expect that lots of generic code will run into problems if you throw a handle-masquerading-as-reference at it. To give just a few examples:

• with references it’s generally reasonable (ZSTs excluded) to cast references to thin raw pointers to e.g. get a unique address to use e.g. as a key for a side table. apparently that won’t work with RSTs
• if you have two &muts you can swap their contents just by knowing the size of the referent and copying memory around (e.g., given your Storage trait, one can write mem::swap(store1.get_mut(), store2.get_mut()))
• if you have any reference at all, you can call size_of_val and align_of_val on it, but for handles that generally doesn’t make sense
• even things like the desugaring of a[b] runs into problems. It currently expands to *a.index(b) or *a.index_mut(b), but if ´index` can return an RST, what does that even mean?

While the details are Rust-specific, the same problem exists in C++. It’s more manageable there because it has copy & move constructors, overloaded assignment operators, etc. but even then the difference is considered severe enough that (for example) it’s considered a mistake that std::vector<bool> is a bit vector that hands out handles.

[Ixrec]

I guess I need to be a bit more blunt and/or repetitive to convey what I meant by “take a few steps back”.

---

The main reason that OaRTs should have reference magic is:

• OaRTs cannot derive traits properly. Imagine trying to implement common traits such as Deref for a container into its OaRT, or Index / IndexMut . The problem is that these traits have methods that take by reference &self / &T , but for an OaRT, we want to take by value ( self / T ).
• OaRTs cannot be passed to certain generic functions that take reference arguments. The most obvious examples are ones that take two arguments &T and &mut T . See also, functions that take multiple references with different lifetimes. This also affects trait implementations (see below).

This is begging the question. I’m asking why do we want OaRTs to derive Deref or be passed to functions that are generic over references in the first place? What sort of generic functions that take references or Deref types would “just work” with OaRTs in a genuinely useful way if we added RSTs?

How can I implement this for my custom storage type with its OaRTs ItemRef and ItemRefMut without reference magic?

Again: Why is that a trait you want OaRTs to implement in the first place?

The purpose of this RST isn’t primarily for client/application code, but for libraries.

I wasn’t questioning where RST code would exist. Obviously it’d be a library thing if we ever added it. I’m saying that we need some examples of client/application code that would benefit from libraries with RSTs in order to provide any convincing motivation for RSTs themselves.

I resorted to using proxy-references because I had a hard restrictions on the data layout of the octree, but I wanted to meet certain C++ collection concepts (similar to implementing a Rust trait). If C++ didn’t have proxy references, I would have been out of luck.

Again: Why did you want to meet those “certain C++ collection concepts”? How does your library meeting those concepts benefit client/application code using your library?

---

Those are all just slightly different ways of re-asking the central question of my previous post (and @rkruppe’s new one): Why do we want Offset-as-Reference Types to act like references instead of acting like offsets? Hopefully it’s a little clearer what I meant now.

[ExpHP]

How does your library meeting those concepts benefit client/application code using your library?

This question baffles me. Where is the line drawn on what’s “client/application” code? If I have some convoluted and duplicated logic in my application and factor it out into a general algorithm, is that application code? If I have a collection type that is optimized for the use cases that appear in my crate, but do not publish it separately, is that application code?

Just last week I was revisiting some old Haskell code that uses Data.Vector.Unboxed to limit memory consumption and improve performance. I had to revise some helper functions for reading and doing math with these types. Many times I got bitten by the fact that Unboxed vectors are not Functors, meaning that using them requires one to throw away years of utility functions that otherwise work on every type known to man.

I feel like this question is asking, why have abstractions? Reasons for wanting abstractions can be found anywhere in any code base. Who cares what they are?

The more meaningful questions to me are:

• For how many types does this enable the use of standard library abstractions? As I see it, the number is comparatively few; most of my own use cases are met by existing custom DSTs, albeit awkwardly.
• How significant are the abstractions? Deref is pretty fundamental.
• What is the technical cost in language and compiler complexity? Recent discussion suggests to me that this outweighs the benefits.
• What future opportunities for abstraction are closed by it? This I don’t know.

[illustrious-you]

I feel like this question is asking, why have abstractions? Reasons for wanting abstractions can be found anywhere in any code base. Who cares what they are?

The reasons for wanting an abstraction usually provide insight into what’s wrong with the existing abstractions. As pointed out in aturon’s latest blog post, understanding the reasons why we want some solution is a key means to drive towards consensus:

Going back to mod statements, several people talked about the role they play in their personal workflow, whether due to their IDE, their lack of IDE, their habits with respect to temporary files, or even the latency of their file system.

No one can be wrong about their own lived experience. And lived experience can bring issues to life in a way that pure empathy and speculation can’t. Thus, a big benefit of the RFC process is the crowdsourcing of lived experiences that it provides.

Far from being irrelevant, those details are the very thing this process shines a light upon.

[duanebyer]

Yeah, I think all of you are right. I’ve been trying to think of a good example of something that RSTs make possible, but haven’t been able to find one. In my defence, I think that I was pretty clear in the RFC that almost nothing new becomes possible with RSTs. Since Rust makes a habit of splitting mutable/immutable versions of a trait (like Index and IndexMut), I don’t think there’s going to be a compelling reason for RSTs anytime soon.

My current plan is to update the original RFC in a few days with some of what I’ve learned from this thread, but after that I will not be moving forward with this.

Thanks again everyone (especially @rkruppe) for your insightful comments. I hope that some of you found something valuable in this discussion (I definitely did at least).

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.