Having mutability in several views of a struct

[nakacristo]

I have been thinking the following idea for some time and I thought I would share it here, since I have not seen discussed elsewhere.

The idea is to be able to split a struct into views such that diferent functions have limited access to the fields. And more importly we could split mutable access to a struct into mutable accesses to disjoint views of the struct.

I think the following code shows this clearly

struct Big
{
	x: usize,
}

struct Composite
{
	a: Big,
	b: Big,
	c: Big,
	d: Big,
}

//We declare different views of the struct
view CompositeAB for Composite(a,b);
view CompositeAC for Composite(a,c);
view CompositeAD for Composite(a,d);
view CompositeBC for Composite(b,c);
view CompositeBD for Composite(b,d);
view CompositeCD for Composite(c,d);

//Each view could have their own implementations
impl CompositeAB
{
	fn compute(&mut self)
	{
		self.a.x+=self.b.x;
		self.b.x*=self.a.x;
		self.a.x+=self.b.x;
	}
}

impl CompositeAC
{
	fn compute(&mut self)
	{
		self.a.x=(self.a+1)*(self.c.x+1);
		self.c.x=(self.a+2)*(self.c.x+2);
	}
}

impl CompositeAD
{
	fn compute(&mut self)
	{
		self.a.x=(self.a+2)*(self.d.x+1);
		self.d.x=(self.a+3)*(self.d.x+2);
	}
}

impl CompositeBC
{
	fn compute(&mut self)
	{
		self.b.x=(self.b+3)*(self.c.x+1);
		self.c.x=(self.b+4)*(self.c.x+2);
	}
}

impl CompositeBD
{
	fn compute(&mut self)
	{
		self.b.x=(self.b+4)*(self.d.x+1);
		self.d.x=(self.b+5)*(self.d.x+2);
	}
}

impl CompositeCD
{
	fn compute(&mut self)
	{
		self.c.x=(self.c+5)*(self.d.x+1);
		self.d.x=(self.c+6)*(self.d.x+2);
	}
}

fn main()
{
	let mut thing = Composite::new();
	{
		//We could get mutable references to disjoint views
		let ab = CompositeAB(&mut thing);
		let cd = CompositeCD(&mut thing);
		crossbeam::scope( move |scope| {
			scope.spawn(move || {
				ab.compute();
			});
			scope.spawn(move || {
				cd.compute();
			});
		});
	}
	{
		//To another combination of views
		let ac = CompositeAC(&mut thing);
		let bd = CompositeBD(&mut thing);
		crossbeam::scope( move |scope| {
			scope.spawn(move || {
				ac.compute();
			});
			scope.spawn(move || {
				bd.compute();
			});
		});
	}
	{
		//The third combination
		let ad = CompositeAD(&mut thing);
		let bc = CompositeBC(&mut thing);
		crossbeam::scope( move |scope| {
			scope.spawn(move || {
				ad.compute();
			});
			scope.spawn(move || {
				bc.compute();
			});
		});
	}
	{
		//This would be an error because of overlap
		//let ab = CompositeAB(&mut thing);
		//let ac = CompositeAC(&mut thing);
	}
}

In most cases, there is only one interesting combinations of views, case in which it can be done just by including a struct as field. But when there are multiple views of interest it is not clear the implementation with the current language. I also realize that this idea could have a too narrow use case, but it could be worth discussing.

[nikomatsakis]

This is definitely hitting at a real problem, specifically the need to have helper functions that touch disjoint sets of fields which can overlap.

This previous thread also discussed the same general topic:

Partial self borrowing syntax

Yeah, but you don’t actually want to be forced to make your fields public. In any case, at this point we are walking right into a whole field of research about how best to specify which APIs can be composed with what – i.e., you might like to be able to say things like "the method foo can be called in parallel with bar", without having to say what state they affect. One way to do this that I personally pursued as part of my PhD is to partition the state abstractly, so that you can say “the met…

The idea of explicit views is interesting; as I mention in that thread, I’ve been leaning towards the idea that it would be nice – at least for private functions – for the compiler to do the inference for you. But I’ve been wanting some kind of explicit syntax as well.

I actually do something sort of like explicit views occassionally in my own coding. That is, I create “shadow structs”, like:

struct RealStruct {
    foo: Foo,
    bar: Bar,
    baz: Baz,
}

struct ShadowStruct<'a> {
    foo: &'a Foo, // foo is immutable
    bar: &'a mut Bar, // bar and baz are not
    baz: &'a mut Baz,
}

[nakacristo]

Well, I saw the problem mostly in a theoretical way. If you have had some occasions in which this is relevant maybe it needs to be addressed. Because having to make a struct with references to each field of other struct seems pretty bad, both in efficiency and ergonomics. In contrast, using a view would be in low level a single pointer to the real struct. Its only problem being that it requires new notation.

Regarding the notation for creating the view from a struct there are several possibilities.

struct Real{...}
view Part for Real(...);

let real_thing: Real;

//This is my original attempt, which does not seem very natural
let part_thing = Part(&mut real_thing);

//I like this notation more
let part_thing = &mut (real_thing as Part);

//Or maybe this, since references should be able to be 'viewed'
let part_thing = (&mut real_thing) as (&mut Part);

And following your example, perhaps the fields in a view should be inmutable by default. Thus we could write

struct RealStruct{
	foo: Foo,
	bar: Bar,
	baz: Baz,
	other: OtherData,
}
view ShadowStruct for RealStruct(foo, mut bar, mut baz);

Then an inmutable ShadowStruct could access immutably to foo, bar and baz, but not to other. And a mutable ShadowStruct could access to foo inmutably and to bar and baz mutably.

The original example in the other thread could be written as follows.

struct Foo {
	x: u32,
	v: [u32; 2]
}

view FooBar for Foo(mut x);

impl Foo {
	fn new() -> Self {
		Foo { x: 0, v: [0; 2] }
	}

	fn spam(&mut self) {
		for _ in self.v.iter() {
			//Legal since v and FooBar are disjoint
			(self as &mut FooBar).bar();
		}
	}
}

impl FooBar {
	fn bar(&mut self) {
		self.x += 1;
	}

}

[nikomatsakis]

Hmm, so let me bring in a bit more context. The main time that I think this problem strikes people is sort of mundane:

pub struct MyStruct {
    some_data: Data,
    some_more_data: MoreData,
}

impl MyStruct {
    pub fn some_function(&mut self) {
        let ref1 = self.ref_to_some_data(); // borrows self
        self.modify_some_more_data(ref1); // error, self is still borrowed
    }

    fn ref_to_some_data(&self) -> &u32 {
        &self.some_data.some_field
    }

    fn modify_some_more_data(&mut self, ref: &u32) {
        self.some_more_data.modify(ref)
    }
}

Here the key point is that ref_to_some_data and modify_some_more_data access disjoint fields. (As I wrote earlier, I would like code this to “just compile” – with no syntactic changes – but let’s discuss that a bit later and focus instead on how you would write out something more explicit if you wanted to.)

I was thinking that maybe “views” might be that you get into via coercions or casts. Basically a &mut Foo could be coerced to a &mut FooView, where FooView is some view on Foo that selects a subset of its fields. Similarly, borrows like &mut foo would have not only a lifetime 'a of the borrow, but a view V that is being borrowed, so that the final type is &'a mut V.

(Note that we don’t presently allow you to write the lifetime of a borrow explicitly; we probably wouldn’t let you write the lifetime of a view either, but instead use context, meaning you might do something like this to select a view explicitly:

let x: &mut View = &mut foo
(&mut foo) as &mut View

Most of the time, though, I imagine you’d be getting your view via the self type of a method that is being called:

fn foo(self: &mut View, ...) { ... }

and when we insert the “autoref” for x.foo(), we would insert one with the appropriate view.

[nikomatsakis]

Oh, a few more thoughts. A view I imagine might be just a subset of the fields with either mut or not in front of them. This then makes me think of the fields-in-traits RFC, I wonder if a “view” could just really be a trait.

[nakacristo]

The main time that I think this problem strikes people is sort of mundane:

Well, with this mundane example one can just rewrite it.

pub struct MyStruct {
	some_data: Data,
	some_more_data: MoreData,
}

impl MyStruct {
	pub fn some_function(&mut self) {
		let ref1 = &self.some_data.some_field;
		self.some_more_datat.modify(ref1);
	}
}

There are more complicated examples for which doing this transform is impossible, as the one in my first post. But as these are artificial examples I wonder if there is actual need for this.

I would like code this to “just compile” – with no syntactic changes

I slightly disagree. I prefer the code to be explicit, otherwise the errors could get quite confusing.

Oh, a few more thoughts. A view I imagine might be just a subset of the fields with either mut or not in front of them. This then makes me think of the fields-in-traits RFC, I wonder if a “view” could just really be a trait.

It seems quite interesting and I do not view big complications.

struct Foo {
	x: u32,
	v: [u32; 2]
}

trait FooBar
{
	mut x:u32,//mut here feels a little weird. But how else to make correctly the ShadowStruct example?
	fn bar(&mut self) {
		self.x += 1;
	}
}

impl FooBar for Foo
{
	x: self.x,
}

impl Foo {
	fn new() -> Self {
		Foo { x: 0, v: [0; 2] }
	}

	fn spam(&mut self) {
		for _ in self.v.iter() {
			//Legal since v and FooBar are disjoint
			self.bar();
		}
	}
}

The mut bit is annoying. But it is useful for this example.

struct Real{
	foo: Foo,
	bar: Bar,
	baz: Baz,
}
trait R
{
	foo: Foo,//inmutable
	mut bar: Bar,
	fn compute(&mut self){...}
}
trait Z
{
	foo: Foo,//inmutable
	mut baz: Baz,
	fn compute(&mut self){...}
}
impl R for Real
{
	foo: self.foo,
	bar: self.bar,
}
impl Z for Real
{
	foo: self.foo,
	baz: self.baz,
}
impl Real
{
	fn compute(&mut self)
	{
		//This is legal because the intersection of R and Z is foo, which is always used as inmutable
		let r=self as &mut R;
		let z=self as &mut Z;
		//Call r.compute(...) and z.compute(...) in parallel or whatever
	}
}

I see you have been discussing traits with fields for quite a long time, while I have been using Rust for just a few weeks. So you should know better which problems there are to adopt this.

[kornel]

Changing this to non-trivial case:

fn main() {
   let my = MyStruct::new();
   let ref1 = my.ref_to_some_data();
   my.modify_some_more_data(ref1);
}

The equivalent fix would require making the fields public:

   let ref1 = &my.some_data;
   my.some_more_data.modify(ref1);

And in practice I find this unacceptable, because it breaks encapsulation. It also allows

my.some_data = new_object

which may break invariants that the implementation needs to uphold.

Because of that I often wished I could expose fields as read-only:

pub struct MyStruct {
    pub(but read-only) some_data: Data,
    pub(but read-only) some_more_data: MoreData,
}

[nikomatsakis]

Changing this to non-trivial case:

So, first, hold-up. I was very specific in my initial example to use private functions: the majority of time that I hit this problem, it concerns internal details of a type. Basically, I would prefer to have some helper functions, but I cannot, because the borrow checker doesn’t let me. This – the private function case – is I think a distinct use case worth considering separately when it comes to ergonomics (maybe not the underlying mechanism).

I find that when I am working with types “externally”, as in your example, it is much more unusual for me to have the insight into “what fields a method may touch”. I tend to treat it as an atomic unit, and I don’t find the rules prohibitive (now, maybe it influenced the author of the type in terms of the public API that they export).

There are some exceptions: the main ones that I’ve noticed involve collections. For example, I like split_at_mut on vectors, which covers more-or-less this same case. Similarly, I sometimes want the ability to have multiple mutable refs into a hashmap.

All of this is not to say that we don’t need to worry about external users at all – but I would say that for an external user, defining an explicit view or trait is not necessarily prohibitive. It makes sense because it is giving your users a “window” into your “mental model”. i.e., if you have a struct Connection that exposes two views, Buffer and Address, then you are telling them they can think of a connection has having those subparts, and then you may have distinct methods that sometimes operate only on the buffer, sometimes only on the address, etc.

Anyway, I am pretty excited about the idea of leveraging fields in traits for this. (Don’t have time to write a bunch here, more later.)

[nikomatsakis]

This – the private function case – is I think a distinct use case worth considering separately when it comes to ergonomics (maybe not the underlying mechanism).

Re-reading this, I don’t think it fully communicated my intent. First, if possible, I certainly prefer a mechanism that acts the same for internal/external cases. Leaning on privacy always feels like a bit of a smell.

But secondly, part of what I meant to say here was that it’s very important that you be able to “fully encapsulate” these views so that they are not visible in your public API. (But I don’t think using traits to describe them is incompatible with that.)

(But circling back to the first point, if we were going to try and make something work semi-automatically – e.g., inferring views – I would definitely limit it to the private fn case. But I’m happy to shelve all talk of inference for now and focus on explicit notation. I suspect that would not be too onerous, this doesn’t arise that frequently.)

[BatmanAoD]

(Is this the right place to woodshed possible approaches?)

I think having a new contextual keyword would be good here; my suggestion is to treat views as an alternative type of reference for self (and, down the road, perhaps for other arguments). So, we would introduce a new type of self parameter for methods:

fn takes_view(&view self)
{
    self.a = 7;
    println!(self.b);
}

Here, view is the conditional keyword, referring to a “view-type”, which would be a compiler-generated struct of the form described above (where the members are all references to members of another struct). Use of &view would require the compiler to:

1. Determine which fields in self are mutated in the function (in this case, a)
2. Determine which fields are read (in this case, b)
3. Generate the correct view-type for that combination of field-references
4. (optional) Merge identical view-types
5. Replace &view self with a mutable (if necessary) reference-to-view-type argument

At the call site, the view-type would be automatically instantiated and passed as the first argument.

[Soni]

my proposed PIT syntax also ties in with partial borrows.

[H2CO3]

Not sure if this needs a new keyword. Can’t borrowck can be taught to carry partial borrow information through reference types themselves? That would mean that the language surface wouldn’t change, it’s just that the borrowing mechanism would become smarter and allow more valid code, in a way similar to the transition to non-lexical lifetimes.

[Yato]

This shouldn’t happen over a public interface otherwise simply changing the implementation of a function becomes a breaking change. I would even limit it to work only within a module, otherwise there is too much action at a distance, which is bad.

[Soni]

Wouldn’t the issues of public interface and stuff be solved with PIT syntax? (we don’t need full on PITs, we just need the syntax)

E.g., standard PIT stuff:

1. In public interfaces, PIT fields must be public:

   pub fn foo(&Foo(t)) -> &Bar {...} // compiles only if `struct Foo { pub t: ... }`
   

   Basically fields must be visible everywhere the function/etc is visible.
2. PITs can only be downgraded, i.e. this is invalid:

   struct Foo { a: ..., b: ... }
   struct Bar { a: &Foo }
   fn f(foo: &Foo(a)) -> &Bar {
       Bar { a: foo }
   }
   

   because Bar wants a full Foo whereas f only has a partial Foo.
3. All the stuff about PIT references, really.

But we should have the following differences:

1. No actual PITs. These would only be valid for references. No let foo: Foo(t), only let foo: &Foo(t) and let foo: &mut Foo(t).
2. No initialization stuff either (&mut Foo(a) -> Foo(b)).

[BatmanAoD]

Not sure if this needs a new keyword. Can’t borrowck can be taught to carry partial borrow information through reference types themselves?

This shouldn’t happen over a public interface otherwise simply changing the implementation of a function becomes a breaking change.

Well, that’s part of why I suggested the &view syntax/keyword. It’s a balance between being explicit and permitting the compiler to do the grunt-work. Views in public functions are part of the public interface, yes, but restricting them doesn’t seem entirely reasonable, nor does forcing the user to manually define view structs.

[BatmanAoD]

Wouldn’t the issues of public interface and stuff be solved with PIT syntax ?

Stupid question: what’s a PIT?

[Soni]

Partially Initialized Types, it was an idea that would unify proposals for uninit references and opaque references (thankfully neither of which because a thing!) and I guess (now) also partial borrows.

it’s rather explicit, because it’s not allowed to do type inference across function boundaries. (i.e. types must be specified in function declarations.) but that actually avoids some misborrowing bugs that can happen sometimes.

[Yato]

Here’s a link to @Soni’s Pre-RFC for PIT.

[Pre-RFC] Partially Initialized Types

there’s been a lot of talk about &uninit references, MaybeUninit, placement-new… I’d like to propose an alternative to all of them: Partially Initialized Types. They’re, arguably, the cleanest approach to all of this. Consider types: struct Foo { a: i32, b: i32 } struct Bar { a: i32, b: Foo } Then, we can have: let x: Bar(b(a)) = Bar { b: Foo { a: 1 } }; However, passing these to functions is mostly pointless: fn foo(x: &mut Bar()) { // can't do anything with x here } foo(x); // but c…

[Yato]

Does view specify which fields are being referenced and what mutability they have, if not then it is still action at a distance. @Soni’s PIT syntax almost does it, if mutability was specified per field but the syntax is almost as bad as just creating a new struct (and is even worse if there are mutiple functions that take the same fields with the same mutabilites).

---

I personally don’t think this proposal can carry its weight, I think a decent solution would create too much noise and add too much weight to the language to be sufficiently useful. I also have never ran into this problem and had a hard time solving it. Normally splitting the data into multiple related structs does the trick.

Also if this sort of thing applies to a public interface, I think it would make it very hard to change private implementation details if needed. I find this to be a bad thing. Therefore, if at all, this should only be limited to crate boundaries.

---

This also seems like something a proc macro could solve. A proc macro could create all the view structs that you need. You would just need to specify which ones you need, then you could use the types that the proc macro generates.

[H2CO3]

Oh yes, that is right. But that in fact applies to any implementation / syntax which tries to infer the borrowed fields from the implementation of the function. So merely the view keyword wouldn’t help here.

I could imagine adding annotations of partial borrowing to reference-type function arguments instead. Something along the lines of fn foo(&self::{ bar, baz }) { … } indicating that foo only borrows self.bar and self.baz (which would then be enforced within the function body, and not inferred.)

(Oh my gosh that syntax looks really ugly btw, just wanted to come up with something quickly.)