Idea: Experimental attribute based support for partial borrows

Partial borrows are considered a very high difficulty problem to solve, mostly due to semver (API stability) concerns, and what the syntax should look like.

Due to this, there has been no progress on partial borrows at all, as far as I'm aware.

I think there is a way we could at least move forward and set up the basic compiler infrastructure for partial borrows, and let nightly users experiment with what kind of new patterns partial borrowing would allow, and how useful it is in general.

The idea is to bypass both the semver and syntax concerns, by introducing a perma-unstable rustc_partial attribute that can be used on function arguments to instruct the borrow checker to split up the borrow into disjoint parts. This attribute can only be applied to function items that are pub(crate), or more restricted in terms of visibility. This would forbid partial borrows from "leaking" into the ecosystem, and make semver concerns much less of an issue.

In my experience, partial borrows are most useful for splitting up complex internal systems, so it would still be very useful, especially for experimentation purposes, even if it can't be used for public facing APIs.

How it would look like

Here is my basic idea of what it could look like. Subject to bikeshedding, but since it would be perma-unstable, I don't think there needs to be too much bikeshedding, as long as it's technically sound.

Let's start with an example (based on code in one of my projects)

impl App {
    // Borrow the field `meta_state` immutably, and `hex_ui` mutably
    pub(crate) fn switch_layout(#[rustc_partial(meta_state, mut hex_ui)] &mut self, k: LayoutKey) {
        self.hex_ui.current_layout = k;
        // Set focused view to the first available view in the layout
        if let Some(view_key) = self.meta_state.meta.layouts[k]
            .view_grid
            .first()
            .and_then(|row| row.first())
        {
            self.hex_ui.focused_view = Some(*view_key);
        }
    }
}

App::switch_layout could then be used like this: (original code)

for (k, v) in &app.meta_state.meta.layouts {
    if ui.selectable_label(win.selected == k, &v.name).clicked() {
        win.selected = k;
        app.switch_layout(k);
        // roughly equivalent to this from a borrowing standpoint:
        // App::switch_layout(&mut app.hex_ui, &app.meta_state.meta, k);
    }
}

The basic syntax is #[rustc_partial(fields...)], where each field is the name of a field on the argument the attribute applies to, and an optional mut specifier, to specify a field as being mutably borrowed (instead of the default of immutably borrowed).

Trying to use it on a pub fn item would result in a compile error:

pub fn foo(#[rustc_partial(mut field)] &mut self)
//           ^ Error: `rustc_partial` cannot be used on `pub` items due to API stability concerns.

This would be a perma-unstable attribute that's only available under a feature flag, and could be removed once a better solution for partial borrows is devised, or if it's decided that partial borrows are not a feature that Rust should have.

While it might seem undesirable to have a perma-unstable feature, it would provide value over not having anything until we have a final design:

  • Implementing it will set up the basic compiler infrastructure for experimenting with different solutions to partial borrows.
  • Application and game developers who don't mind using nightly can use it to alleviate a lot of borrow checking based woes, and can focus on solving problems instead of constantly restructuring their code just to appease the borrow checker.

There is also precedent for unstable features existing which don't have a clear final design, like specialization, which among other things has soundness issues, but is still useful to have.

3 Likes

(Hello from Reddit.)

FWIW, the most recent brainstorming that I am aware of was in this thread: Blog post: View types for Rust - language design - Rust Internals

I don't mean to nitpick, but "constantly refactoring their code" is exactly what is meant by "rapid iteration" in game development. It might be worth clarifying that the kind of refactoring being referred to is strictly the kind of refactoring that is only necessary for proving disjointness to the borrow checker.

One thing about this proposal (and similar proposals) that concerns me is how to teach it to Rustaceans. By opting-in to partially borrowing self, the method can only receive the specified fields of Self. This method is unable to call any other Self method that receives a nonintersecting set of fields (and their associated "access types").

It seems that this is the case but, given the fact that I am currently having trouble articulating the concern, it makes me wonder how well it can be documented. And what else can be done to avoid the inevitable need to extend its capabilities as users need more and more coarse-grained access to fields across method call boundaries.

In well-contained cases like the examples provided, specifically the ones that are capable of being factored out to standalone functions, this would be a welcomed ergonomics addition to the language.


Specifically, the point made about breaking encapsulation ("niceness 1" in your older presentation) to work around the issue is a weakness in Rust today. There are clear benefits to doing something, anything about it.

The good thing is that if you try to call a bad method the compiler will complain, and rust-analyzer might even fix your code for you. Which is a semver churn, but that's usually only a concern for libraries.

The only remaining trouble is that this exposes a kind of implementation detail: if you implemented a method differently (but having the same result) maybe you would need to change your method signature. This means that libraries might be overly cautious at first to fully utilize this feature, because doing so would lock them into not changing the implementation in some ways (and reducing the fields you access is a semver-compatible change, so it may be taken down the road). That is, if you promise that a method will never read a certain field you better be sure of it, otherwise this may mean a future major version which in some cases is a hassle.

Yes, but the suggestion in OP is to only allow this for pub(crate) or less. And I have mostly found I wanted this in internal code, not my public API. Here is an example I can think of somewhat recently:

I had a state struct for a file format specific merging algorithm, plus a bunch of helper functions on that state struct. In a few cases I wanted to hold borrows while calling mutating helpers. The borrows were on unrelated parts of the state. Obviously this doesn't work today.

The other common case for me is with iterators plus mutating via helper functions on different parts of the same state struct.

In both cases you could possibly split it up into sub-structurs, but it is not always your helpers only need to touch the same subset of fields. I often find I have cases of "borrow a and modify b, c" as well as "borrow b, modify a and c" for the same code. In that case there is no way of splitting it up into structures.

I have yet to want to do this in a public API. So I don't think this is a major concern. At least not for the proposed perms-unstable experiment. It is absolutely something to think of once we get feedback from such an experiment and have to figure out a stabilizable syntax for it.

3 Likes

One thing that comes to mind is how this interacts with crates that implement a form of self-borrowing. Is something like #[rustc_partial(mut field, ref_to_field)] going to be detectable?

I agree, I changed the wording there.

I don't think how to teach it has to be necessarily tackled, as this would be an experimental unstable feature for advanced users who already know about the problem of partial borrows, and want to express partially borrowed arguments.

And what else can be done to avoid the inevitable need to extend its capabilities as users need more and more coarse-grained access to fields across method call boundaries.

As this is an unstable feature, it would be open to experimentation, based on what needs arise in real usage, and seeing if they can be tackled. I don't think we should necessarily avoid extending it, but the base form is powerful enough in my opinion that it solves a lot of real issues that arise in complex interprocedural code.

2 Likes

Can you expand on what it means to "detect" the attribute?

Do you mean that crates that implement self-referential types need to detect partial borrows, because they operate based on the assumption that the whole of self is borrowed for every method call?

Can you show an example?

FWIW, I believe a nightly only syntax that doesn't clash with any currently stable syntax can be part of the feature, even if it is not finalized yet. Personally, I would make it something along the lines of the following, which is perfectly invalid today:

impl App {
    // Borrow the field `meta_state` immutably, and `hex_ui` mutably
    pub(crate) fn switch_layout(self { &meta_state, &mut hex_hi, .. }, k: LayoutKey) {
        hex_ui.current_layout = k;
        // Set focused view to the first available view in the layout
        if let Some(view_key) = meta_state.meta.layouts[k]
            .view_grid
            .first()
            .and_then(|row| row.first())
        {
            hex_ui.focused_view = Some(*view_key);
        }
    }
}

This makes it look closer to the associated function approach, while still allowing the method call syntax to continue working.

2 Likes

I'm very much in favour of adding such feature.

I'd go even further and make it just work for all crate-private methods without inventing any syntax at all.

Newcomers to Rust already expect it to work like this – they naturally reason that getters and setters shouldn't lock the whole object, and that self.method(&mut self.field) should work.

The opaque API stability concern doesn't exist within a crate where all source is coexisting, visible, and editable. There's no extra syntax for disjoint borrows within functions, and for no-hassle refactorings there shouldn't be any when splitting code into functions.

3 Likes

My idea was that by using an attribute based syntax, we can avoid bikeshedding of the syntax being a blocking factor in implementing this feature. If we start arguing about syntax, I'm afraid this proposal will have much less of a chance to go anywhere.

2 Likes

One possible issue with that is that the borrow checker would have to do a much more thorough analysis for every function call, which could possibly slow down compilation significantly. I have no test data to back this up though. By marking partial borrows explicitly, the compiler can know where it needs to do a more thorough analysis.

2 Likes

This would go against the Rust goal that the signature of a function should be the only thing that affects checking its calls (which is already not true for some reasons, but still).

4 Likes

Would this only work for self? That would be unfortunate, I have had use cases where I need to work with partial borrows on two different structures at once.

I would expect this to be implemented like the "view types" behind the scenes, but with the fields inferred from the function body, rather than spelled out by the user. I think it should be manageable. Rust already does a similar analysis for Send in async functions. Also Rust's compilation speed is usually not bottlenecked on type checking, but mainly code generation in LLVM, linking, and sometimes proc macros.

Yes, and I think that for private methods that aren't public API, it's worth to break this rule for the usability benefits of more fine-grained and usable borrow checking.

2 Likes

Aren't these passes run serially? I.e. there isn't left over capacity for type checking that goes unused while running the other parts. That is not how general purpose CPUs work. You can only say that something (e.g. the GPU) is bottlenecked by something else (e.g. the CPU) if it would end up sitting idle.

What might be true (I haven't checked, but it sounds plausible) is that the type checking is a very minor part of the compile time, such that optimising it isn't a good "bang for the buck" according to Amdahl's law. That doesn't mean we should make it slower just because either, since it still does contribute to the over all compile time.

Also, I would personally prefer this to be explicit in type signatures, since type signatures are for the benefit of the human, not just the compiler. Global type inference (like in Haskell) might be something the compiler can do. My brain can't.

2 Likes

The first post in this thread references a write-up about Rust being a poor fit for game (not engine) development. I think this is a very interesting write-up, since it's from a perspective of an iterative development style that Rust is currently too rigid for, due to overemphasis on correctness and long-term maintenance at cost of getting in the way of rapid iteration. Author's major pain points included inflexibility of borrow checking, as well as "forced refactorings" making trying out ideas costly in terms of effort put to make the code compile.

With that in mind, I think if disjoint borrows were simply relaxed across private functions, it would strictly reduce the friction required to move code around, and allow simple patterns like self.method(&mut self.field) to just work, instead of needing awkward workarounds, refactoring structs into multiple sets, or syntactically noisy changes of wrapping things in RefCell or Mutex. But if this also required declaring all the fields explicitly across the call stack, it wouldn't be as convenient (still an improvement though). It would make Rust get more syntax to learn and write, instead of merely making the compiler complain less about code that could have worked if Rust wasn't so inflexible.

1 Like

What makes a language a good for for one niche makes it poor in another. Is it possible for a single language to please everyone? I don't know that it is. Maybe you could have a crate-wide opt-in to different rulesets, similar to how you can configure diagnostics and lints. But that might make it harder to teach the language.

I care more about correctness and maintainability. But then my day job involves hard real-time industrial vehicle control software (most of it isn't safety rated, but some absolutely is). I think Rust today strikes a good balance of safety and practicality. I would like to see it lean more into safety (only fallible allocations) but realise that isn't practical for the majority of users.

So I don't think any language can fit every use case (or we wouldn't have multiple ones, nor would we have we have domain specific languages). Then the question becomes who Rust should be for. Currently there is a lot of options for making games and other things where you don't care about correctness (to varying extents), there aren't many safe systems programming languages. Ada and Rust are the options I know of, unless you go exotic with formally proven code generating a C program or similar. Other people may disagree.

So let's look at the pros and cons of implicit partial borrowing from the systems programming perspective:

We obviously don't mind features that make our lives easier and code faster to write. Rust is so much faster than C++ to code in! In part thanks to being stricter I would argue, but a better ecosystem of libraries also help. So partial implicit borrowing would indeed reduce a bit of refactoring churn, at first glance. I would also argue that Rust has fearless refactoring, and that isn't impacted, the compiler will still check your work.

What about downsides? Well, compile times potentially as borrow checking before only had to look on the signatures of other functions. I don't know how this works internally so I'll defer to those who do on this one.

The bigger downside comes when doing code review or code reading. Code is read more often than written (maybe not in games, but in my world this is true). And humans aren't good at non-local reasoning.

This is especially problematic in the context of unsafe code where the compiler can't check your work. My experience when doing unsafe review is that I dislike implicit anything. So what are the risks of missing something important to due implicit partial borrows when doing unsafe code review? I don't know, not without trying it, which is hard to do on a feature that doesn't exist.

In what situations could partial borrowing interact with unsafe code in unexpected ways? I could imagine ways if it is possible to partially borrow arrays for example. What about partial borrows in enums (how does that even work to begin with)? The key thing here is that it increases the surface area of the code around an unsafe block that has to be reviewed for how they interact to determine if all precondition hold.

So here is an idea to satisfy both the implicit and explicit camp of partial borrowing: have a list of fields but have a special syntax such as "..." to indicate "automatic minimal set". This would mirror struct matching syntax where you don't care about the extra fields.

  • You still have to opt in to partial borrowing in general. So that at least is explicit.
  • If you prefer to not worry about which fields, you can mark it as such (explicitly). Then you don't have to worry down the line as you refactor things.
  • If you would rather be explicit you can be, and be sure things would change implicitly behind your back.

There is a possibility of adding opt-in clippy lints down the line about using the implicit feature for those who really care about it.

I suspect (without evidence [1]) that the majority of people who would be interested in this are beginners, since it is easy to run into this issue by writing code that is common in a language like Java or Python. To that end, I think teaching should be given a higher priority.

I appreciate that the proposal is intended to be syntax neutral and perma-unstable. But that ignores the bigger picture problem that needs to be solved, making Rust more accessible.

There is a subset of game developers that appear to have no concern for using unsafe to write unsound code as long as it allows them to circumvent the borrow checker. [2] Which raises the concern for me greatly, even as a game developer myself.


  1. Questions that appear frequently on URLO could support this suspicion, however. Like this one from yesterday: Cannot borrow *self as mutable more than once at a time...for the nth time...I know - help - The Rust Programming Language Forum (rust-lang.org) ↩ī¸Ž

  2. I won't provide direct links, but examples can be found in the peripheral social conversations that occurred in response to the loglog games article. ↩ī¸Ž

1 Like

This is deliberately intended to be an experimental feature to get things moving, not something that will end up stabilized, so I don't think promoting it to beginners is a good idea.

As for whether the majority who need it are beginners or not, I'm not sure, but I have been programming Rust since 2014, and I still run into cases even today where I want partial borrows. The author of the linked article is also a long time Rust user.

2 Likes