### Author:  [@CAD97](https://internals.rust-lang.org/u/CAD97)
#### Post date: [March 2, 2022, 4:49pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/4 "2022-03-02T16:49:28Z")
References to unsized have provenance over `size_of_val(&)` bytes. It's generally just exactly however many bytes the reference refers to based on the type and value of the (potentially wide) pointer.
The nominal borrow stack at the pointed location gets the Retag operation applied to it when doing a reborrow of the reference (basically, whenever passed between function boundaries; when lifetime variance, autoderef can apply) and that Retag has a unique tag for that new reference reborrow. That reference can only be used to access values from memory with that tag value.
This is a gross oversimplification of the Stacked Borrows model, but should illustrate why references only have provenance over the "expected size" in the current model.
---
### Author:  [@197g](https://internals.rust-lang.org/u/197g)
#### Post date: [March 2, 2022, 5:20pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/5 "2022-03-02T17:20:08Z")
> [@bascule](#):
>
> I'm guessing there's probably some precedent around adding such a feature to `core` . If you happen to know of any, I'd appreciate a link.
The precedent is: this operation under those requirements (safe, and the expected signature) is impossible with regards to stacked borrows. The chances of landing it in std, in the current state, is thus slim. Once you have split the two halves into references you can not get a reference for the complete buffer from them again. None of the halves have sufficient rights (as per provenance tracking) to access the other. You need to use a reference/pointer to the original complete buffer and even it's super unsafe as it would invalidate _any other_ references that you may currently have into the complete buffer. Since the latter can't be outruled by lifetimes alone, no such safe operation exists.
To fix this would require introducing a new primitive operation that somehow joins the access rights of pointers. Handy example, visualized by which regions are covered by each reference.
```rust
^ top of the stack
| header: &mut | rest: &mut | 'a
| header0: &mut | rest0: &mut | 'b
| buf: &mut | 'c
----------------------------------> memory
```
Under Stacked Borrows, the only way to somewhat soundly rejoin `header` and `rest` would be to rederive a new pointer from `buf`. But deriving a new pointer from `buf` is an access that conflicts with `header0` and `rest0`—which invalidates them. No function that receives _only_ `header` and `rest` can disprove the existence of header and rest0, rendering any such function necessarily unsafe (as in requiring the caller to disprove it). In particular, such reference `header0` and `rest0` mustn't be used after joining and they mustn't be function parameters either.
---
### Author:  [@kpreid](https://internals.rust-lang.org/u/kpreid)
#### Post date: [March 2, 2022, 6:32pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/6 "2022-03-02T18:32:59Z")
A weaker alternative which should not present provenance problems:
Define an operation which takes a reference to a 'large' slice and a 'small' slice and returns the index range in the large slice where the small slice is located. Then, you can safely join the _ranges_ if their ends meet, and produce a joined slice from that range of the 'large' slice.
This requires comparing pointers which might not be in the same object, but it doesn't require expanding any single pointer's provenance.
---
### Author:  [@CAD97](https://internals.rust-lang.org/u/CAD97)
#### Post date: [March 2, 2022, 6:52pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/7 "2022-03-02T18:52:26Z")
> [@197g](#):
>
> it would invalidate _any other_ references that you may currently have into the complete buffer
If there are Uniq (`&mut`) tags floating, yes, but Shr (`&`) tags won't invalidate other Shr tags. (I don't recall exactly how using Shr acts when Raw is above it on the stack, but...) I can use the root reference just fine, and even implement the operation safely:
```rust
// just byte slices to make this easier on my phone
// but you can do it with _overlapping just as safely
fn fuse(parent: &[u8], a: &[u8], b: &[u8]) -> Option<&[u8]> {
let lhsa = a as *const _ as usize - parent as *const _ as usize;
let lhsb = b as *const _ as usize - parent as *const _ as usize;
let rhsa = lhsa + a.len();
let rhsb = lhsb + b.len();
// todo: require they're neighbors
parent.get(lhsa.min(lhsb)..rhsa.min(rhsb))
}
```
---
### Author:  [@197g](https://internals.rust-lang.org/u/197g)
#### Post date: [March 2, 2022, 11:44pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/8 "2022-03-02T23:44:28Z")
It's not for the lack of trying that this alternative wasn't brought to fruition. That exact one was prototyped, and insufficient, for `str-concat`. We tried to keep it alive but death by SB keeps coming up. The only escape is having access to the original reference with its original _shared_ tag and ensuring that no other tag exists. Which provides two possibilities:
- `&[u8]` (ShareRO) for everything: Which is effectively only a slightly more compressed form of storing `(&[u8], Range)` with no particularly amazing gain that would warrant std-inclusion or special treatment. And also woefully insufficient for a lot of actual use cases.
- `*mut T` (SharedRW) for everything: Which [std already has](https://github.com/rust-lang/rust/blob/master/library/alloc/src/collections/btree/borrow.rs) but since its access is `unsafe`, because it derives an incompatible `Unique` pointer, it's also not a particularly good replacement. I _suppose_ you could offer a sound `write` function on these pointers at least? (Edit: I may prototype this, as it actually sounds like I missed it. We _can_ perform _some_ operations with the raw pointer alone).
If I read the paper correctly, and remember the details: The problem is simply a fundamental mismatch between the borrow stack semantics that SB provides, and the semantics we want. Note that this location mismatch is completely independent of the _kind_ of tag that we'll create. Hence it must always be fully compatible with all above, which restricts us to the above two shared cases to keep all of the above items valid.
```nohighlight
| | <-- where our derived tag should live
| header: &mut | rest: &mut | 'a
| header0: &mut | rest0: &mut | 'b
| | <-- where SB inserts items derived from buf
| buf: &mut | 'c
----------------------------------> memory
```
(Is this the lang-design equivalent of an annoying extension request issue report on a library?)
---
### Author:  [@toc](https://internals.rust-lang.org/u/toc)
#### Post date: [March 3, 2022, 8:07am UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/9 "2022-03-03T08:07:29Z")
```rust
let mut buf = [0u8; 1024];
let header_len = 42
let body_len: usize;
{
let (header, rest) = buf.split_at_mut(header_len);
build_header(header)?;
body_len = build_body(rest)?;
}
let finished_doc = &buf[..header_len + body_len];
```
At the risk of asking for the example to be overcomplicated, can we complicate the example so the above (or the moral equivalent) isn't the solution?
It seems like we need to drop all mutable references into `buf` before or during any recombination, so the above won't be possible if we are at a point in the program where we only have a reference to the header/body slices. In that case I would expect some reified constraint that the slices are contiguous, but that constraint could just have a reference to the underlying object which still means we can just drop the subslices.
---
### Author:  [@bascule](https://internals.rust-lang.org/u/bascule)
#### Post date: [March 7, 2022, 7:00pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/10 "2022-03-07T19:00:19Z")
> [@toc](#):
>
> At the risk of asking for the example to be overcomplicated, can we complicate the example so the above (or the moral equivalent) isn't the solution?
So this is in fact more or less the "solution" I'm using:
>
...but the problem is this only works where the original reference was in scope.
I would like for this function in a buffered encoder to be able to return a `&[u8]` as well, rather than a `usize` containing the length:
>
...but the problem is splitting the original slice to hand part of it off to a struct which stores a reference to it:
>
...or in other words, I'm trying to decompose the original mutable buffer in a way I can statefully separate responsibilities for encoding various parts of the buffer to types which are tasked with one particular aspect of the encoding.
Based on the comments here, it sounds like what I'd like really isn't possible in the stacked borrows model.
I guess this is fine, and wrapping things up in high-level functions which hold a mutable reference to the original contiguous buffer, with the stateful buffered parts returning the encoded length as a `usize`, which the high-level function can use to reslice the original buffer.
---
### Author:  [@197g](https://internals.rust-lang.org/u/197g)
#### Post date: [March 7, 2022, 8:03pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/11 "2022-03-07T20:03:40Z")
Hm, a reference type that is for all intents and purposes `SharedRw` and you can't be allowed to derive a direct reference to the underlying content. That … sounds like `&Cell`? Writing a sound join for ref-to-slice-cells specifically shouldn't be too hard, that's basically the trivial case of joining via an underlying `&` of the whole region. Can you make a list of methods that would you need to be added to `&Cell<[u8]>` and `&Cell` for those types to be potent (and ergonomic) enough for your usecase? Because `copy_from_slice` sounds perfectly doable.
---
### Author:  [@toc](https://internals.rust-lang.org/u/toc)
#### Post date: [March 7, 2022, 8:58pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/12 "2022-03-07T20:58:45Z")
> [@bascule](#):
>
> or in other words, I'm trying to decompose the original mutable buffer in a way I can statefully separate responsibilities for encoding various parts of the buffer to types which are tasked with one particular aspect of the encoding.
What about returning `Result<(&'o [u8], &'o mut [u8])>`?
---
### Author:  [@bascule](https://internals.rust-lang.org/u/bascule)
#### Post date: [March 7, 2022, 9:34pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/13 "2022-03-07T21:34:47Z")
I actually already rely on methods like that (i.e. when encoding PEM, after Base64 encoding is complete the PEM encoder still needs to add a post-encapsulation boundary):
>
...but the problem here is it would be nice to return the _entire_ encoded document back to the caller in a single contiguous slice.
Having a high-level API which retains the reference to the contiguous buffer and can thus do that, and a lower-level one which returns a `usize` of the total encoded length (with the high-level API using that `usize` to slice the original buffer) seems like a reasonable tradeoff.
---
### Author:  [@system](https://internals.rust-lang.org/u/system)
#### Post date: [June 5, 2022, 9:35pm UTC](https://internals.rust-lang.org/t/safe-slice-rejoining/16235/14 "2022-06-05T21:35:29Z")
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.