Type erasing non-`'static` types

libs
1

Earlier discussion about this topic

Motivation

What I want is pretty simple. I have a library and want a TypeMap (HashMap<TypeId, Box<dyn Any>>) as an intermediate storage for the user to insert to and retrieve from, but with values that don't live for the whole program.

Thought Process

In basic rust thinking this shouldn't be too hard. Just make the dyn Any have an additional lifetime and make sure that we can only insert values that live for at least that lifetime. Because TypeId doesn't say anything about the lifetimes, and definitely can't give us them as usable lifetimes, when we downcast the value back to the type, we should get back a type where all the lifetimes are the one of the typemap. Remember, shortening lifetimes is safe and is done all the time.

So that was the plan, and it has a lot of problems. Both TypeId and the Any trait, basically the whole type-erasure suite only works for 'static types.

With the knowledge of the discussion from Would non-`'static` TypeId be at all possible? I knew that TypeId doesn't have any rules for lifetimes and with the reaction of closing the RFC linked at the top, I knew that people don't want to do changes to it either.

So, what did I do? Simple. I made my own TypeId. But with extras. It is a wrapper around TypeId with the rule that it excludes the lifetimes. Basically, it should just get the TypeId of the 'static version of the type. Then, I just write my own LifetimedAny<'life> which doesn't have a 'static requirement, but just for 'life, make it use my id for comparison, and cast the type to a type where all the lifetimes are 'life on downcast.

Now, how will I do that? Let's just grab our beloved unsafe block and.. oh.. In rust, the lifetime language, we can't play with lifetimes in unsafe blocks. cries. grabs proc-macro. So I made a trait which has an associated type with a lifetime generic which should give me the same type as the one who implements it, just with all the lifetimes being replaced with the given one and made it implementable via a derive macro. Now that we have the 'life version of a type and have checked that the type represents (my way of the Any::is method, excluding lifetimes), we need to actually downcast our value into that type. Which we... can't.. We just can't assure that the associated type is the same size as the implementor, let alone the same type. So we also need to give the trait the functions to cast it.

And voila, my experiment: GitHub - DasLixou/zonbi: Type-Erase tools for non-`'static` types Zonbi. (japanese for zombie, because zombie-types. found it funny. zombie was already taken)

Come to a point, what do you want?

Good question. Really, what is this? An Pre-RFC? A help-cry?

I'm not sure either. I am pretty happy about the prototype I came up with, and it works like a charm! The problem is, that it is a proc macro, thus it has the problems every other proc macro has. The end-user of my lib also needs to import the zonbi crate because $crate paths don't exist for proc macros and they also need to put #[derive(Zonbi)] on every type they wan't to put into it. Just imagine you would have to put #[derive(Any)] on every type. You wouldn't want that. But other than Any, Zonbi isn't just a trait we can commonly implement, we need to do it for every type, even if it doesn't have any lifetimes, because of lifetimes. This is pretty stupid, but currently can't be described in any wise in the standard library.

Coming back to what this is, it's probably all of those. A point to gather information and ideas, I want to see where else people want this. A place to discuss how we could implement this automatically but still have it written down in the standard library (hopefully without a #[heh_just_magic] compiler internal attribute). A topic to discuss this since 2016 wished problem and hopefully find a solution.

Feedback welcome, thanks <3

2

Here is an example of a use-after-free in safe code:

#[derive(zonbi::Zonbi)]
struct Repro<'a>(&'a str);

fn main() {
    let mut s = Repro("");
    (&mut s as &mut dyn zonbi::AnyZonbi)
        .downcast_mut::<Repro>()
        .unwrap()
        .0 = &".".repeat(40);
    println!("{:?}", s.0);
}

"M¿½¹_\0\0\u{1b}Ԩ\u{e}È8!,........................"

This is not true of all lifetimes, only for covariant lifetimes. Some reading material about variance:

5 Likes
3

Oh wow! Thanks for the reply, I really wasn't aware of this.. So if I understand it right the problem here is that we make a shorter living representation of the type where it is possible for us to write shorter-living borrows into, but then afterwards we can still use the orignial long living type with the invalid lifetime, right? hmmm...

Could we somehow just make it forbidden to cast to a &mut dyn version, so that it needs to be the owner of the original value? Or do you have other ideas?

4

I played a bit around and made a new release of zonbi which makes this behavior impossible. I can't thank you enough, your comment has made me go from "yeah I understand lifetimes" to finally understanding them and what all that covariant etc. stuff means. :heart_hands: I added a Cage<'life, Z> which must hold the zonbi, and only it implements the AnyZonbi trait and it can't be "lower-lifetimed".

5

I think with Cage, the problem can still occur:

#[derive(zonbi::Zonbi)]
struct Repro<'a>(&'a str);

impl<'a> Drop for Repro<'a> {
    fn drop(&mut self) {
        println!("{:?}", self.0);
    }
}

fn main() {
    let mut s = zonbi::Cage::new(Repro(""));
    (&mut s as &mut dyn zonbi::AnyZonbi)
        .downcast_mut::<Repro>()
        .unwrap()
        .0 = &".".repeat(40);
}
6

Im not at my computer right now so I can’t check, but I think this should be fine, because I now directly link the lifetime of the anyzonbi with the lifetime of the struct, so it shouldn’t differ from writing that same code without the anyzonbi inbetween

7

It is a user-after-free in safe code, which is not supposed to be possible.

1 Like
8

Sorry, it was vaguely worded, thought you just thought it was possible. But no matter what I try, I can't reprod. The code you sent is running perfectly fine and it prints the 40 dots in quotes without any corruption. When I change the code to be a sub function like this

fn main() {
    let mut s = zonbi::Cage::new(Repro(""));
    sub(&mut s as &mut dyn zonbi::AnyZonbi)
}

fn sub(a: &mut dyn zonbi::AnyZonbi) {
    a.downcast_mut::<Repro>().unwrap().0 = &".".repeat(40);
}

i get following error:

error[E0716]: temporary value dropped while borrowed
  --> examples\repro.rs:16:45
   |
15 | fn sub(a: &mut dyn zonbi::AnyZonbi) {
   |        - has type `&mut dyn AnyZonbi<'1>`
16 |     a.downcast_mut::<Repro>().unwrap().0 = &".".repeat(40);
   |     ----------------------------------------^^^^^^^^^^^^^^- temporary value is freed at the end of this statement
   |     |                                       |
   |     |                                       creates a temporary value which is freed while still in use
   |     assignment requires that borrow lasts for `'1`

For more information about this error, try `rustc --explain E0716`.

Like maybe I'm just stupid and overseeing something, but for me it works fine.

9

There does not need to be memory corruption for it to be a use-after-free. The String is dropped on line 15 at the semicolon. The Repro is dropped on line 16 at the curly brace, and uses the dropped string after free.

You can use cargo miri run to observe it deterministically.

error: Undefined Behavior: constructing invalid value: encountered a dangling reference (use-after-free)
note: inside `<Repro<'_> as std::ops::Drop>::drop`
    --> src/main.rs:6:9
     |
6    |         println!("{:?}", self.0);
     |         ^^^^^^^^^^^^^^^^^^^^^^^^
     = note: inside `std::ptr::drop_in_place::<Repro<'_>> - shim(Some(Repro<'_>))` at $rust/library/core/src/ptr/mod.rs:514:1: 514:56
     = note: inside `std::ptr::drop_in_place::<zonbi::Cage<'_, Repro<'_>>> - shim(Some(zonbi::Cage<'_, Repro<'_>>))` at $rust/library/core/src/ptr/mod.rs:514:1: 514:56
note: inside `main`
    --> src/main.rs:16:1
     |
16   | }
     | ^
2 Likes
10

Uhm.. pure confusion..

made this:

fn main() {
    yikes("")
}

fn yikes<'a>(a: &'a str) {
    let mut s: Cage<Repro<'a>> = zonbi::Cage::new(Repro(a));
    (&mut s as &mut dyn zonbi::AnyZonbi<'_>) // the '_ here
        .downcast_mut::<Repro>()
        .unwrap()
        .0 = &".".repeat(40);
}

which compiles, but replacing the '_ with 'a, I get the error I want. But I only implement AnyZonbi<'life> for the 'lifetime of the Cage 'life, so it shouldn't be possible to create any lower lifetimed AnyZonbi. Or is it that here, the lifetime is actually covariant? Speaking of covariant, how is such thing prevented in e.g. UnsafeCell? is it because of the lang attribute? Could we (when the covariant thing is the problem) just make zonbi also give such attribute? Or is there an even simpler solution?

11

As far as I know, there is no sound way that the API you want (Any<'life> with downcasting) can be implemented.

12

So only with special compiler treatment? (the compiler does special treatment to UnsafeCell to make it invariant, do I get that right?)

13

I doubt variance is the issue, but you can make your type invariant using PhantomData

14

but making traits invariant? :sweat: this seems way more complicated and deep down the lifetime iceberg than I imagined

15

Making a trait invariant is not really something that makes sense, as variance is something that affects types, not trait objects. The closest thing to a trait that is also a type is a trait object, and that's already invariant over its lifetimes.

Anyway, this is not the problem here. Your main invariant seem to be "Z: Zombi<'life> holds only if all lifetimes in Z are 'life". Then, along with the fact that AnyZombi<'life> is implemented only for Cage<'life, Z: Zombi<'life>> it gets you that any inner Z of a AnyZombi<'life> has all lifetimes equal to 'life. Thus if you try to downcast it to some other type Z1: Zombi<'life> and they have the same TypeId when lifetime erased, they are also the same type because all their lifetimes must be 'life and thus match as well.

Your invariant is however simply broken as your derive doesn't respect it: it is generating implementations like impl<'__zonbi_life> Zombi<'__zonbi_life> for Repro<'a>, which implement Zombi<'life> for Repro<'a> for any lifetime 'a, not just the lifetime 'life. Fixing that should fix your issue.

I'm also not so sure about your Casted associated type, as it doesn't really seem to be used at any point in this proof. Maybe you meant to use it in place of the invariant (i.e. instead of Z: Zombi<'life> having all lifetimes equal to 'life you have Z::Casted with all lifetimes equal to 'life), but then you failed to use it properly, as Cage<'life, Z: Zombi<'life>> holds a Z, not a Z::Casted. You could perhaps make Cage::new take a Z::Casted, but IMO fixing the derive is simply better, as it would allow you to also remove the Casted associated type, making Zombi an object safe trait and thus using it instead of AnyZombi<'life> (thus also allowing you to remove Cage). At this point you've just recreated the better_any crate.

1 Like
16

Oh wow, thanks. I thought I wouldn't need to do that because I already do Self: '__zonbi_life. Also, how did I not find the better_any crate :open_mouth: that is amazing! Thanks for the comment, I feel stupid now. The fact that I searched for such a crate doesn't make it any better :c

17

That only constrains the lifetime on one side. It lets you know that any lifetime won't be smaller than '__zonbi_life, but that doens't mean they can't be bigger. In the example above the lifetime was 'static, and that was indeed bigger than the 'life parameter of the Cage, which was a local temporary lifetime (the one of the &".".repeat(40) expression). The issue then is that the reference with 'static lifetime is overwritten by a reference with a 'life lifetime, which is smaller and is thus unsound. You need them to be exactly the same.

1 Like
18

May I then ask, with the better_any crate, has there been any more discussion about this topic and bringing this to std?

19

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.