I’ve wondered the same thing recently after/while pondering on unsized coercion in the context of finding #85099. Turns out, unsized coercion it mostly separate from subtyping. In effect converting a T into an S, which isn’t allowed.
The reason why RefCell<T> is invariant in T is that otherwise, you could have S subtype of T, take an &RefCell<S>, coerce it into &RefCell<T>, then write a T into that, and read the value back as an S from a copy of the original &RefCell<S> reference.
Now, unsizing does feel similar to subtying in a lot of ways. There’s a type T and another type dyn Trait and you’ve got an “is-a” relationship between them. Yet, the coercion from T to dyn Trait only works in types where T exists behind exactly one layer of indirection. So something like &RefCell<T> to &RefCell<dyn Trait> works. But then, this conversion doesn’t present any way of replacing the dyn Trait inside of that RefCell with a different value of type dyn Trait like it was possible for the types S and T above. Why? Well, you cannot more any dyn Trait values because they’re unsized, and any self: &RefCell<dyn Trait> method from Trait that can modify the RefCell’s contents calls a sound method on the underlying type &RefCell<T>, so it doesn’t really offer any way to exploit it either.
The reasoning above isn’t much more than: Turns out the types being unsized and the way trait objects work just don’t offer any way to create unsoundness. I don’t find it particularly convincing myself since I cannot really get any deep intuition about why no way to create unsoundness exists; I just know that the ways I tried don’t seem to offer any weak points.
If for example we’d try to eliminate the problem of being unable to write to the RefCell because it contains an unsized type, we might box the type, so use &RefCell<Box<T>>. But that’s two indirections now so the unsound coercion is gone… We could remove the outer reference and, well…, now we can convert RefCell<Box<T>> into RefCell<Box<dyn Trait>>! But that’s not a particularly dangerous operation; it could be achieved by RefCell::into_inner and RefCell::new. In fact, even for subtypes S of T, direct conversion of RefCell<S> to RefCell<T> aren’t problematic from a soundness perspective (yet they are disallowed by variance; you’d need to do the into_inner+new approach here); having RefCell be invariant is somewhat of a hack that’s only there to ensure that shared references (like &, Rc or Arc) to RefCells aren’t covariant.
Another interesting point on the interaction of trait objects and variance: There are subtle cases of what works and what doesn’t work when you’re comparing dyn for<'a> Foo<'a> with dyn Foo<'b>.
Certainly, intuitively a dyn for<'a> Foo<'a> is a dyn Foo<'b>, so this is subtyping-like. And in fact it does interact with subtyping/variance, so something like
trait Foo<'a> {}
fn foo<'b>(x: Box<Box<Box<dyn for<'a> Foo<'a>>>>) -> Box<Box<Box<dyn Foo<'b>>>> {
x
}
compiles. But the same coercion appears to also work in contexts where unsized coercion would be allowed. For example this also compiles
fn bar<'b>(x: Box<RefCell<dyn for<'a> Foo<'a>>>) -> Box<RefCell<dyn Foo<'b>>> {
x
}
fn bar_1<'b>(x: RefCell<Box<dyn for<'a> Foo<'a>>>) -> RefCell<Box<dyn Foo<'b>>> {
x
}
fn bar_2<'b>(x: RefCell<Box<RefCell<dyn for<'a> Foo<'a>>>>) -> RefCell<Box<RefCell<dyn Foo<'b>>>> {
x
}
but it doesn’t compose with other covariant wrappers, e.g.
fn baz<'b>(x: Box<Box<RefCell<dyn for<'a> Foo<'a>>>>) -> Box<Box<RefCell<dyn Foo<'b>>>> {
x
// error[E0308]: mismatched types
// --> src/lib.rs:14:5
// |
// 22 | x
// | ^ one type is more general than the other
// |
// = note: expected struct `Box<Box<RefCell<(dyn Foo<'b> + 'static)>>>`
// found struct `Box<Box<RefCell<(dyn for<'a> Foo<'a> + 'static)>>>`
}
or
fn qux<'b>(x: Box<RefCell<Box<dyn for<'a> Foo<'a>>>>) -> Box<RefCell<Box<dyn Foo<'b>>>> {
todo!() // wouldn’t work either
}
and here, at least the last example qux would also be unsound&RefCell<Box<dyn Foo<'b>> or Arc<RefCell<Box<dyn Foo<'b>>> which would actually be unsound. With a Box of course it’s not a problem, in fact it can be implemented by dereferencing and Box::new.
(all code examples in a playground)
I do believe that all of this isn’t sufficiently documented in the reference, at least as far as I’m aware. Especially the way that for<'a> Foo<'a> worked is something I could only find out by trying out what compiles and what doesn’t. Looks like the functions bar[_(1|2)] aren’t even compiling on not-too-recent compilers. Seems like they’re only accepted since Rust 1.46 (no mention of this in the release notes though).
. So the thing is that, for instance, a 
