"arbitrary_self_types" with dyn compatibility

Hello rust experts, I am trying to build a rust game engine with a highly ergonomic API. I am currently using the nightly feature arbitrary_self_types, and is wondering how possible would it be for arbitrary_self_types to be compatible with trait objects.

I have a type Manifestly<’m, T> that serves as a custom borrow guard that contains a RefMut and the UUID, which I need to use as the self type, which derefs to type T.

pub struct Manifestly<'m, T: ?Sized> {
    pub(crate) inner: (Uuid, *mut Registry),
    pub(crate) borrow: std::cell::RefMut<'m, T>,
}

I also have a type Manifested<T> that serves as UUID handle.

pub struct Manifested<T: ?Sized + Manifest + 'static> {
    pub(crate) inner: Option<(Uuid, *mut Registry)>,
    phantom: PhantomData<T>
}

In my architecture all primary objects live inside RefCells in a hashmap inside the “Registry”, mapped to UUIDs.

pub struct Registry {
    manifestations_by_uuid: HashMap<Uuid, (TypeId, RefCell<Box<dyn Anything>>)>,
}

Calling “borrow()” on a Manifested<T> struct borrows the RefMut from the Registry and gives you the Manifestly<’m, T> borrow guard.

In my framework both Manifested<T> and Manifestly<’m, T> supports dyn types as T due to the ?Sized bound, eg: Manifested<dyn Essence>, Manifestly<’m, dyn Essence>.

and it would be ideal for my trait API to look like this

//========== Trait declarations ==========//
// Trait Inspect is for custom egui inspectors
#[manifestable]
trait Inspect {
  fn inspect(self: Manifestly<Self>, ctx: &mut InspectContext, ui: &mut egui::Ui);
}

// Trait Essence act as a component of an ECS architecture
#[manifestable]
trait Essence {
  fn start(self: Manifestly<Self>, ctx: &mut EssenceContext);
  fn update(self: Manifestly<Self>, ctx: &mut EssenceContext);
}

(Note: #[manifestable] and #[manifest_as(Trait)] are proc macros for my inventory-based type registry system - they handle implementing of core traits handle and type registration.)

//========== Implementation following trait API: ==========//

#[manifestable]
struct TransformEssence {}

#[manifest_as(Inspect)]
impl Inspect for TransformEssence {
  fn inspect(self: Manifestly<Self>, ctx: &mut InspectContext, ui: &mut egui::Ui) {
    
  }
}

#[manifest_as(Essence)]
impl Essence for TransformEssence {
  fn start(self: Manifestly<Self>, ctx: &mut EssenceContext, ui: &mut egui::Ui) {
    
  }
  fn update(self: Manifestly<Self>, ctx: &mut EssenceContext, ui: &mut egui::Ui) {
    
  }
}

//========== Usage code ==========//
fn call_polymorphically(essence: Manifested<dyn Essence>) {
    let guard = essence.borrow(); // Manifestly<'_, dyn Essence>
    guard.update(); // ❌ Error: not dyn compatible because trait method uses arbitrary self type
}

However, Manifestly<Self> as the self parameter makes the trait not dyn compatible.

let essence: Manifested<dyn Essence> = get_some_essence();
let manifestly = essence.borrow();  // Manifestly<'_, dyn Essence>
manifestly.start(&mut context);  // ❌ Currently doesn't work with dyn

Declaring “where Self: Sized” makes the method not available on the trait object.

#[manifestable]
trait Essence {
  fn start(self: Manifestly<Self>, ctx: &mut EssenceContext) where Self: Sized;
  fn update(self: Manifestly<Self>, ctx: &mut EssenceContext) where Self: Sized;
}

// adding “where Self: Sized” to the methods makes those methods not available on the trait object “dyn Essence”.

Why I need “self: Manifestly<Self> to be trait-object compatible”:

  • ability to call a trait method that uses “self: Manifestly<Self>” calling the method from Manifestly<dyn MyTrait>. I would have a polymorphic collection of Manifested<dyn Essence>, and for each of them, I would want to call borrow() on the Manifested<dyn Essence> to obtain Manifestly<’m, dyn Essence> and invoke the trait method.

  • the ability to temporarily unborrow the RefMut inside a method using it. I have a trait “Encompass” in my game engine which is implemented for every primary object. It recursively iterates the owned children of each object, used for deep cloning purposes. Lets say I have an object A, and object B is the parent of object A. inside the start() method of object A, I might want to deep clone object B whose Encompass will cover object A and need to borrow object A to iterate over object A’s children as part of the whole process, and needs object A in an unborrowed state. But to call start() of object A, object A needs to be borrowed in the form of a RefMut from its RefCell in the registry. Having &self as the self arg in start() means you can’t drop “self” to unborrow the RefMut. But having RefMut<Self> (or in my case Manifestly<Self>) as the self arg means you can temporarily drop “self” to unborrow it so that the object will be freed for Encompass’s object traversal to borrow it.

  • self parameter containing the UUID. For API cleaniness, there would be operations like ctx.record_undo() and ctx.schedule_save() which needs the UUID identity of “self”, which uses the UUID to look up the actual object in the registry to perform those operations.

Impl Inspect for TransformEssence {
  fn inspect(self: Manifestly<Self>, ctx: &mut InspectContext, ui: &mut egui::Ui) {
    if ui::button(“add one”).clicked() {
      self.x += 1;
      ctx.schedule_save(self); //UUID contained in Manifestly<Self> self arg allows operation to target the object in the registry
    }
  }
}
  • using enum instead of trait objects would not work because users of the game engine would write their own types that cannot be included in a library-defined enum

So, Is arbitrary_self_types with dyn compatibility theoretically possible given my Manifestly contains RefMut? Would an RFC for this be welcome, or is this fundamentally incompatible with Rust's object model?

I'm happy to provide more details, test implementations, or help with RFC drafting if there's a viable path forward. If this is possible it will enable my game engine to have a very clean and ergonomic API.

In principle, there is comparibility of arbitrary_self_types with dyn compatibility, but it’s more limited. The current implementation uses the unstable trait DispatchFromDyn to mark the types that can be self types in dyn compatible traits.

You can’t implement DispatchFromDyn for your Manifested type though, because DispatchFromDyn [a bit like Copy] is only an opt-in, with the actual mechanism for doing the dispatching for trait objects being baked into the compiler and coming with its own requirements on your type. To be allowed to use DispatchFromDyn, we currently define (according to the documentation) the following requirements

For the conversion to work, the following properties must hold (i.e., it is only safe to implement DispatchFromDyn for types which have these properties, these are also checked by the compiler):

  • EITHER Self and T are either both references or both raw pointers; in either case, with the same mutability.
  • OR, all of the following hold
    • Self and T must have the same type constructor, and only vary in a single type parameter formal (the coerced type, e.g., impl DispatchFromDyn<Rc<T>> for Rc<U> is ok and the single type parameter (instantiated with T or U) is the coerced type, impl DispatchFromDyn<Arc<T>> for Rc<U> is not ok).
    • The definition for Self must be a struct.
    • The definition for Self must not be #[repr(packed)] or #[repr(C)].
    • Other than one-aligned, zero-sized fields, the definition for Self must have exactly one field and that field’s type must be the coerced type. Furthermore, Self’s field type must implement DispatchFromDyn<F> where F is the type of T’s field type.

The last one is your main issue, basically boiling down to DispatchFromDyn only being available to true pointer types, that contain no additional information besides the pointer.

I believe these restrictions are either motivated with some more fundamental concerns on complications with layout & calling conventions, or maybe they are mostly limitations of the current implementation. I’m not sure off the top of my head; also see this comment in the compiler source, I guess.

Essentially, if you call e.g. a fn(self: Box<Self>, …) trait method on a Box<dyn TraitName> type, the vtable for TypeFoo: TraitName impl has some entry with a fn(self: Box<TypeFoo>, …) function pointer for the concrete implementation. So you need some way of converting the Box<dyn TraitName> into a Box<TypeFoo> without knowing the concrete type TypeFoo, and then you need to be able to call that fn(self: Box<TypeFoo>, …), again without knowing that it’s actually the concrete type TypeFoo. So at least you gotta be sure that Box<T> is identical in layout for every possible Sized type T, and moreover that function calls with self: Box<T> argument (plus additional args, plus a return type; but none of those containing T) have the exact same calling conventions for all Sized types T. The same kind of thing would be necessary for any custom type in place of Box.

3 Likes

Thank you for the detailed technical explanation about DispatchFromDyn!

So it sounds like the current mechanism relies on pointer-ABI-compatibility, which rules out types with additional fields like mine.

In my case, Manifestly<'m, T: ?Sized> actually has uniform layout for all T:

pub struct Manifestly<'m, T: ?Sized> {
    inner: (Uuid, *mut Registry),  // 24 bytes (fixed)
    borrow: RefMut<'m, T>,         // 16 bytes (fat pointer)
}
// Total: 40 bytes for any T: ?Sized

Since RefMut<'m, T> is already a fat pointer for unsized T, the size/alignment is the same whether T = ConcreteType or T = dyn Trait.

So the remaining question would be are the DispatchFromDyn restrictions:

  1. Fundamental - Layout/ABI concerns that can't be worked around?
  2. Implementation limitation - Could theoretically be relaxed for types that can prove uniform layout?

I'm happy to use workarounds if this is fundamentally impossible, but wanted to understand if there's a potential path forward (even if it would require significant compiler work).

(I know you like to use different font sizes, but is kind of hard to read on my phone. I can zoom in the page to read the small text, but then the text width is wider than my screen so I have to keep panning left and right. Usually it isn't a big deal, but here we have 15 lines, making it quite hard to track where I am. Perhaps consider a different approach?)

1 Like

Maybe get a browser with a min font size control?

image

Can't find that in Brave (and that is the only mobile browser with good adblocking it seems, Firefox runs very poorly on my phone as well).

You’ve gotta use the good zoom setting :wink: :wink:

2 Likes