Pre-RFC: Define the behavior of `repr(transparent)` when all fields are zero-sized types

language design Unsafe Code Guidelines
1

Summary

Generalize #[repr(transparent)] to include structs/one-variant enums with all zero-sized types.

Motivation

In order to fully implement the newtype pattern in unsafe code, Rust introduced the #[repr(transparent)] attribute. This allows us to create a wrapper around a type which is guaranteed to be identical in layout and ABI to the original type, but which can have additional type-level information. This is very useful in unsafe contexts, especially for FFI.

However, according to the current specification, #[repr(transparent)] cannot be used to create newtype wrapper around a zero-sized types. According to the Rust Reference, "[t]he transparent representation can only be used on a struct or an enum with a single variant that has:

  • a single field with non-zero size, and
  • any number of fields with size 0 and alignment 1 (e.g. PhantomData)"

Language in the Rustonomicon is similar but less detailed.

Note that this restriction is not currently fully enforced by the compiler. The compiler currently (as of 1.72.0) appears to allow a struct with only 1-ZST fields (zero-sized types with alignment 1) to have the #[repr(transparent)] attribute, contrary to the documentation. The compiler will not permit the attribute to be used when one of the fields is a ZST with alignment > 1, except when this arises through generics. Here are some examples of declarations the current compiler allows and disallows.

// allowed
#[repr(transparent)]
struct ManyZSTs((), PhantomData<i32>, [u8; 0]);
    
// allowed
#[repr(transparent)]
struct NoFields;
    
// #[repr(transparent)] not allowed here because 
// u16, and hence [u16; 2], has alignment 2
struct ZSTAlignTwo([u16; 0]);
    
// allowed
#[repr(transparent)]
struct GenericWrapper<T>(T);

type ZSTWrapper = GenericWrapper<[u16; 0]>;

There have been several issues/pull requests related to this topic.

Guide-level explanation

The repr(transparent) attribute should be used when defining a struct, or an enum with only one variant, that is guaranteed to have the same size, alignment, and ABI as another type.

Such a struct or enum will have at most one inner field. All non-inner fields must have size 0 and alignment 1. A programmer can choose to explicitly mark the inner field with #[inner].

In many cases, it is very obvious which field should be marked #[inner]. When there is only one field that the programmer could have explicitly marked as inner, that field is the inner field.

A type defined with the repr(transparent) attribute will have the same size, alignment, and ABI as its inner field, if it has one.

If there is no inner field, then we know all fields must have size zero and alignment 1 (since all fields are non-inner fields). Consequently, the repr(transparent)-defined type will also have size zero and alignment 1. However, there is no guarantee at this time that all types with size zero and alignment 1 have the same ABI, so the repr(transparent)-marked struct or enum may have a different ABI from some or all of its fields.

Examples

use std::cell::UnsafeCell;
    
#[repr(transparent)]
pub struct Cell<T> {
    inner: UnsafeCell<T>,
}

In the above example, the inner field is inner: UnsafeCell<T> - it's easy to see this, because there's no other field at all. A Cell<T> is guaranteed to have the same size, layout, and ABI as an UnsafeCell<T>.

use std::marker::PhantomData;
    
#[repr(transparent)]
struct Invariant<'a, T> {
    value: T,
    phantom: PhantomData<fn(&'a()) -> &'a ()>
}

The phantom field could not have been designated as the inner field, because value could have a type like i32. However, the programmer could have picked value as the inner field, because PhantomData always has size 0 and alignment 1. Therefore, value is the compiler-inferred inner field.

The type Invariant<'a, T> will have the same size, alignment, and ABI as T, and it will depend invariantly on the lifetime 'a.

#[repr(transparent)]
struct InvariantUnit<'a>((), PhantomData<fn(&'a()) -> &'a ()>);

In the above example, both () and PhantomData are guaranteed to be 1-ZSTs. We could have chosen to annotate either one as the inner field. Because there's more than one possible choice, neither field is the inner field.

#[repr(transparent)]
struct InvariantUnit<'a>(#[inner] (), PhantomData<fn(&'a()) -> &'a ()>);

In the new code, we have an inner field of type (), so InvariantUnit and () have the same layout and ABI. We could instead have annotated the other field; InvariantUnit would then have the same layout and ABI as PhantomData.

Reference-level explanation

Semi-formal documentation for the new repr(transparent)

Define a 1-ZST to be a type with size zero and alignment 1.

A struct (or an enum with a single variant) may be marked with the repr(transparent) attribute if all the fields of the struct (or of the single enum variant) are guaranteed to be 1-ZSTs, except at most one. Violation of this condition is a compile-time error.

At most one field of a struct or enum marked with repr(transparent) can be marked with the attribute inner. If a field is marked with this attribute, all other fields must be 1-ZSTs. Violation of this condition is a compile-time error. Using the inner attribute in any other context is a compile-time error.

If there is a field of a repr(transparent) struct or enum which could, after generic substitution, not be a 1-ZST; or if there is exactly one field in the struct or enum; or if a field in the struct or enum is marked inner, this field shall be known as the "inner field". Note it's possible for there to be more than one reason why a field is the inner field; all three of these criteria could be satisfied, for instance. However, it's not possible for more than one field to meet at least one of the above criteria. If no field satisfies at least one of the criteria to be an inner field, the type has no inner field.

A repr(transparent) type with an inner field shall have the same size, alignment, and ABI as its inner field.

Otherwise, it follows that all fields of the repr(transparent) struct or enum are 1-ZSTs. The struct or enum itself shall be guaranteed to be a 1-ZST. However, no guarantees about the ABI of such a struct or enum are made at this time.

Explanation

We want to allow as many cases as reasonable where #[inner] is not necessary, since we want to avoid breaking existing code. The case where there is only one field is an obvious case where we don't need to explicitly mark the inner type. The case where one field is known not to be a 1-ZST is another. In my view, it is a natural (though not totally unproblematic) extension to say that when one of the fields is generic and could, upon generic substitution, not be a 1-ZST, this field is the inner type.

In a situation where we have multiple fields, all guaranteed to be 1-ZSTs, there seems to be no principled way of deciding which field's ABI the struct on the whole should adopt (if we grant the assumption that two 1-ZSTs could conceivably have different ABIs). However, it does seem obvious that the type should be guaranteed to be a 1-ZST. The same guarantee can be made for structs with no fields at all.

Drawbacks

Under my specification, we could have silent breakage of code that switches from a generic declaration to a concrete one (eg Invariant<()> to InvariantUnit). We could also have silent breaking if the type being wrapped changes from being a non-1-ZST to a 1-ZST. Note that these changes are already potentially breaking; my proposal simply fails to fix them. This breaking would eventually be addressed by a lint, so the breaking would not be silent. It could also be addressed by adopting the best practice of generally using #[inner], which could be enforced by another lint.

Rationale and alternatives

Alternative 1: declare that all 1-ZSTs have the same ABI, and that a repr(transparent) struct/enum with all 1-ZST fields is a 1-ZST.

If we declare that all 1-ZSTs have the same ABI, this means we don't actually need to mark the inner type at all. It doesn't matter if all fields are 1-ZSTs, since the resulting type will be a 1-ZST and thus have the same ABI as all its fields.

Drawbacks of this alternative: structs with zero elements that are declared with repr(C) could be problematic across extern "C" FFI boundaries. Do we actually know that returning a 1-ZST declared with extern "C" will be equivalent to returning void on the C side? Are there any other extern ABIs that Rust supports, now or in the future, that could cause a problem?

Implementing my proposal leaves the door for Alternative 1 open in the future.

Alternative 2: declare that a repr(transparent) struct with all 1-ZST fields is equivalent to ()

This was considered here. It potentially eliminates the need for explicit #[inner] annotation, since we define the ABI for all possible repr(transparent) structs.

The main issue with alternative 2 is similar to that of alternative 1. In fact, alternative 1 is strictly stronger - if all 1-ZSTs are equivalent, then a repr(transparent) struct with all 1-ZST fields, itself a 1-ZST, must be equivalent to ().

But what if we don't want to commit to living in the world of Alternative 1? Let A be a 1-ZST with an ABI different from ().

#[repr(transparent)]
struct AWrapper(A);

If we adopt the above rule, AWrapper must be ABI-equivalent to (). Thus, if A is not ABI-equivalent to (), we just wrote something which looked like a wrapper around A, but is actually an ABI-incompatible type. That is the sort of subtle bug which could lead to nightmarish FFI confusion and undefined behavior.

Alternative 3: Do not try to define the ABI of a repr(transparent) type with all fields 1-ZSTs, and do not introduce #[inner]. Do state that the unique field of a repr(transparent) struct that's not a 1-ZST is the inner field.

This alternative leaves us no way to construct a newtype wrapper around a 1-ZST. That is quite unsatisfactory.

Furthermore, it leaves the status of generic repr(transparent) structs quite unclear. A generic wrapper struct

#[repr(transparent)]
struct Wrapper<T>(T);

would not have any guarantees at all when T is a 1-ZST.

Alternative 4: A repr(transparent) struct/enum with all fields 1-ZSTs has the ABI of its first field

This was also considered here. This has the issue that going from a generic struct to a concrete struct might silently break code. Let A be a 1-ZST with an ABI different from (). Going from

#[repr(transparent)]
struct GenericWrapper<T>((), T);
type SpecificWrapper = GenericWrapper<A>;

to

#[repr(transparent)]
struct SpecificWrapper((), A);

would mean SpecificWrapper no longer wraps A; it instead wraps ().

Alternative 6: instead of annotating a field with #[inner], annotate the type with #[repr(transparent, inner = inner_field)].

This may not work well for tuple structs. Adding or reordering fields would potentially break code.

Alternative 7: do nothing

See motivation for why we shouldn't do nothing.

Prior art

None, as far as I know.

Unresolved questions

None, as far as I know.

Future possibilities

To assist users in navigating generalized repr(transparent), we could add two supporting lints.

no_inner_field

This lint will be triggered when a type is defined with repr(transparent), but there is no inner type. Users may find this unexpected, since in their minds, they likely intended one field to be the inner type. The full power of this representation comes from having an inner field; lacking one is very likely a mistake. This should probably have a default level of warn or force-warn, since giving it a higher level will break code that currently compiles (such as InvariantUnit above).

implicit_inner_field

This lint will be triggered when a type is defined with repr(transparent) and has an inner field that is not explicitly annotated. In this situation, a few undesirable properties are possible. Rewriting the struct from generic to concrete could cause there to be no inner type (for instance, going from Invariant to InvariantUnit above). Furthermore, if there is a single 1-ZST field, adding additional 1-ZST fields would cause there to be no inner type.

However, there will be plenty of situations where it's totally clear which field is supposed to be (and is in fact) the inner type. Furthermore, there is a lot of preexisting code that does not explicitly annotate the innner field (because this annotation doesn't exist). In these situations, this lint would just be an annoyance. Thus, I'd probably say this lint should be at the allow level by default.

3 Likes
2

If #[inner] comes to exist, I think the language also, still, having implicit inner fields with a nontrivial selection rule is unnecessarily surprising to future newcomers. How about we go a little farther and have a simple, explicit rule, as follows?

In the next edition, a #[repr(transparent)] struct must either:

  1. have exactly one field, or
  2. have an #[inner] attribute on exactly one field.

In the current edition, this restriction would be provided as a lint, which would have a suggestion to add #[inner] to the non-1-ZST field. This provides edition migration, and also might or might not be warn-by-default.

This way, the interpretation of #[repr(transparent)] would no longer depend on invisible layout information about the fields' types at all.

2 Likes
3

Isn't the behaviour here already lang-FCP'd in https://github.com/rust-lang/rust/issues/77841#issuecomment-716575747?

A repr(transparent) type T must meet the following rules:

  • It may have any number of 1-ZST fields
  • In addition, it may have at most one other field of type U

If that other field is present, then T behaves like type U for ABI purposes. Otherwise, T consists only of 1-ZST types, and it behaves like (), which is the "canonical" 1-ZST.

So if that's not what the compiler does today, I think it can just be changed.

5 Likes
4

Is there any obstacle to declaring that a repr(transparent) struct with only 1-ZST fields is ABI-equivalent to all of its fields? (Or, equivalently: all 1-ZST types are ABI-equivalent to ().) That seems like the best possible option, so we should pursue more complicated alternatives such as requiring an explicit "inner" field only if necessary.

C doesn't really have zero-sized fields, so ABIs designed for C will probably not cover this case. It might be on us to define what happens with 1-ZST (and higher-alignment ZST, but these are not relevant here). For the return type we can just say that it behaves exactly like void in C. For arguments I'm not sure what we do for (), but we have to be doing something, and we should be doing that same thing for all 1-ZST. (Some ABIs just skip these arguments AFAIK, but not all do. That's fine as long as () and other 1-ZST are all treated the same way.)

We had an explicit FCP for this as @scottmcm noted, so I think this is just outdated documentation.

The fact that the compiler rejects repr(transparent) on struct ZSTAlignTwo([u16; 0]); is a bug that should be fixed.

1 Like
5

I can't come up with any reason to need to pass zst differently on an ABI level. In fact I don't see why they would be passed at all instead of eliminated. You mention that some targets do that though, do you have any examples?

Knowing the details of those targets and why they do what they do might shed some light on if there is a need for different ZSTs to be passed differently.

And is there a good reason for it or just random historical reasons? (I know that there are some truly weird ABIs out there after all, like M68k passing integers and pointers in different registers, so I guess everything is possible.)

6
2 Likes
7

I like the comment there.

Unfortunately it sounds like no one knows why though? Presumably it is something that was decided upon early during rust history and we are now stuck with, since it would break the ABI (and it is not a thing that C has, so it can't be inherited from there).

I don't see any reasons from that code why all zst couldn't be equivalent though, so that approach seems simpler and sane to me.

8

Somewhat interestingly, (), #[repr(C)] struct Zst;, and #[repr(transparent)] struct Zst(()); all cause improper_ctypes warnings, but #[repr(C)] struct Zst(()); doesn't.

Even more interestingly imo, the #[repr(transparent)] warning is phrased

warning: `extern` block uses type `Zst`, which is not FFI-safe
 --> src/lib.rs:2:16
  |
2 |     fn oops(_: Zst);
  |                ^^^ not FFI-safe
  |
  = note: this struct contains only zero-sized fields
note: the type is defined here
 --> src/lib.rs:6:1
  |
6 | struct Zst(());
  | ^^^^^^^^^^
  = note: `#[warn(improper_ctypes)]` on by default

which suggests that the lack of a warning for #[repr(C)] is a bug, not intentional.

The improper_ctypes warning can sometimes be a bit overeager (e.g. warning on references to improper_ctypes, despite the references themselves having a fully defined and guaranteed C ABI), but the warning does suggest that zero-sized structs do not have a stably guaranteed ABI through extern "C". (extern "Rust" of course we are fully capable of changing whenever and however we want.)

If "target C" contains an extension that allows zero-sized structs to exist and passes them (and it's not just that it "permits" zero-sized type values by making them actually have size[1]), then we should probably still try to match "target C" though, so long as it doesn't make our semantics completely unreasonable.

  1. Some people will say that if "target C" supports defining a struct with no fields and that creates a struct with a nonzero size, then #[repr(C)] should replicate that behavior in Rust on that target. I respectfully disagree in this case. It's certainly an extra pitfall involved in translating C headers to Rust, but there are already plenty of similar target specific pitfalls that should ideally be considered when translating less-portable C. (A notable one being that composing overalignment and underalignment in MSVC behaves differently than in Rust, due to how MSVC handles a split between "required" and "preferred" alignment.) ↩︎

9

If we do say that a transparent struct with all 1-ZST fields must have the same ABI as (), I think we also need to specify that all 1-ZST types have the same ABI. If we have a 1-ZST with a different ABI than (), there would be no way to make a repr(transparent) wrapper around this type. Moreover, we would have to do special case analysis for a generic transparent struct when a concrete parameter is a 1-ZST with a different ABI than (). This seems unsatisfactory.

Some C compilers on some platforms appear to treat returning a ZST differently from returning void, though it's unclear whether the difference is just "skin-deep". See these examples, for example. I'm not too familiar with these versions of assembly language, but it looks like at a minimum, MIPS allocates space on the stack to store the ZST to be returned. I have no idea what's going on with MSP, but a lot of extra instructions are generated to call a function that returns a ZST than to call a function returning void. If all 1-ZSTs have the same ABI as (), then returning a 1-ZST in Rust must correspond to returning void whenever we use extern "C".

As far as ZST arguments go, I would suspect that most C compilers would treat any two 1-ZST arguments identically as far as ABI. This suspicion is not based on any thorough research or expertise, so I would not rely on it.

10

The rustc ABI logic makes all ZST (not just 1-ZST) return types PassMode::Ignore. So either this is harmless or the rustc ABI logic is buggy...

Yes, that follows pretty much by transitivity.

So... how would we go about guaranteeing that all 1-ZST are ABI-compatible? Assuming we don't currently have any ABI that violates this, were would such documentation go? (I don't think this needs an RFC, T-lang FCP should be enough.)

And how would we find out if any ABI does violate this? The ABI adjustment code for some targets is rather complex after all... for example, this special-cases FieldsShape::Union; does that mean a 1-ZST union gets a different ABI than a 1-ZST struct? I think not; in a 1-ZST struct it will not recursively find any Scalar/ScalarPair and anyway this is only used to find float fields... but who knows what else is lurking in these ABI adjustments.

12

Actually I found a counterexample:

Maybe we have to say that empty arrays do not count as 1-ZST... at that point we should probably find a new name for this. E.g. a type is "ABI-trivial" if it has the same ABI as (). And repr(transparent) only allows any number of ABI-trivial fields plus one extra field.

There are more targets in this scary list though and I've not checked at all how they treat various kinds of 1-ZST:

2 Likes
13

Thanks for finding this Rust counterexample. I was pretty sure such things could exist based on my limited knowledge of ABIs, but I did not know how to efficiently search for them. In another thread, I considered this idea of "ABI-trivial", which I called ()-types (types with the same layout and ABI as ()). The issues there are that (1) it is a breaking change, and (2) we have not actually defined when a type is guaranteed to be ABI-trivial and when it isn't. If [u8; 0] is ABI-trivial on some platforms but not on others, how would we handle that case?

The merits of requiring non-inner fields in a transparent declaration to be ()-types are, in my view, as follows. First, repr(transparent) provides 2 guarantees; ABI and layout. There is a sort of pleasing symmetry that it also requires non-inner fields to have a certain ABI and layout. It also makes sense that ABI-trivial types are the canonical "ignored" values, and they are ignored for determining the ABI and layout of the resulting struct/enum. Second, this does ensure that there is a way of wrapping any type we wish with repr(transparent), which IMO is an important requirement.

14

As my later example shows, this is not just a ZST problem. The extern "C" sparc64 ABI violates even simpler repr(transparent) cases, such as [u8; 8] vs Wrapper<[u8; 8]>. So when looking for a fix it's not really worth focusing on ZST, the problem is broader than that.

2 Likes
15

What is not clear to me from this thread or the linked bug is if this is the SPARC C ABI being weird (passing an array differently from a struct containing an array) or if it is Rust on SPARC being weird (not matching the SPARC C ABI due to bugs).

But even so (after referring to to rustinomicon) it seems pretty clear that repr transparent should FFI as it's inner type. So any ABI weirdness should only apply to repr C.

That doesn't quite seem to answer the other part though: should there be multiple types of ZSTs with different ABIs (which SPARC also seems to have)? Is this needed to actually match a C ABI? Or just to prevent breaking existing code since rust messed up it's C with extensions ABI first time around?

1 Like
16

I looked at the documentation for the sysv64 and win64 parameter passing ABIs.

sysv64 unconditionally passes (C++) types with non-trivial destructor by reference. By my reading, since it classifies each eightbyte of a (small) aggregate it fails to properly classify zero-sized objects (with trivial destructor), but all options at that point differ from by reference. So whether ZST arguments differ on sysv64 depends on whether a Rust type with an impl Drop gets caught by "a C++ object [which] has either a non-trivial copy constructor or a non-trivial destructor[1]." Because the justification in the footnote is address sensitivity, and Rust structs aren't, I'd argue it shouldn't, but it would be possible to argue it should apply to any language type with a destructor, not just languages with C++ style copy.

The win64 ABI treats all function parameters identically based on size, but since it handles user defined return values which are not 1, 2, 4, 8, 16, 32, or 64 bytes by out pointer, it results in an incompatible ABI for ZST returning functions than ones which return nothing (Rust (), C/++ void).

This illustrates at least why ZSTs might have different ABIs — the ABI is typically written with what is possible in C and C++ in mind, which doesn't include the possibility of passing/returning zero sized values. (C does not permit the definition of a fieldless struct or a zero-length array field, and C++ gives empty classes size 1 IIRC (caveats apply).) Almost as a rule, since passing zero byte objects is meaningless at the machine level, the ABI wasn't defined with that possibility in mind.

With that in mind, imo the question divides roughly into

  • Does the "target C compiler" permit the definition of properly[2] zero sized types? If so, Rust's copy of that target C ABI should match. Hope ZST and no return value are treated identically.
  • Is there another language with first party support which intentionally uses the same C compatible ABI and does have zero sized types? The same goes: match that behavior. The same again if the ABI documentation explicitly acknowledges zero sized objects.
  • Do we consider the Rust implementation of extern "C" OSSified, and that the parameter passing ABI of fixed repr types cannot be changed, even in the face of an improper_ctypes warning? If so, we need to keep bug-for-bug compatibility with current rustc. If not and not the above, we control the unstable ABI and can change it however necessary.

  1. A de/constructor is trivial if it is an implicitly-declared default de/constructor and if:

    • its class has no virtual functions and no virtual base classes, and
    • all the direct base classes of its class have trivial de/constructors, and
    • for all the nonstatic data members of its class that are of class type (or array thereof), each such class has a trivial de/constructor.

    An object with either a non-trivial copy constructor or a non-trivial destructor cannot be passed by value because such objects must have well defined addresses. Similar issues apply when returning an object from a function. ↩︎

  2. Actually taking zero bytes as a field, not something like MSVC C which gives otherwise empty structs int or pointer ABI (I don't recall the specifics). ↩︎

1 Like
17

It does allow a zero-sized array as function argument though... but that's just equivalent to a pointer I think? Do we promise that passing a zero-size array in Rust is compatible with that?

The Rust implementation of that ABI passes all arrays by-reference. So some zero-sized objects are classified as by-ref.

OTOH if passing Rust arrays by-value does not have to be compatible with a C function declared as void fun(int[N] x), then maybe this match arm can be skipped entirely for count == 0 (since those cannot exist as fields in a C type as you say)? (We need to be a bit careful with unsized types, but obviously those don't have a C equivalent.)

The new test I added here passes on win64-GNU, so I think currently we are treating () like struct Zst;.

18

In C you cannot pass an array, right? int foo(int[N] array) is the same as int foo(int* array). So as far as the C ABI is concerned, arrays only exist as struct fields.

At least I hope we don't promise that int foo(int[N] array) is compatible with extern "C" fn foo(array: [ffi::int; N]) -> ffi::int.^^

19

The size on an array parameter is just part of the documentation, it does not affect the ABI at all.

20

Okay so what I am hearing (above and elsewhere) is that in principle, we can always define the Rust version of extern "C" such that repr(transparent) works. We just need to ensure that zero-sized fields are skipped during layout adjustments. That cannot be incompatible with any standard C program since standard C programs cannot have such fields. Right?

The main risk then is running into a target where someone else has already extended the ABI to allow zero-sized fields, and they did that in a way that is incompatible with just skipping them. We currently don't know if that is the case for any of the targets -- except that we ourselves defined extensions e.g. to the sparc64 C ABI that are incompatible with our own goals, and there's a risk that changing that ABI could break existing code. OTOH sparc64 is an unmaintained tier 2 target so... maybe that risk is not very high.

What I am now very confused about, however, is this part of our ABI logic. That comment claims very definitely that certain platform C ABIs do not ignore zero-sized struct arguments. What is that based on, given that C has no such thing?

1 Like
21

GNU C extension, looks like.

Though since #[repr] is about ABI as well as layout, we'd be within rights to only match GNU C extension behavior for #[repr(C)] and not default repr types.