Byte-aligned Vec

[andygrove]

Hi,

I’m working on a rust implementation of Apache Arrow and one of the requirements is to have 64-byte aligned arrays of primitive types so I cannot simply use Vec<T> but have had to implement my own type that uses libc and lots of unsafe code.

I saw that Rust 1.25 introduced byte-aligned structs with #[repr(align(16))] and if I could somehow do the same for Vec I would be very happy. I’m wondering if any language features are in the works for this or whether there are other strategies for this?

Thanks,

Andy.

[jethrogb]

I’m assuming you don’t need to align the Vec itself but need to have a Vec where the elements are aligned? You can just define a type with the correct alignment and create a Vec of that:

#[repr(align(64))]
struct MyType {
    /* ... */
}
type Aligned = Vec<MyType>;

[andygrove]

No, I actually need the start of the data aligned. So if I have a Vec I need the data to start on a 64-byte aligned address but then have the items 4-byte aligned in the i32 example.

[jethrogb]

Oh, that’s an interesting requirement. I don’t think that’s supported in any standard/safe way. Can you allocate a Vec that’s too big and start indexing at a particular element offset to get your required alignment?

[andygrove]

That could work I guess. Seems kinda hacky and wasteful, but definitely safer than what I’m doing now.

[andygrove]

I guess what I asking for is Vec::with_capacity(n: usize, align: usize). I wonder how much work it would be to add something like that to Rust? It is probably not something I can tackle since I’m not a C expert.

[sfackler]

We wouldn’t want to add that API to Vec itself because it’d need to track the requested alignment as a new field. We’re getting close to stabilizing some of the allocator API which will let you make a custom type that does this though.

[andygrove]

allocator API does sound like the way to go for this. Are these stabilizations being targeted to a particular Rust release yet?

[sfackler]

It’s not totally pinned down, but hopefully sometime near 1.28 or so?

How does the Java and C++ implementations deal with the alignment requirement? You can always directly talk to the C allocator through posix_memalign/free to make these allocations now.

[andygrove]

OK, thanks. Yes, I’m using posix_memalign for now.

[jmst]

Just make a type that is a 64-byte aligned array of 16 i32s (or just 64 u8s, although if you are using it as i32s it’s more elegant to use i32s), make a Vec of that, and then a newtype over that Vec that implements Deref to &i32 by transmuting.

Add an extra size field to the newtype if you need to keep track of size at primitive granularity.

[andygrove]

That is pure genius. Simple and elegant. Thank you!

[kornel]

Just a safety note: that can be done only with read-only Deref for Vec. Any push to transmuted vec could re-allocate storage as i32s.

[system]

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.