This is usually solved by using some serialization library.
At minimum you could use byteorder to read individual numbers (I assume that 8 is for u64).
You could also make the messages use MessagePack, bincode, protobuf or something else, and have the compiler generate correct serialisation and deserialisation code automatically.