IEEE 754 is better than all alternatives, at least among fixed-width formats. This is more pronounced in other areas, but division by zero resulting in infinity is certainly not objectively terrible or a quirk: It works out splendidly for many numeric algorithms. I appreciate that you have no use for that behavior but please don’t make the mistake of extrapolating from that to all use cases.
Additionally, IEEE 754 provides traps/exceptions not only for illegal operations that would produce NaN, but also for division by zero, overflow, underflow, and even ordinary rounding errors. Most languages, including Rust, ignore that and offer no good way to use those capabilities, but that is not really the IEEE’s fault. Speaking of which… good support for floating point traps would be an easy and low-overhead way to check for division by zero when you care about it. I have no idea what “good support” would look like, though.