Yeah, I would at least experiment with trying to make it adapt to attack before deciding on this. It’s not even statistical collision resistance you’re going to be missing, just attack resistance. If you can catch an attack early (i.e. if you can define “imbalance” rigorously enough) and switch to SIP or something, then you might as well start with xxhash as it’s a speed champ and has perfectly good statistical properties on non-hostile inputs.
As for determinism, it seems to me that a with_seed operation that you can provide a constant seed to ought to do?