There's long been a desire to have smarter inference in a whole bunch of cases, like being able to index arrays by more types without affecting inference, being able to do mixed comparisons without affecting inference, etc.
I don't think there are any easy fixes. But the GitHub - rust-lang/types-team: Home of the "types team", affiliated with the compiler and lang teams. is chartered with improving this kind of thing -- probably starting with finishing the new inference engine -- so if you're interesting in working on this stuff, you might want to join in with them.
You can always put code in if false if you want it to affect inference and not run -- I use that in Pre-pre-RFC: syntactic sugar for `Default::default()` - #7 by scottmcm , for example.