FWIW as precedent, we do allow shifting by arbitrary types, including signed ones, even though shifting by a negative is never correct (and panics in debug), like how indexing a slice by a negative is never correct (and panics).
I would happily allow indexing by everything if we could do it without inference impact.