Hmm, raises an interesting API option. If there was a const such that one could say .or_insert(Vec::EMPTY) and because it's a const be confident that it's not a concern.
And it looks like if you do that, clippy is happy with it: https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=4cecdfcc136a38260852437de859b1d4
Looks like the const blocks that CAD97 mentioned also cause clippy to not warn for this: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=dc850d10fc9a205846c86b73f02ad60c