I can’t find the code in LLVM anymore so perhaps it has been fixed now. What was happening was that arguments were considered always aligned to the pointee’s alignment even without any attributes.
The only remaining such logic appears to be specifically for loads (maybe stores too?), i.e. an unannotated load from a pointer which didn’t come from an alignment-specifying-source is always aligned, which seems reasonable.