Are you talking about x86-64 Linux? Because this is exactly what happens. Memory requests by applications (via private mmap or sbrk) don't fail, since they're satisfied by mapping the zero page over the entire range. When the application tries to write to one of these, the page fault handler remaps to a free page. If it can't find one, it triggers the OOM killer. This is the principle of overcommit.
I don't remember ever seeing malloc fail on Linux.
In general, the greediest process is killed, which might well be the one triggering the killer.