@cuviper’s proposal appears to be the most common, and I’ve found several documents citing it to IEEE 754-2008. The only contrary definitions I could find are Intel and ARMv7; ARMv8 provides the option to follow IEEE(?).
POSIX/C99 math.h fmax wants the non-NaN value:
NaN arguments shall be treated as missing data: if one argument is a NaN and the other numeric, then these functions shall choose the numeric value.
The RISC-V spec supports 3.0, unless you have signalling NaNs (user ISA spec, §7.2):
For FMIN and FMAX, if at least one input is a signaling NaN, or if both inputs are quiet NaNs,
the result is the canonical NaN. If one operand is a quiet NaN and the other is not a NaN, the
result is the non-NaN operand.
Intel SSE2 MAXSD behavior isn’t even symmetric:
If only one value is a NaN (SNaN or QNaN) for this instruction, the second source operand, either a NaN or a valid floating-point value, is written to the result. If instead of this behavior, it is required that the NaN of either source operand be returned, the action of MAXSD can be emulated using a sequence of instructions, such as, a comparison followed by AND, ANDN and OR.
ARMv7 VFP VMAX appears to give NaNs precedence as for any other arithmetic operation, but it’s only described in pseudocode.
ARMv8 (ARMv8 Architecture Reference Manual, §C7.2.90) has FMAX with the ARMv7 behavior, and FMAXNM which matches RISC-V:
NaNs are handled according to the IEEE 754-2008 standard. If one vector element is numeric and the other is a quiet NaN, the result placed in the vector is the numerical value, otherwise the result is identical to FMAX (scalar).