Beter support for x86-64-v2, x86-64-v3, and x86-64-v4 targeting

[indygreg]

(This post is a logical continuation of https://internals.rust-lang.org/t/x86-64-v2-and-x86-64-v3-targets-for-linux-and-windows/14030.)

The X86-64 System V ABI working group has defined x86-64-v2, x86-64-v3, and x86-64-v4 microarchitecture levels so you can target more modern CPUs instead of limiting compiled instructions to the ~2003 era x86-64 instruction set (the default for ~all compilers targeting x86-64). These are conceptually distinct target triples because the target CPU is different and produced binaries would (likely) be incompatible with older CPUs since they would contain modern instructions (like AVX) not understood by older CPUs.

Modern versions of Clang and GCC support these CPU targets and triples. And some Linux distros (e.g. RedHat) are moving to support x86-64-v2 (and maybe -v3) precompiled packages out-of-the-box.

Discussion in Add x86-64-v2, x86-64-v3, and x86-64-v4 as available target_cpus · Issue #82024 · rust-lang/rust · GitHub and an experiment at Build the compiler with -Ctarget-cpu=x86-64-v2 by est31 · Pull Request #79043 · rust-lang/rust · GitHub demonstrate that Rust today supports the x86-64-v2, x86-64-v3, and x86-64-v4 CPU targets by passing e.g. -Ctarget-cpu=x86-64-v3 as a compiler flag.

I think it is only a matter of time before industry realizes that the default of building x86-64 binaries not leveraging ~20 years of x86-64 ISA innovation is sacrificing a lot of performance potential and we start seeing a shift to ditch the original x86-64 ISA as the default compiler target. (The Clear Linux distro and its superior performance partially enabled by aggressive ISA targeting defaults is an existence proof that a lot of software is artificially limited in performance due to legacy ISA targeting.)

While it is possible to tell Rust today to emit more modern x86-64 instructions using flags like -Ctarget-cpu, the lack of formally defined triples for x86-64-v* causes real problems. For example:

• cargo build --target x86_64_v3-unknown-linux-gnu don't just work because the x86_64_v* variants aren't recognized.
• rustup can't install a [default] toolchain that targets a modern CPU level (presumably because Rust toolchains for these targets aren't published).
• Relying on .cargo/config files to override the default CPU via e.g. build.rustflags is brittle since other config files may override this field and undo default targeting. (This is obviously part of a general problem with how these config files work.)
• The ubiquitous platforms crate doesn't recognize triples like x86_64_v2-unknown-linux-gnu since this triple is not published to https://raw.githubusercontent.com/rust-lang/rust/master/src/doc/rustc/src/platform-support.md.

I feel like the current ergonomics of x86-64-v* targeting in Rust are sub-optimal. And this will likely get worse over time as x86-64-v2+ targeting becomes the norm and more and more people get tripped up by Rust not doing/supporting what their C/C++ toolchain does. We could even see situations where Linux distros are shipping Clang/GCC compilers that emit x86-64-v2 or x86-64-v3 by default but a rustup installed Rust is still targeting x86-64 by default. Blog posts claiming poor Rust performance vs C/C++/Go/etc due to ISA targeting differences ensue.

From my perspective as an end-user, it seems like Rust having some native support for x86_64_v2-*, x86_64_v3-*, and x86_64_v4-* target triples seems to make a lot of the ergonomic issues with modern ISA targeting go away.

Support for x86-64 microarchitecture levels in Rust target triples could take various forms. Perhaps Rust could internally normalize to an existing x86_64-* Rust toolchain but thread the newer target CPU to LLVM for codegen. Or maybe Rust goes all in and publishes actual x86_64_v*-* Rust toolchains. Maybe someday rustup sniffs for CPU features and automatically installs an e.g. x86_64_v3-unknown-linux-gnu toolchain when running on a modern CPU. There's a lot of room for partial and incremental changes towards better x86-64 targeting support in Rust.

(If you squint hard enough, similar problems exist for other CPU types. ARM, notably, has many CPU/ISA variants. And Rust today seems to define triple variants for many of them. Although this is arguably due to copying LLVM triples and LLVM likely did it out of necessity given lack of ISA compatibility between many ARM variants. But within what Rust/LLVM calls aarch64 there could be room to introduce better triple support for CPU targets to enable features like SVE.)

Should Rust have better support for x86-64 microarchitecture targeting and what form should that take, if any? And since I feel passionately about the topic (and the potential for the software industry to reduce its carbon footprint by improving efficiency through modern ISA targeting), how can I help?

[bjorn3]

@indygreg

These are conceptually distinct target triples because the target CPU is different and produced binaries would (likely) be incompatible with older CPUs since they would contain modern instructions (like AVX) not understood by older CPUs.

By the same argument every x86_64 cpu should have it's own target as code compiled for a newer target wouldn't work for an older target. -Ctarget-cpu is meant for using features of newer cpu's. --target is for incompatible architectures and OSes. You can link code compiled for -Ctarget-cpu=x86-64 into one compiled for -Ctarget-cpu=x86-64-v3, but you can't link code compiled for x86_64-unknown-linux-gnu into code compiled for x86_64_v3-unknown-linux-gnu as mixing target triples is not allowed.

@indygreg

Relying on .cargo/config files to override the default CPU via e.g. build.rustflags is brittle since other config files may override this field and undo default targeting. (This is obviously part of a general problem with how these config files work.)

That suggests we should get a separate target-cpu field in .cargo/config rather than lumping it together with other rustflags. This would prevent another .cargo/config from unintentionally overriding the target cpu and would also fix cross-compilation.

@indygreg

The ubiquitous platforms crate doesn't recognize triples like x86_64_v2-unknown-linux-gnu since this triple is not published to https://raw.githubusercontent.com/rust-lang/rust/master/src/doc/rustc/src/platform-support.md.

platforms recognizes rustc triples, not gcc/llvm triples. Trying to use it as such is not a good idea I think. We already don't support triples like x86_64-pc-linux-gnu or x86_64-unknown-linux-gnu-elf which I believe gcc and llvm accept. From the perspective of rustc triples are opaque strings, while gcc and llvm interpret it as arch-vendor-os-env-abi and allow skipping some parts I believe.

@indygreg

(If you squint hard enough, similar problems exist for other CPU types. ARM, notably, has many CPU/ISA variants. And Rust today seems to define triple variants for many of them. Although this is arguably due to copying LLVM triples and LLVM likely did it out of necessity given lack of ISA compatibility between many ARM variants. But within what Rust/LLVM calls aarch64 there could be room to introduce better triple support for CPU targets to enable features like SVE.)

I believe on ARM the variants all have incompatible ABI's and as such need to be different targets.

The only advantage of having x86-64-v2/3/4 triples I think is that it would allow compiling the standard library for it.

[tschuett]

I believe ABI is less of a problem for ARM:

GitHub - aws/aws-graviton-getting-started: This document is meant to help new...

The odd thing is that real world cores are , e.g., ARMv8.2-a + additional features.

[comex]

@bjorn3

The only advantage of having x86-64-v2/3/4 triples I think is that it would allow compiling the standard library for it.

Which is a significant advantage.

But there's also the question of which feature level is used by default. Suppose a distro does start shipping "Clang/GCC compilers that emit x86-64-v2 or x86-64-v3 by default" - i.e., if you run gcc foo.c, you get an x86-64-vN binary rather than base x86-64.

At minimum, that distro would probably want the copy of rustc it ships to default to x86-64-vN, for consistency's sake.

Beyond that, a rustup-installed rustc should probably also use that default. If nothing else, trying to compile for base x86-64 would be a promise rustc couldn't keep. After all, most programs will link to libraries from the system, including default libraries like libc and crt*.o, as well as libraries linked by -sys crates. On such a distro, all of those libraries would be built for x86-64-vN. If any of those libraries either (a) is a static library, or (b) is a shared library but has different symbol names on x86-64-vN (unlikely but not impossible if the distro is treating it as a separate triple), then the resulting binary is likely to be incompatible with base x86-64 systems. (In particular, crt*.o are always static, though since they're trivial and written in assembly, they're unlikely to actually use newer x86-64 features…)

But changing the default isn't just a matter of patching the default cpu for the x86-64-unknown-linux-gnu target. If the user is cross-compiling for some other x86-64 Linux system, then at minimum the modified default must not apply; to do otherwise would violate the principle that cross compilation should produce identical results on any host system. Beyond that, ideally rustc wouldn't just revert to base x86-64 but would know how to pick an appropriate default for the target system.

To me that feels like a job for a separate target triple...