Bumping glibc requirements for the Rust toolchain

[petrochenkov]

A recent survey shows that nothing may be using glibc between 2.5 and 2.12

SLES 11, glibc 2.11 reporting in.

[cuviper]

• Currently rustc is compiled in a CentOS 5 container which uses glibc 2.5
• However, the actual glibc version required ends up being 2.3.4

You scanned lib/*.so, but don’t forget bin/ too. rustc and rustdoc are trivial, but cargo and rustup have significant contents. They don’t seem to add anything in this case though. This still doesn’t tell us what dependencies glibc-2.12 will introduce, nor how our use may expand in the future. We should still assume that anything less than the glibc we build with is unsupported.

We should also watch for libgcc GCC_X.Y versions. RHEL5 has gcc-4.1.2, but in fact we do have a link to _Unwind_GetIPInfo@GCC_4.2.0. This one works though, because that exact symbol was also backported between 4.1.1 and 4.1.2.

It’s also worth noting that the devtoolset-gcc on RHEL6 (that @ishitatsuyuki is proposing to use) is specifically crafted to remain ABI compatible with the base-OS’s gcc-4.4.7 for libgcc etc. That’s something to be careful about when we build custom GCCs for our build containers.

[Cyberax]

Yes, in some minor applications.

[aidanhs]

I was a bit shocked by the people talking about RHEL 5 being in use so I looked it up and yes, it is still alive and supported (ish):

https://access.redhat.com/support/policy/updates/errata (also search for “Life-cycle Dates”)

In addition, for Red Hat Enterprise Linux 5, customers may purchase annual Add-on subscriptions called Extended Life-cycle Support (ELS) that provide similar support to Production Phase 3 through November 30, 2020.

Only 3.5 years to go, I’ll note it down in my calendar now! (I also note that RHEL 4 support only ended three weeks ago…)

Really though, I don’t think sticking with Centos 5 is great. We already build half of userland and that’s not going to get better in the next 3.5 years…after which we jump forwards to the positively futuristic Centos 6, which was old enough to be awkward to develop on at times 3 years ago. Nor is moving to Centos 6 ideal since we make anyone on RHEL 5 unhappy. Static linking isn’t really viable due to compiler plugins and cargo hitting the network.

So if we’re serious about RHEL 5, my dream solution is to bump the Docker image to use the latest Ubuntu LTS, build the older stuff that’s necessary to get the right environment for creating a portable rustc and run a checker to verify that we’re only linking to expected symbols. Probably not easy (there’s a reason everyone just uses the CentOS images), but possible. And the thought of ‘upgrading’ from CentOS 5 to 6 in 3.5 years is depressing - we’ll optimistically get rid of the userland compilation scripts, then incrementally add them back over time as we discover bugs in the versions from the package manager.

p.s. glibc is only one part of the story, since we can’t use most/all of these highlighted syscalls (sendmmsg surprised me too), but that’s a more tractable libraries problem rather than a “I can’t run rustc” problem.

[cuviper]

So if we’re serious about RHEL 5, my dream solution is to bump the Docker image to use the latest Ubuntu LTS, build the older stuff that’s necessary to get the right environment for creating a portable rustc and run a checker to verify that we’re only linking to expected symbols. Probably not easy (there’s a reason everyone just uses the CentOS images), but possible.

That’s basically what we have for all of the cross-arch builds, sans checker, using crosstool-ng in most cases to build the compatible toolchain. But you really end up building almost everything for this case, so there’s little advantage to using a newer Ubuntu as the base image. Native x86 builds can only loosely be called “cross”, but when you’re targeting a glibc different than the host, it’s essentially the same situation.

[jethrogb]

So if we’re serious about RHEL 5, my dream solution is to bump the Docker image to use the latest Ubuntu LTS, build the older stuff that’s necessary to get the right environment for creating a portable rustc and run a checker to verify that we’re only linking to expected symbols. Probably not easy (there’s a reason everyone just uses the CentOS images), but possible.

I think this is a fine plan. There is no reason for Rust to pin to an arbitrarily new GLIBC version, except if LLVM requires it.

Also see https://web-beta.archive.org/web/20100613022915/http://www.autopackage.org/apbuild-apgcc.php

p.s. glibc is only one part of the story, since we can’t use most/all of these highlighted syscalls (sendmmsg surprised me too), but that’s a more tractable libraries problem rather than a “I can’t run rustc” problem.

Rust doesn’t make syscalls manually. The libc that comes with your OS should know which syscalls to call for your kernel.

[aidanhs]

But you really end up building almost everything for this case, so there’s little advantage to using a newer Ubuntu as the base image.

(below I’m going to make a fake distinction between ‘userland’ (bash, python) and ‘toolchain’ (compilers))

Well, you end up building the toolchain. Things that you don’t end up building (I think?) include curl, python, cmake and git (more things to be added over time I’m sure), so I’m not sure “almost everything” is quite right. At the moment we’re building both the userland miscellanea and bits of the toolchain - why not just build the toolchain? (yes, I’m simplifying just a little for effect )

In the end, we will move off CentOS 5 at some point. I see the options as 1) CentOS 6 now/later and we get a new toolchain/environment with a new set of bugs and, independently, we’re adding new necessary hacks to both toolchain and userland as technology changes over time) or 2) get us off CentOS 5 now, make the dist builders look like the other builders, give us a rolling userland environment (so we don’t have to caretake) independent of the actual toolchain and make the toolchain version choices more explicit and controlled.

The only reason I can really see to avoid 2 is if you want to build rustc on RHEL 5. Right now you can just follow the dockerfile instructions and it’ll probably work. Given a world where 2 is implemented, you’re probably out of luck. I don’t find this particularly compelling given you’ll be able to build anywhere you have Docker and send it to your RHEL 5 machine, but I could be convinced.

Rust doesn’t make syscalls manually. The libc that comes with your OS should know which syscalls to call for your kernel.

I’m more referring to crates in the wider ecosystem that may (I don’t really know how much it happens) use interesting syscalls for performance, not expecting to be run on old Linuxes. It’ll just need PRs, but it’s still a speedbump.

[cuviper]

Point taken about building the toolchain vs other programs.

And FWIW, while I work for Red Hat, I’m not declaring a need on our part for continued Rust support on RHEL5. We certainly don’t ship any new software ourselves in the extended life-cycle.

But I do favor the conservative attitude, not to rock the boat here without reason. It works, and a few users have spoken up that they want it, so I think it needs stronger motivation to change.

Of course, their old rustc will keep working too…

[ishitatsuyuki]

Some arguments:

• There’s no need to support bleeding edge things for ancient distributions. They just don’t want to migrate, then a stable and out-of-date version will keep the old things (such like an unmaintained crate) working. Rust sometimes includes changes that reject old code.

• For extremists, the musl toolchain is to the rescue. They can compile things independently from the ABI.

[sfackler]

None of those arguments cover why it’s actually beneficial to make this change.

IIRC you can’t build rustc against MUSL anyway since plugins won’t work.

[aidanhs]

But I do favor the conservative attitude, not to rock the boat here without reason. It works, and a few users have spoken up that they want it, so I think it needs stronger motivation to change.

I’m absolutely not suggesting dropping support for RHEL 5 users and in general I too tend to be conservative. But since the boat has been rocked, we may as well talk about it.

For extremists, the musl toolchain is to the rescue.

They’re not extremists, they’re big businesses. And big businesses like to use things like NSS and internal crates.io mirrors - combining these two things won’t work with musl. And, as has been mentioned, plugins won’t work so you’ll miss out on serde derive as well.

None of those arguments cover why it’s actually beneficial to make this change.

For me it’s a recognition that we have to upgrade someday and that remaining on CentOS 5 imposes a non-zero maintenance burden, so why not do a compatibility-preserving upgrade now to benefit from an eased burden.

[cuviper]

so why not do a compatibility-preserving upgrade now to benefit from an eased burden.

By this, do you mean some modern base image with cross-tools targeting a RHEL5-equivalent? That’s less controversial, for sure. (and different than what @ishitatsuyuki was attempting.)

[aidanhs]

Precisely. Seems I need to work on my clarity when proposing things.

[sfackler]

Oh yeah, I would definitely be on board with a more modern platform if it can target the right glibc version!

[hsivonen]

However, the actual glibc version required ends up being 2.3.4

Is this meant to apply for x86/x86_64 only or is this meant to apply to other ISAs?

That is, is it a bug that Rust for big-endian 32-bit MIPS seems to have a higher glibc requirement?

/tmp/tmp.a7zWeNAkYp/rustup-init: /lib/mips-linux-gnu/libc.so.6: version `GLIBC_2.18' not found (required by /tmp/tmp.a7zWeNAkYp/rustup-init)
/tmp/tmp.a7zWeNAkYp/rustup-init: /lib/mips-linux-gnu/libc.so.6: version `GLIBC_2.16' not found (required by /tmp/tmp.a7zWeNAkYp/rustup-init)

(This is on Debian wheezy, which has glibc 2.13-38+deb7u10.)

[ishitatsuyuki]

No, it’s intended. Only the x86_64 and i686 Linux toolchains are built with compatibility options, since it would be impossible to bootstrap on older system if we broke them.

[ishitatsuyuki]

By this, do you mean some modern base image with cross-tools targeting a RHEL5-equivalent?

Unfortunately, there’s no maintained toolchain builder that can target glibc 2.5, due to the significant maintenance burden. I wasn’t able to build glibc 2.5, 2.6 and 2.11 on a modern system (glibc has non-standard conformant code and relied on compiler bugs), and I hesitate to try other old ones. crosstool-ng still have support for 2.12, while buildroot only supports the recent three versions.

[alexcrichton]

Oh sorry thanks for asking @hsivonen, it’s definitely worth clarifying!

Right now all of the various platforms have different glibc requirements. Most of them are managed through a cross-compiled toolchain, however. We’ve got ubuntu 16.04 containers running crosstool-ng to compile a toolchain with an older glibc requirement. We tend to try to pick a glibc as old as possible.

Currently the mips builders however are just using the stock mips toolchains from Ubuntu 16.04, and as you can see they’re not “very old”! We would of course always accept a PR to use a custom toolchain! Configuration for one mips builder can be found at https://github.com/rust-lang/rust/blob/master/src/ci/docker/dist-mips-linux/Dockerfile and the others are in the same docker folder.

[cuviper]

Unfortunately, there’s no maintained toolchain builder that can target glibc 2.5

You could still use CentOS binaries for glibc and kernel-headers, as I did for ppc64le here.

[eeeeeta]

How hard is it to get MUSL completely static rustc? Because that solves all sorts of problems, and is great for enterprise.