Pre-RFC: Split up Tier-2 platforms

tools and infrastructure
1

Summary

Split the current Tier 2 level of platform support into a new Tier 2 and Tier 3, moving the current Tier 3 down to Tier 4. The new Tier 3 would have the same definition as the current Tier 2. The new Tier 2 would have similar requirements as are in place for officially supported tools; it would require that CI be present for that platform and running tests, and releases are gated on CI passing or the platform being demoted to Tier 3, but not every single commit is gated.

Motivation

At the moment, Rust has Tier 1 level support for three operating systems, Window, macOS, and Linux, and two ISAs, the 32 and 64 bit x86 architectures; a total of 7 targets, with two different Windows toolchains (GNU and MSVC) and no 32-bit macOS build.

Tier 1 status in the Rust project means fully supported; there are builds of both the compiler and standard library available, and the Rust CI system runs a full build and test for the platform on every commit before it is merged to master.

Rust has Tier 2 support for 77 different targets, with 36 different ISAs, ~13 different kernels (including bare metal), and with several different toolchains (GNU, Android, MUSL, etc) for some of the kernels.

However, Tier 2 support has a much broader definition; it just means that some subset of the Rust project is guaranteed to compile for that platform, by having that a mandatory part of CI. That subset could be just core, std, or the full compiler. And it makes no guarantees about tests.

Tier 3 is then the catch-all for other ports present in the tree; there may be some code to support those platforms in Rust source, but there is no official CI setup for them and no guarantees made by the Rust project. This may be because the toolchain is proprietary, the platform is niche, or just no one has taken on the work to set up CI for them.

Tier 1 and Tier 3 seem to have fairly reasonable definitions, but Tier 2 can indicate a very widely varying level of support in practice. While building on a platform in Rust gives a greater sense of certainty than building in, say, C or C++, it's still no substitute for having the full test suite run. Downstream distributors have noticed this; the Debian packaging of Rust disables the test suite because (among other reasons) it builds and runs on a wider variety of ISAs, and the Debian maintainer couldn't keep up with getting the test suite to run reliably on platforms not supported by the core project; and additionally, Debian imposes self-hosting requirements on many platforms, while the Rust project Tier 2 definition only requires cross-compiling.

Additionally, since Rust 1.0, I don't believe that any target other than the -msvc targets have migrated to Tier 1 status. At the moment, there are RFCs for defining the target tier policy more formally, to write down the unwritten rules for when a target can be promoted or demoted, as well as one for promoting aarch64-unknown-linux-gnu to Tier 1.

However, there is a fairly big gulf between the bare minimum definition to be included in Tier 2 (there must be at least a working cross-compile of libcore to the target platform), and what is required to promote to Tier 1 (all of libstd working and testing).

Proposal

I would like to propose breaking Tier 2 up up into Tier 2 into Tiers 2 and 3. The new Tier 3 would have the same definition as the current Tier 2, and the new Tier 4 would be the same as Tier 3. The proposal would add a new Tier 2 which has stronger requirements than the current Tier 2, but weaker requirements than Tier 1.

The new Tier 2 under my proposal would have the following definition:

  • CI will build the full compiler (if applicable) and standard library for the platform
  • CI will guarantee that the build succeeds before merging to master, just like the current Tier 2 definition
  • CI will additionally do a self-hosted build (if the target is a and run the test suite on the platform, but failures will not block merging to master. Failures will be noted in a similar way too toolstate, to make it easier for users of that platform to avoid updating to broken versions.
  • Releases are guaranteed to include working builds for this platform; during the beta period, regressions introduced to the test status will be fixed, marked to ignore if not relevant, or the platform will be demoted to the new Tier 3 status (equivalent to the current Tier 2 status)

This new Tier 2 solves a couple of problems. It provides a better stepping stone towards Tier 1, as you can evaluate more easily how prompt the community is at fixing regressions introduced, including regressions affecting the test suite. It allows for working out the kinks in a platform's CI before committing to Tier 1 support. It also gives users and downstream distributors a better sense for how well supported a platform really is, as "does core build" is really a fairly low bar for a platform to pass.

Note that the new Tier 2 is not intended only as a stepping stone to Tier 1; some platforms may stay in Tier 2 indefinitely, if they are more niche, have slow or flaky build systems, or have other reasons why gating every build on tests passing would be excessively burdensome.

Alternatives

One possible alternative would to keep the three tiers, but add information to Tier 2 on self-hosting status and testing status; for instance, adding an extra column about testing and self-hosting status in CI. I think that there are a few reasons not to prefer this; for one, Tier 2 already has a lot of different platforms with varying levels of support (core, std, and the full compiler). For another, whether the test suite is passing and whether the compiler is self-hosted are different axes than which components are present, and what level of commitment there is to that status, so it makes some sense to present that information in a different way.

Another would be to add sub-tiers to Tier 2, such as Tier 2A, Tier 2B, etc, depending on different levels of support. That seems to be somewhat more confusing than just adding tiers. Adding a new tier only causes some confusion during the transition, which can be helped with good documentation, while sub-tiers add additional cognitive load as long as they exist. Additionally, I don't know that anyone relies on tiers in a machine readable way, it's just a way of documenting both internally and externally the level of support expected for a given platform, so a transition to an updated numbering shouldn't be too difficult.

Another alternative would be to use this definition for Tier 2, and just demote everything which doesn't meet this definition to Tier 3. That would mean that Tier 3 might have a mix of platforms that are built and available via rustup, and ones which are not, but Tier 2 would now only have platforms which are guaranteed to work as they are passing the test suite and (if applicable) self-hosting.

A third alternative would be to use this definition to replace the definition of Tier 1. At the moment, there's a fairly high bar to enter Tier 1, because having every change gated by the platform compiling and passing the full test suite is a high bar, and any issues with flaky CI hosts, flaky tests on certain platforms, slow build and test times, or difficulties in contributors being able to get access to the platform to debug and test fixes, means that it's worthwhile being very conservative before committing to Tier 1. However, if the Tier 1 definition were loosened a bit, along with a subset of platforms (such as the current Tier 1) being designated as dev platforms which are guaranteed to have every build pass the test suite, would allow for downstream users of stable releases to trust that the platform has been stabilized during the beta period, while not introducing too much additional burden on development.

The original pre-RFC suggested that the new Tier 2 definition should include that the platform is a host platform with a self-hosted compiler, but it was pointed out that not even the current Tier 1 definition includes that requirement. The proposal has been updated so that hosting is now orthogonal, so the new proposal is just about self-hosting if applicable, and running the test suite.

Discussion

Does this, or something similar, sound like reasonable fodder for an RFC? If so, I can edit the above into a more RFC-like format and submit it.

23 Likes
2

My only concern is that there might be technical factors that make it difficult to achieve some aspect of the proposed definition.

From what I remember, the original definition of Tier 1 was "what does our CI host have native build machines for?" and Tier 2 was everything else that gets testing.

3

Rust used different CI hosts for different platforms for a while; Travis for macOS and Linux, and AppVeyor for Windows. I think it's only relatively recent that it's switched to a single CI host (Azure, then GitHub actions).

The current CI system, GitHub Actions, allows for self-hosted instances, which is how the aarch64-unknown-linux-gnu build works, with is in FCP for being promoted to Tier-1. So it's clearly possible to have builders which aren't supported by the CI host of choice at the moment.

And actually, that is exactly part of the motivation for this proposal, which I forgot to mention earlier. Right now, the only hosts which have self-hosted builds and tests run are the Tier-1 platforms, which have a high degree of commitment from the compiler, infra, and release teams, and those which are expected to imminently become Tier-1, like aarch64-unknown-linux-gnu. But that means that there's a pretty high barrier to getting hooked into the CI system in any kind of meaningful way. For aarch64-unknown-linux-gnu, this meant Arm had to donate hardware for the infra, and dedicate engineering funding for maintaining the platform in Rust.

By separating out a new tier-2 which has CI run, but merges aren't gated on CI, it gives a place to get this infrastructure in place and ensure it's reliable before making the push towards the fairly large commitment which is Tier-1. For instance, if someone in the community were to set up a MIPS or PowerPC or RISC-V CI instance, and maintain it, those platforms may be able to be designated as the new Tier-2 without fully committing to Tier-1 status, so enjoy the benefits of CI being run regularly and a dedicated period to fix up regressions during the beta without blocking normal dev work if their infra or tests aren't yet 100% reliable, or aren't fast enough to keep the Bors queue moving.

An example that will likely be coming up soon is aarch64-apple-darwin. That's a relatively new target, but which there will probably be rapid pressure to get it to tier 1. By having it in a tier 2 where CI and tests are run for a couple of releases, it will make it easier to work out any kinks before promoting it.

1 Like
4

I wasn't saying anything against getting more builders... just that there may be non-obvious difficulties in implementing that exact definition for a new Tier 2.

1 Like
5

I wonder if Tier-2 also needs to be split sideways to better denote the support for "target-only" targets like thumbv6m-none-eabi. These will never get the ability to build rustc, and are unlikely to get a (full) std; but are in my experience very high quality and well supported, so the parts that are possible to port seem to meet the requirements of the new Tier-2.

15 Likes
6

I'd like to +1 the general sentiment of this thread (that tier 2 has too broad a definition) and +1 Nemo157s comments about breaking up tier 2 vertically as well.

There are a ton of platforms that won't ever reasonably be able to build rustc and its a shame we have no way of labeling those targets as highly stable and reliable due to the simple fact that they lack the resources to build a full compiler. We need a way to differentiate targets that are meant to fully host rustc and those that are merely targets for cross compilation and will always be.

This is also relevant to AArch64 Windows which may not have the ability to run full CI in the near future given that there are no cloud providers which provide AArch64 Windows hosted solutions and AArch64 Windows is currently only used on consumer grade hardware (i.e., laptops) which may be too slow to run Rust's CI at a reasonable speed. Having the ability to cross compile AArch64 Windows tests on an x86-64 host and then run those tests on lower powered hardware would be a huge boost to the confidence of that target as cross compilation target even if we can't guarantee its validity as a host target.

9 Likes
7

When tier 2 is split into tiers, I think it makes sense to rename them to prevent confusion. For example, they could be called A/B/C/D, or Alpha/Beta/Gamma/Delta. Or we could replace the word "tier" with a new term, to make it clear that the definition has changed.

4 Likes
8

...or Tier 2 could be split into one of

  • 2A, 2B, 2C
  • 2-Cross, 2-Test, 2-Candidate, ...
  • ...
3 Likes
9

...or 2-Left, 2-Center, 2-Right if you want something that's like A/B/C but with even less built-in connotation for one end of the sequence being superior to the other.

10

I don't think anything in the Tier 1 definition requires that the platform supports a host compiler. It only means that tests are run and required to pass, it doesn't say which tests. I think it is perfectly within the current definition to have a Tier 1 cross-only target.

Also, some existing tier 2 targets already have some tests running and gated on passing. There's nothing that says that can't be done, and it is generally done on a basis of what is desired and what can be afforded (CI resources, CI time, and available developers). But that doesn't mean they are ready to have a public commitment as to their quality.

I'm not sure I see how much value it is to have a "temporary" tier to step towards tier 1. CI gating can be added to any tier 2 target, and moving it to tier 1 is just a documentation change. If there is serious intent to move to tier 1 (like there is behind aarch64-unknown-linux-gnu), then it can jump to tier 1 as soon as it is ready. If there isn't sufficiently serious backing, then it can stay in tier 2.

It might also be helpful if you mention which specific targets you are interested in having tier 1 support for. Some are unlikely to have much burden (if they have a lot of overlap with existing targets), and some will be much harder to support.

11

I think moving powerpc64le-unknown-linux-gnu to tier 1 might be a good idea, IIRC the Rust project already has CI infrastructure for it.

12

The Rust components history website groups the targets in 5 categories:

  • Tier 1
  • Tier 2
  • Tier 25
  • Tier 3
  • Unknown tier targets

Could someone please explain what the "Tier 25" is and why the tier of some targets is unknown?

13

Tier 2.5 wasn't ever really a thing (an "accident"). It was for targets that were built, but not available via rustup. It has been removed. The components history website is just out of date. It should be updated, preferably being synced automatically with https://doc.rust-lang.org/nightly/rustc/platform-support.html.

14

From the feedback, it sounds like people would prefer to keep "is a host platform" and "is tested with test suite passing" as separate axes; and I'm fine with that. I will update my proposal with that as the main proposal, and my current proposal as an alternative.

I might have been misled by the fact that all current Tier 1 platforms, plus the FCP aarch64-unknown-linux-gnu platform, are host platforms, plus the Tier 1 definition says "Official binary releases are provided for the platform," but I suppose that can mean binary release of just libstd and not the compiler itself.

Which platforms, besides aarch64-unknown-linux-gnu which is expected to be promoted to Tier-1 soon? Digging through the CI config and logs can be a bit complex to figure this out.

That's one of the main points of this proposal; to better document which platforms are being tested regularly as part of CI, and what the level of commitment is to keeping those tests passing.

This proposal isn't just for a temporary stepping stone tier. Some platforms may stay in this "Tier 2" indefinitely. The value as a stepping stone to Tier 1 is an additional value, but not the primary value of the proposal.

It is a big commitment to put something in Tier 1. The bors queue is very frequently a limiting factor in dev speed and turnaround time for the Rust compiler and standard library, and there is a lot of effort spent in asking for retries after spurious failures or managing rollups. Each new platform that is added as a gating factor is an additional source of spurious failures, which could slow down turnaround time, or real failures which could require investigation and fixing.

An example that might be long-term Tier 2 under this proposal could be powerpc64le-unknown-linux-gnu. There are systems available for open source projects to use for CI, but it is a somewhat niche platform and we may not want to rely on them being available as a gating factor for bors.

The thing is, I don't necessarily want Tier 1 support for all of these platforms. I want something like what I described as a new Tier 2; I want documentation of which platforms are passing tests, and the ability to use rustup to only update to versions which are passing tests just like with toolstate, and a commitment to either fix test failures during the beta stabilization period or explicitly note the change in status by demoting the platform to a lower tier if it is deemed to be too much of a burden to maintain.

In the immediate future, the Tier 1 candidate that comes to mind is aarch64-apple-darwin. This is being worked on right now, and it is currently Tier 2, with a cross-compiled standard but not yet as a host. It's possible that this platform could benefit from my new Tier 2 proposal before being promoted, or it's possible it's worth promoting to Tier 1 immediately.

But others that I would like to be supported at the level of the new Tier 2 that I propose include all of the official Debian ports, as I frequently write software which uses a Debian based build system and packaging, so having a well supported Debian packaging of rustc is quite helpful. It is fairly common for new versions of rustc to be held up in Debian because of build failures on some of Rust's Tier 2 platforms, and the Debian package has had to disable running the test suite because of too much maintenance burden keeping the tests running on those platforms when the Rust project itself doesn't commit to tests passing on those platforms. Note that there are other challenges which also affect Debian packaging, like LLVM versions, but that's a separate issue, and I think it's worth addressing the architecture support independently.

While I'm not a *BSD user, I could imaging that they would be better served by Tier 2 status under the newer definition.

It would also be good to have Tier 2 status for common embedded platforms, like Android and iOS, and the more popular bare metal embedded Arm targets, like thumbv6m-none-eabi, thumbv7m-none-eabi, thumbv7em-none-eabi, thumbv7em-none-eabihf. However, it may take some more work on the test infrastructure before these could be realistically tested. These are platforms for which it is unlikely that they will ever be Tier 1, because of the additional challenges of testing and debugging on them, but having Tier 2 status under the proposal would be useful.

1 Like
15

I've updated the proposal based on several people's feedback that they'd prefer to keep whether a platform can be a host orthogonal from the level of support and testing, and I've added an alternatives section to discuss some of the alternatives which have been discussed, as well as a few that I have thought of.

I included this in my newly added "Alternatives" section. I think that this is likely to be more confusing long-term than just adding a new tier, updating the definitions of the existing tiers, or including these distinctions as named columns in the support matrix within a tier.

Oh dear, that sounds very confusing. I think if you are going to have any sub-tiers, you just name them based on what they mean; if you were going to do this, I would just add columns to the support matrix within the tier that are clearly labelled, like "Tested" and "Self-Hosting".

But anyhow, I think I would still prefer to keep one single linear Tier structure based on level of CI build/test coverage and commitment by the teams to maintaining that without regressions, and have the other information like what components are built be included in a matrix or just named sections within the tier.

In the alternatives, I've listed a few different ways to do this; while my slight preference is for my current proposal, of making this new definition the new Tier 2 and having Tiers 3 and 4 for the current Tiers 2 and 3, it's also possible that we could just tighten up the definition of Tier 2 to include CI with tests which must be passing for releases, and then demote a bunch of things to Tier 3, or loosen the definition of Tier 1 to this new definition, as most people outside of rustc developers themselves and people depending on nightly features don't really care that much about what works on every commit, they only care about what works on the released compilers.

In summary, the main thing I want is a Tier which is similar in level of support as the tools like clippy and rustfmt; CI including tests is run for them, but doesn't necessarily block merges, it just notes the status for informational purposes and rustup to know about, while allowing for time to fix up any regressions later in the dev cycle or during the beta stabilization period.

16

I don't think it'd be any more confusing than using A/B/C/D in a situation where 2A/2B/2C/2D are not ordered outline list entries equivalent to 2.1., 2.2., 2.3., and 2.4.

17

Actually, my idea was to rename Tier 1 to Tier A, Tier 2 -> Tier B/C and Tier 3 -> Tier D. Sorry for not making this clear.

18

Ahh. I think we do want some level of "good, better, best" ordering, even if the "better" part is subdivided horizontally.

19

The purpose I see for tiers is to be able to make long term decisions on whether to use Rust or not in a project, as they indicate how relaible the toolchain is expected to be for the targets the project is supposed to run on. With that in mind, having a tier that's expected to be in flux is not really useful, as nobody would make decisions on the commitment that the target is now tested but it would be demoted as soon as there is nobody to fix the tests.

I'm curious what your use for tiers is though :slight_smile:

Also, the experimentation before promoting a target to Tier 1 could be done (and it's currently done for AArch64 Linux) on Tier 2 without any problem: doing more QA than we promised doesn't break the guarantees of a tier.

I don't think adding more stuff to toolstate (or a similar system) is the right choice.

Toolstate is a hack implemented to work around the limitations of using submodules to bring the tools in the monorepo. When a change is made to the internals of the compiler the tools needs to be updated, but to update the tool we need a working rustc build with the change, creating a chicken-and-egg problem. Toolstate "fixes" the problem by allowing tools to be broken in the brief period of time between the change and the tool being updated.

Unfortunately toolstate also brings problems to users and rustc developers alike:

  • Users need to learn that nightly components sometimes "disappear", and they get errors when they try to add a missing component to an existing installation. While a tool they use is being fixed they also lack upgrades and potential compiler bugfixes.
  • Rustc developers can't land some kinds changes in the "toolstate week" right before beta branches out, to ensure all components are always available in the beta branch.

Given all these problems I think the best solution is to get rid of toolstate, instead of adding more things to it. There already efforts to do so (1, 2) by switching from submodules to subtrees, and clippy is already migrated preventing it from ever being broken (it's not tracked by toolstate anymore). Unfortunately we're waiting on a git bug to be fixed before we can switch all the other tools to subtrees.

3 Likes
20
  • wasm32-unknown-unknown runs a subset of tests
  • wasm32-unknown-emscripten runs most of the testsuite
  • nvptx64-nvidia-cuda runs a small number of tests
  • i586-unknown-linux-gnu,i686-unknown-linux-musl aarch64-unknown-linux-gnu arm-unknown-linux-gnueabihf arm-linux-androideabi x86_64-unknown-linux-musl runs the full test suite
  • thumbv6m-none-eabi, thumbv7m-none-eabi, thumbv7em-none-eabi, thumbv7em-none-eabihf run a very small set of tests

I don't think it should necessarily be documented (I am even reluctant to make the list above since it may imply things it shouldn't). What is documented is a public commitment. What is tested, and what is committed are two separate things.

I would also echo Peitro's post. Toolstate is a hack to work around the way our git repositories work, and we would like to avoid it.

1 Like