So to be clear, I do believe we did follow the process. I promised @aturon i'd explain more, might as well do it now
https://github.com/rust-lang/rfcs/blob/master/text/1122-language-semver.md and https://github.com/rust-lang/rfcs/blob/master/text/1105-api-evolution.md are the two RFCs relevant to Rust following SemVer. While Rustdoc isn't the language, the API evolution RFC doesn't really have anything applicable to this situation. As such, I was going by the language one.
We never made a guarantee about what form of Markdown we were using. The RFC has a section for underspecified semantics:
There are a number of areas where the precise language semantics are currently somewhat underspecified. Over time, we expect to be fully defining the semantics of all of these areas. This may cause some existing code -- and in particular existing unsafe code -- to break or become invalid. Changes of this nature should be treated as soundness changes, meaning that we should attempt to mitigate the impact and ease the transition wherever possible.
Emphasis mine. So, soundness changes... here
When compiler or type-system bugs are encountered in the language itself (as opposed to in a library), clearly they ought to be fixed. However, it is important to fix them in such a way as to minimize the impact on the ecosystem.
The first step then is to evaluate the impact of the fix on the crates found in the crates.io website (using e.g. the crater tool). If impact is found to be "small" (which this RFC does not attempt to precisely define), then the fix can simply be landed. As today, the commit message of any breaking change should include the term [breaking-change] along with a description of how to resolve the problem, which helps those people who are affected to migrate their code. A description of the problem should also appear in the relevant subteam report.
Subteam reports don't exist any more, but, when looking at doing this, we said "Crater can't evaluate this change. So what do we do?" The plan was, do the work, and then see what happens with the standard documentation distributed with Rust. This isn't always representative, but if we found major issues, we'd know about them. So, we did the work, and then tried building
libstd's docs. The vast, vast majority of the diff was changing
" in HTML attributes, in other words, something that a better normalizer would find. This was also the only change to get the tests to pass, we had this hardcoded in the rustdoc tests.
We knew there would be other small differences, but only in rendering, which is very different than causing people's builds to break. Also, this is what nightly is for; we can always back stuff out later if it turns out there was massive breakage. But we would need to rely on people looking at things, since again, crater can't do anything at all here. So we landed it, and advertised it heavily, to little reception. Nobody replied on Reddit or Twitter, there were a few comments about timelines in the users' thread.
A few days later, some people ran into the test failure issue. This was a very obscure bug, because both versions render things correctly, it's all about what's considered a test or not. Furthermore, this breakage would be limited to the crates themselves, and not their user's. We got a total of five reports about this in total, all with trivial fixes.
Now that we had information that we could actually detect changes at all, we can take a different strategy. Everyone is on board with not breaking people's code, so we wanted to get in a cargobomb run, which could actually give us an idea of the impact here. The problem with doing so is that basically only @brson can get cargobomb going; I tried to get it working but could not. And now is a super busy time for him.
So, eventually, he got one going, and was able to give reports for 1/3rd of the total. At this point, there were ~30 total failures, not all of them rustdoc related. That's still non-zero, and possibly more than we were comfortable with, and after two weeks of arguing about this, we looked to the RFC again:
In cases where the impact seems larger, any effort to ease the transition is sure to be welcome. The following are suggestions for possible steps we could take (not all of which will be applicable to all scenarios):
- Identify important crates (such as those with many dependants) and work with the crate author to correct the code as quickly as possible, ideally before the fix even lands.
- Work hard to ensure that the error message identifies the problem clearly and suggests the appropriate solution.
- If we develop a rustfix tool, in some cases we may be able to extend that tool to perform the fix automatically.
- Provide an annotation that allows for a scoped "opt out" of the newer rules, as described below. While the change is still breaking, this at least makes it easy for crates to update and get back to compiling status quickly.
- Begin with a deprecation or other warning before issuing a hard error. In extreme cases, it might be nice to begin by issuing a deprecation warning for the unsound behavior, and only make the behavior a hard error after the deprecation has had time to circulate. This gives people more time to update their crates. However, this option may frequently not be available, because the source of a compilation error is often hard to pin down with precision.
Seems good; we had already been doing the first one, sending in PRs or explaining what was broken, and all of the previous breakage would be fixed. The extra partial Cargobomb people would take longer though, so cases 3 and 4 come into play: in this case, a flag that would display a warning, but not fail people's builds. This was quickly implemented, and aimed at removing some time later after we could help fix the stuff that had come up.
At this point was when this was deemed "not enough" and things came to a head.
So yes, I still think this process was followed overall, and is still being followed, just like any other change. Even without reverting everything, nobody's stuff will break. The warning will let people see this change even if they aren't closely following Rust development. Fixing any issues will cause stuff to work properly both before and after, so transitioning is easier and doesn't need to be coordinated with Rust versions. It's all business as usual as far as I'm concerned.
.... I'll just leave it at that for now.