Why use nightly for docs.rs?

[jpleyer]

@notriddle

If docs.rs committed to using stable, how would a bug like that be fixed? A patch release of the compiler made specifically for docs.rs's benefit? A workaround that docs.rs has to carry for twelve weeks, followed by rebuilding twelve weeks' worth of crates before the workaround can be removed?

Do not use upstream version of docs.rs for the website. Release versions. If these versions are faulty, then revert to an old working one. Design a test suite (eg. test very common crates such as the std library, maybe serde, tokio, etc.) or define a rigorous standard which people can rely on. This is of course a lot of work. But is there anything that in principle would make such an attempt impossible?

[Vorpal]

@Nemo157

Testing against nightly is not something that I see that commonly done.

I definitely do that, especially since as far as I know crater doesn't test on all architectures (or some of the build failures I have run across with nightly would have been caught, RISCV seems especially prone to breaking).

[notriddle]

@jpleyer

Design a test suite (eg. test very common crates such as the std library, maybe serde, tokio, etc.) or define a rigorous standard which people can rely on. This is of course a lot of work. But is there anything that in principle would make such an attempt impossible?

The thing that makes stable stable isn't that it has a test suite: the nightly build of Rust already passes all of the automated tests. Docs.rs has a test suite, too, but it's pretty new and not nearly as thorough as the compiler's.

The secret behind stable's stability is that it's downstream from manual, community-driven testing by everyone who wants nightly-only features. During the beta period, low-risk high-reward fixes for bugs (that are usually found by nightly users; very few people run beta) are backported onto the beta branch that eventually becomes stable. If it weren't for this manual testing, you'd be better off just living at HEAD.

There's also a Crater build run on the beta branch immediately after forking it, but since Rust devs strategically run Crater builds on suspicious PRs before merging them into nightly, it usually doesn't find much.

@jpleyer

This is of course a lot of work.

Docs.rs is maintained by a team of five people, and it's not a full-time job for any of them.

• The docs.rs team doesn't have time to perform comprehensive, manual testing themselves. Even the much more numerous compiler team doesn't do that.
• They aren't going to run two publicly-advertised instances of docs.rs side-by-side, either. It would be too much work, and would also be bad for SEO.
• Self-hosting a nightly build of docs.rs is a lot more work than downloading and running a nightly build of the compiler. The version that docs.rs publicly runs is going to be the version that almost everybody uses.
• Very few projects self-host their rustdocs, either, so if docs.rs doesn't support nightly rustdoc features, then those features probably won't get any community feedback before they're stabilized.

[jpleyer]

@notriddle

The secret behind stable's stability is that it's downstream from manual, community-driven testing by everyone who wants nightly-only features. During the beta period, low-risk high-reward fixes for bugs (that are usually found by nightly users; very few people run beta) are backported onto the beta branch that eventually becomes stable. If it weren't for this manual testing, you'd be better off just living at HEAD.

That is totally true and not at all in conflict with my previous statement. In a dream world: You can still use nightly for your everyday compiling with all the new features, but why require docs.rs to use it as a default? If you want nightly, go for it it would be great to have a option in docs.rs to switch from the default stable to nightly. Then it is either all stable or all nightly (for both local builds and docs).

@notriddle

Docs.rs is maintained by a team of five people, and it's not a full-time job for any of them.

I know that this is very much to ask. And my question was particularly questioning if there is any reason other than that. That does not mean it is not important. It is maybe one of the most important aspects and I highly appreciate all the work that has gone into docs.rs so far.

[CAD97]

@jpleyer

a option in docs.rs to switch from the default stable to nightly. Then it is either all stable or all nightly (for both local builds and docs).

Maybe subtle detail: docsrs is very far from all one version. The actual rustdoc content is statically generated once and used forever, so the actual doc content you get from docsrs is a pseudorandom hodgepodge of different rustdoc versions. If you visit an old enough crate, it won't even have theming support and surprise you're looking at light mode docs now.

There's nothing particularly special about rustdoc stable versus rustdoc nightly, other than if you're permitted to use gated features. It would be interesting to survey how often a beta backport impacts rustdoc — if it's rarely enough, there'll be roughly no difference other than the lag in seeing improvements.

A better way to track spurious failures/regressions and queue rebuilds for when things get fixed would be massively helpful, yes. Docsrs built today might not use nightly, sure. But the cost of using unstable isn't all that much; it's not particularly unstable if you aren't using nightly features.

[jpleyer]

@CAD97

Maybe subtle detail: docsrs is very far from all one version. The actual rustdoc content is statically generated once and used forever, so the actual doc content you get from docsrs is a pseudorandom hodgepodge of different rustdoc versions. If you visit an old enough crate, it won't even have theming support and surprise you're looking at light mode docs now.

I did not think about this to be honest. It is an interesting detail for sure.

@CAD97

A better way to track spurious failures/regressions and queue rebuilds for when things get fixed would be massively helpful, yes. Docsrs built today might not use nightly, sure. But the cost of using unstable isn't all that much; it's not particularly unstable if you aren't using nightly features.

I agree.

[syphar]

@CAD97

@jpleyer

a option in docs.rs to switch from the default stable to nightly. Then it is either all stable or all nightly (for both local builds and docs).

A better way to track spurious failures/regressions and queue rebuilds for when things get fixed would be massively helpful, yes. Docsrs built today might not use nightly, sure. But the cost of using unstable isn't all that much; it's not particularly unstable if you aren't using nightly features.

I would like to pick this thought up.

We are planning to support manual & automatic rebuilds, where (IMO) both are mostly blocked by work that needs to be done, and partially by needed build server capacity.

How do you imagine abetter way to track spurious failures/regressions ? Do you mean for crate authors (that would be more this issue), or docs.rs alerting when error rates change after a new nightly version? IMO the latter would only be helpful for very widespread issues, and there are quite rare.

[CAD97]

Mostly I'm thinking about somehow notifying crate authors when docrs builds fall, and maybe the ability to queue rebuilds for when an issue is closed (and the fix deployed). Trying to automate that detection is likely more work than benefit it could create; the limit would be if a short term issue fix would requeue failed builds from the live period.

And less "this hasn't been discussed" but "agree with prior statement in the thread that it's desirable."

[djc]

We published rustls 0.23 last week and it failed to build on docs.rs due to some Cargo features shenanigans. We only found out after an issue was filed. Sending out a notification to the crates.io owners would be a great improvement!

[jpleyer]

How do you go about fixing this problem then? Do you simply bump the version of the crate and release new versions until docs.rs builds fine? Because I had a similar problem but did not want to make a minor version bump since my crate was still very young and so I just stuck with it for some time. Considering that the crate is already published to crates.io, there seems no other good option to me but I am curious.

[bjorn3]

Looks like they did a patch release with a couple of other fixes: Release 0.23.1 · rustls/rustls · GitHub

[syphar]

@CAD97

Mostly I'm thinking about somehow notifying crate authors when docrs builds fall, and maybe the ability to queue rebuilds for when an issue is closed (and the fix deployed). Trying to automate that detection is likely more work than benefit it could create; the limit would be if a short term issue fix would requeue failed builds from the live period.

I split off queueing manual rebuilds into a new issue so it's clearly separated from automated rebuilds.

[fintelia]

One historical element that I don't see mentioned in this thread is that when docs.rs was created, it was way more common for people to be developing on nightly Rust. This was shortly after 1.0, so there was a stable release channel available, but many features were missing. My recollection is that many crate themselves only supported nightly. And if you want to generate documentation for a crate that only compiles on nightly, you have to use the nightly compiler...

[jdahlstrom]

And it’s still seen in some rustdoc choices/conventions like showing all nightly APIs even in the stable docs – and you can’t even hide them! Probably my main peeve with the std docs.