The title basically says it. Let me give an illustrative example.
Example
I was recently fixing a bug in servo/pathfinder which I came across only because I was compiling the plotters crate with the current nightly compiler. This problem was not present in the nightly-2024-02-01 version (maybe even some later versions, I did not check every). Since the development is mainly done on stable, this bug would not have been noticed at all. But when submitting a new version to crates.io and trying to build it on docs.rs this process will fail since docs.rs uses a nightly compiler to build all crates.
In general, I can appreciate that some features such as the doc(cfg(...)) are only available on nightly and desirable for writing concise documentation.
A possible fix would be to support multiple compilers (stable, nightly) which can be set in the [package.metadata.docs.rs] section of Cargo.toml but I do not know how much effort it would be.
Considering all this, I do not intend to push for any immediate changes as I do not see too many actual problems but I find the choice a bit odd considering that safety and guarantees are a very high priority in this community in general (which I really appreciate btw).
TLDR
What is the reason to use the nightly compiler for docs.rs?
Flipping this the other way around, why does a crate build on stable but not on any nightly? I'm assuming its from nightly auto-detection which imo should not be used. It should be an explicit choice to enable features that could be coupled with specific nightly versions.
I think that this argument does not capture what I was getting at. The nightly version of rustc is nightly for a reason: it is not stable. But the documentation for docs.rs depends on this toolchain which can only offer limited guarantees in terms of stability. I also had multiple occasions of compiler bugs in the nightly compiler (it even said so explicitly when panicking).
This would mean that your crates' documentation build does depend on the most volatile version of your compiler toolchain and it could theoretically change specific behaviour from one day to another.
I am not sure that if a crate does not build repeatedly on nightly, that we can really assume that something is going on. It is certainly worrying but I would not make any generalized statement here. This might depend on the specific problem.
But I agree that with enabling nightly-specific features you pretty much enter this land of "unknown future changes" which might break your system. But at least at this point you made a concious choice.
Most of this was before my time, but from what I know the main reason to use nightly in docs.rs is that docs.rs is using some features in rustdoc that were specifically built for docs.rs, and only exist in nightly compilers.
Iām not sure if it would be possible to make it work without these features, it would definitely involve a big chunk of work, and would slow down docs and/or reduce functionality. Or it would involve stabilizing features that are only used by docs.rs.
Thanks for pointing this out. My initial question was mostly targeting if there is a rationale behind using nightly. I can understand that introducing all these options is much more tedious to maintain.
This has not really been my experience exclusively developing with nightly locally for years (outside buggy dependencies enabling unstable features automatically). The number of breaking bugs on nightly releases in stable features has been similar to the number of breaking bug fixes in stable releases. Any bugs that do creep in have been fixed normally within a couple of days.
I found that in CI the latest nightly breaks every few months. Not sure that it would affect doc generation, but it has been things like RISCV64 Linux suddenly being broken for a week, the MIPS build just disappearing etc (toolchain demoted to tier 3), bare metal targets breaking, etc.
On x86-64 Linux (and presumably other tier 1 targets) it is pretty stable, but not everything targets only that.
And "fixed within a couple of days" is kind of useless for docs.rs, since you then need to file a bug to get them to rerun your crate. Meanwhile your users can't actually check your API docs for your new version, since that is missing. They would have to build that locally.
It would make a lot more sense for docs.rs to use latest stable (and they could stabilise the features needed, or use the env variable trick to use nightly on stable).
For crates that use nightly features (rust-toolchain file) they could build those specific crates on nightly. But that is a vanishingly small number of crates these days (I would love to see some statistics on the percentage of crates that need nightly on crates.io, grouped by year of publication).
There's a few more crates offering optional API that uses nightly features, and they would want that API be documented, too. Speaking of documenting optional API, there are crates that use unstable rustdocfeatures to make those more readable, too.
Once the build capacity is expanded it should be possible to setup automated or user-triggered rebuilds. This would help with more than just issues with nightly compiler bugs, it'd also help with situations like broken dependencies that have since published a fixed version that I think are a more common failure cause.
One of the advantages of having docs.rs using nightly is that it triggers these sorts of failures. It acts in some way as a more continuous form of crater, with more of an incentive to diagnose and fix the issues it finds. One of the things I keep meaning to work on (or at least sketch out a plan for) is a way to surface these issues to crate maintainers earlier, at the moment the only way really is to go check the page a bit after you publish a version to see if it failed or not. It should be possible to have some indication that there are issues with your crates when you visit any docs page.
I strongly disagree with this statement. I my opinion crates should be tested before they are made publicly available under a new version. Publishing on docs.rs should (in my opinion) not act as additional CI. Otherwise I would like to schedule a new build whenever I push or get a PR. This is unrealistic in my opinion, simply from an infrastructure perspective as well.
Best case scenario: This never happenes. How can we avoid this? Make the build steps which are done by docs.rs really obvious for end users. And make them as close to the default build steps that the most basic tools such as cargo doc already implements. The bigger this discrepancy without knowledge about it is, the more crates will start to fail.
I'm talking about testing from the perspective of the Rust Project, not the crate that is being published. Testing against nightly is not something that I see that commonly done.
The main place this differs (and where I see a decent proportion of the build failures come from) is user-configured package.metadata.docs.rs, which there's now a tool to parse and include in a build to make it trivial to test in CI:
Ultimately if we were building docs.rs today, it's not obvious to me that we would use nightly for this -- I think it's more of a historical artifact. IMO it would be nice to move docs.rs to stable at some point, since this would presumably make the output better suited for the vast majority of users. I don't know how tight the integration between rustdoc and docs.rs is at this point, and whether development velocity for docs.rs would suffer significantly from waiting for stable releases of rustdoc features it needs.
It probably would be good to actually make a tracking issue for all the unstable features in use, a quick skim shows:
cargo features:
-Zrustdoc-map
to link to dependencies on docs.rs
-Zhost-config, -Ztarget-applies-to-host
to handle package.metadata.docs.rs.rustc-args when cross-compiling
-Zrustdoc-scrape-examples
somewhat as a form of stress-testing the feature
rustdoc features (all these flags require -Zunstable-options):
--emit, --static-root-path
to split out the different varieties of artifacts and not replicate all the static files per-crate
--resource-suffix
to avoid filename overlap issues when rebuilding a crate with a newer rustdoc version e.g. the search-index I think still uses this, most uses were replaced by --emit and --static-root-path
--extern-html-root-takes-precedence
to ensure #![doc(html_root_url = ...)] doesn't cause off-site links
--extern-html-root-url
used under the hoods by cargo -Zrustdoc-map
There's also explicit support for package.metadata.docs.rs.cargo-args containing -Zbuild-std to allow documenting for tier-3 targets.
This isn't really an issue, the unstable features in use change very rarely, and are mostly about improving on the interface rather than adding any new capability. It's more that some of these things that docs.rs needs to work have just been unstable for years, with no real motivation to get them stabilized when docs.rs is the only major user (some of them are likely useful for organizations building their own internal documentation sites, but I at least haven't heard anything about this usecase).
We do, occasionally, have docs.rs specific bugs. Mostly, it's because of the strange way it interfaces with rustdoc (patching HTML files to add its own nav bar, plus using CLI params that nobody else uses).
If docs.rs hits one today, the solution is to land a fix in rustdoc, wait about a day, then rebuild all affected crates. The quicker we can catch it and release a fix, the less we have to rebuild.
If docs.rs committed to using stable, how would a bug like that be fixed? A patch release of the compiler made specifically for docs.rs's benefit? A workaround that docs.rs has to carry for twelve weeks, followed by rebuilding twelve weeks' worth of crates before the workaround can be removed?