Summary
Currently, docs.rs builds documentation for 5 targets by default for each crate published to crates.io.
This (pre-)RFC proposes for docs.rs to only build x86_64-unknown-linux-gnu
by default.
Motivation
Most crates are the same on all targets, so there's no point in building them for every target.
Most crates do not compile successfully when cross-compiled. As a result, trying to cross-compile them wastes time and resources for little benefit. By reducing the number of targets built by default, we can speed up queue time for all crates and additionally reduce the resource cost of operating docs.rs.
For most crates, the documentation is the same on every platform, so there's no need to build it many times. Building 5 targets means that builds take 5 times as long to finish and that docs.rs stores 5 times as much documentation, increasing our fixed costs. If docs.rs only builds for one target, then
- queue times will go down for crate maintainers
- storage costs will down for the docs.rs team, and
- there will be fewer pages built that will never be looked at.
In particular, the very largest crates built by docs.rs are for embedded systems
(usually generated using svd2rust
). These crates can have several gigabytes
of documentation and thousands of HTML files, all of which are the same
on the current default targets.
For crates where the documentation is different, there's a simple way to opt-in
to more targets in Cargo.toml
:
[package.metadata.docs.rs]
targets = ["target1", "target2", ...]
Guide-level explanation
Target triples are used to determine the platform that code will run on. Some crates have different functionality on different platforms, and use conditional compilation.
docs.rs has an idea of a 'default target', which is the target shown on docs.rs/:crate/:name
.
docs.rs also builds other targets, which can be accessed by visiting docs.rs/:crate/:name/:target
,
and are linked in a 'Platform' dropdown on each documentation page.
Currently, docs.rs builds documentation for 5 tier 1 platforms by default:
x86_64-unknown-linux-gnu
i686-unknown-linux-gnu
x86_64-apple-darwin
i686-pc-windows-msvc
x86_64-pc-windows-msvc
After this change, docs.rs will only build one platform by default: x86_64-unknown-linux-gnu
.
This only changes the default, you can still opt-in to more or different targets if you choose
by adding targets = [ ... ]
to the docs.rs metadata in Cargo.toml
.
All existing documentation will be kept. No previous releases will be affected, only releases made after the RFC was merged.
For most users, this will have no impact. Some crates that target many different platforms will be affected, especially if their features are substantially different between platforms. However, it is easy to opt-in to the old behavior, and they will have plenty of advance notice before the change is made.
For example, winapi
chose to target all Windows platforms as follows:
targets = ["aarch64-pc-windows-msvc", "i686-pc-windows-msvc", "x86_64-pc-windows-msvc"]
Reference-level explanation
- If a crate published to crates.io has neither
targets
nordefault-target
configured in[package.metadata.docs.rs]
, it will be built for thex86_64-unknown-linux-gnu
target. - If it has
default-target
set but nottargets
, it will be built fordefault-target
but no other platform. - If it has
targets
set but notdefault-target
, it will be treated as if the first entry intargets
is the default target. - If both are set, all
targets
will be built anddefault-target
shown on/:crate/:version
. Targets are deduplicated, so specifying the same target twice has no effect. - If
targets
is set to an empty list,- If
default-target
is unset, it will be built forx86_64-unknown-linux-gnu
. - Otherwise, it will be built for
default-target
.
- If
Examples
- This crate will be built for
x86_64-unknown-linux-gnu
.
[package.metadata.docs.rs]
all-features = true
- These crates will be built for
x86_64-pc-windows-msvc
.
[package.metadata.docs.rs]
default-target = "x86_64-pc-windows-msvc"
[package.metadata.docs.rs]
targets = ["x86_64-pc-windows-msvc"]
- These crates will be built for all Windows platforms.
In this case the default target will be aarch64-pc-windows-msvc
.
targets = ["aarch64-pc-windows-msvc", "i686-pc-windows-msvc", "x86_64-pc-windows-msvc"]
In this case the default target will be x86_64-pc-windows-msvc
.
default-target = "x86_64-pc-windows-msvc"
targets = ["aarch64-pc-windows-msvc", "i686-pc-windows-msvc", "x86_64-pc-windows-msvc"]
- This crate will be built for
x86_64-apple-darwin
.
default-target = "x86_64-apple-darwin"
targets = []
Background: How many crates will be affected?
The answer to this question is not clear. docs.rs can measure the number of visits to
/:crate/:version/:platform
, but this only shows how many users are looking at pages built for different targets, not whether those pages are different. It also does not show pages which are different but have no visitors viewing the documentation.
With that said, here is a list of the visits to non-default platforms between 2020-06-29 and 2020-07-12. It does not include crates with fewer than 5 visits to non-default platforms.
The list was generated as follows:
Shell script
zgrep -h -o '"GET [^"]*" 200 ' /var/log/nginx/access.log* | grep -v winapi | cut -d / -f 2-4 | grep -v '^crate/' | grep -f ~/targets | cut -d / -f 1 | sort | uniq -c | awk '{ if ($1 >= 5) print $0 }' | sort -k 1 -h
Note that winapi is excluded since it explicitly gives its targets in Cargo.toml
.
Background: Why do crates fail to build when cross-compiled?
- The most common reason by far is that they have native C dependencies.
Most crates with C dependencies have a dependency on pkg-config
somewhere,
and pkg-config
does not allow cross compiling by default.
See for example the build failure for crates-index-diff 7.0.2
:
error: failed to run custom build command for `openssl-sys v0.9.57`
--- stdout
run pkg_config fail: "Cross compilation detected. Use PKG_CONFIG_ALLOW_CROSS=1 to override"
Here, pkg_config
fails because docs.rs is cross compiling from x86_64-unknown-linux-gnu
to i686-unknown-linux-gnu
,
and the entire build fails as a result.
- The second most common reason is that crates do feature detection in
build.rs
via the$TARGET
variable passed by cargo, and assume that the host and target are the same (e.g. pass options forlink.exe
tocc
). This is similar to the above case, but withoutpkg_config
involved.
In both these cases, it is possible to fix the crate so it will cross-compile, but it has to be done by the crate maintainers, not by the docs.rs maintainers. The maintainers are often not aware that their crate fails to build when cross-compiled, or are not interested in putting in the effort to make it succeed.
Background: How long are queue times currently?
docs.rs has a default execution time limit of 15 minutes. At time of writing, there are only two crates with exceptions to this limit: 20 and 25 minutes respectively.
Most of the time, when the docs.rs queue gets backed up it isn't because of a single crate clogging up the queue, but instead because many crates have been released at the same time. Some projects release over 300 crates at the same time, and building all of them can take several hours, delaying other crates in the meantime.
By building only a single target, we can reduce this delay significantly. Additionally, this will reduce delays overall significantly, even for projects that only publish a single crate at a time.
Drawbacks
- Crates that have different docs on different platforms will be missing those docs if the maintainer takes no action.
Rationale and alternatives
- We could keep the status quo, building all 5 targets for all crates. Doing so would avoid breaking the documentation for any crate, but keep queue times relatively high, and over time would increase our resource usage a great deal.
Prior art
- GoDoc has docs for only one platform (by default, Linux).
- Racket has docs for only a single platform but does not say which platform ('This is an installation-specific listing').
Unresolved questions
- How long should docs.rs wait before changing the default?
- How many crates will this affect?
Note that 'affect' is ambiguous here since a crate could have different
docs on different platforms but not intend to have differences. For example, rustdoc is non-deterministic and generates trait implementations in different orders depending on the platform:
futures::mpsc::Receiver
does not use#[cfg()]
, but has different traits orders betweenx86_64-unknown-linux-gnu
andi686-pc-windows-gnu
.
Future possibilities
- docs.rs plans to add build logs for all targets in the future (currently it only has logs for the default target). Building for one target would save generating many of the logs - we'd only generate them for targets that were explicitly requested.