The Rustdoc Redux

announcements
1

Hey all,

rustdoc is a wonderful tool for Rust developers, giving us the ability to write great docs for our code. In a recent survey of Rust developers, good docs were the #1 thing people looked for when choosing a dependency.

rustdoc has a long and venerable history in Rust; it was first added in 2011, making it almost six years old!

However, rustdoc is tough to contribute to, as it uses the compiler's guts to generate the docs. This means that, if you want to change one line of rustdoc's CSS, you must first bootstrap the entire compiler, including building LLVM. It's... not pleasant. For more on how Rustdoc works, see this great post by docs team member @QuietMisdreavus.

Because rustdoc is so hard to contribute to, it rarely gets worked on. People like @alexcrichton just before 1.0 and @GuillaumeGomez and @ollie27 today (among others) have done some heroic work making rustdoc better, but very few people enjoy working on rustdoc. As such, bugs pretty much don't get fixed:

A-diagnostics is the only tag that's got more bugs than A-rustdoc.

So, what to do? Over the last year, (usually after having one too many :beers:) I get mad at rustdoc and threaten to re-write it. I've even hacked together multiple prototypes. Now that the end of the book v2 work is in sight, I'm going to be doing more work on the tooling we have for docs. rustdoc is target number one, as it's the foundation of everything else.

After long discussions with members of the dev-tools and compiler teams, and some thoughts out how to improve rustdoc, I am happy to announce that I'm working on a replacement for rustdoc. It's a ground-up re-write. Normally I'd be hesitant to go for a re-write, but after long discussions, the fundamental architecture of how rustdoc works needs to be re-done, and so a refactoring would be tantamount to a re-write anyway.

Some goals for the rustdoc redux:

  • rustdoc should be easy to contribute to, and maintain generally
  • rustdoc should be easier to customize
  • long-standing bugs should be fixed
  • Be good for more than API docs, like guides

These high-level goals imply many other things. For example, making rustdoc easy to contribute to means "you should not need to build the compiler to build rustdoc."

Anyway, it's very early days for this work. Here's two screenshots from today:

701Ɨ527 21.6 KB

1366Ɨ406 18.4 KB

Rustdoc can currently get your crate docs, the list of top-level modules, and their docs. That's it. There's no CSS, even! However, the core of it is there: take Rust code, produce HTML, CSS, and JavaScript.

There's a lot of work left to do, and so, if you're interested, I'd love some help! This is a great way to get started working on a Rust project, even if you haven't done much Rust before. There's a lot going on: Rust code, JSON generation, working with web technologies, re-thinking the presentation, all kinds of stuff. Heck, you can work on the front-end without even installing Rust!

Since it's so early on, you have the chance to make significant contribution to a tool that every single Rust developer uses. I'd love to form a little team around rustdoc and make it amazing.

The repository is here:

Check out the issue tracker; I've made a number of issues already, but am going to be making more throughout today. Several of them will be "discussion" style issues to work up ideas, so if you're not interested in writing code, you may want to get involved in those discussions!

Thanks!

46 Likes
2

Congratulations to the announcement!

My personal wishlist for the rustdoc rewrite:

  • Creating docs even if there is a compiler error. This would probably also be needed by RLS.
  • Reconsidering to add KaTex support
  • Not regressing on performance. Right now it seems to be implemented in ember. When visiting crates.io (even with latest firefox nightly with webrender enabled, which is supposed to be and actually is blazingly fast) clicking on every link first greets me with a ā€œloadingā€ page and only then the actual page builds up. Sometimes stuff is cached, but most times I get the loading page. Iā€™m not a web dev, so maybe I just donā€™t recognize wanted behaviour here, but to me it feels like a bug. Its pretty annoying at the least. I understand that crates.io had to ship with the bug as it was new, but its different (I think?) with rustdoc, as there is time and already a mostly okay base. Not sure if the bug applies to rustdoc as well, but if it does, maybe we could wait with the switch to the new rustdoc until it has fixed the bug (maybe fastboot could do it)?
  • Not regressing on search-by-url. I have a firefox search keyword for searching in Rustā€™s std docs, would be cool if I could continue to use it.
  • Maybe also not breaking the URLs. To me personally its not a big issue, as I can manage to wiggle through, but I think it makes a bad image for rust if someone googles stuff and clicks an official link only to be greeted by a 404 page.
13 Likes
3

Please please please donā€™t make JavaScript a hard dependency of rustdoc (or if that happens I hope at least the old rustdoc will be available in some form). The existing output is great. It has never once lagged or crashed my browser. I dread the moment I see the doc.rust-lang.org tab eating hundreds of megabytes of RAM, spinning my CPU and crashing my browser, just like Discourse, Twitter, Slack, and innumerable other ā€œweb applicationsā€.

44 Likes
4

None of those things are inherent to JavaScript; rustdoc today actually already uses a lot of JavaScript. No JS at all would regress many significant features.

I expect it to be smaller and faster than today.

And, for people who extremely hate JavaScript of any kind, thatā€™s why pluggable frontends exist; you can totally do that if youā€™d like.

8 Likes
5

Okay, that's reassuring enough to me.

4 Likes
6

Yeah, but I use NoScript, and I'd like this to mostly work out of the box. I know search won't work without JS, but I can live with that. As an end user (and not contributor), I'd like to see this use the least amount of JS possible.

2 Likes
7

All of these things are either do-able or planned:

  • For KaTex, historically weā€™ve rejected it because it adds to much overhead to those who donā€™t use it, but I think we can do it via a flag.
  • Not regressing on performance is important to me; I expect this to speed it up.
  • ā€œsearch-by-urlā€ isnā€™t a term Iā€™m familiar with, I think I know what you mean here, but if youā€™ve got a link or something, thatā€™d be :+1:. If itā€™s what I think it is, no plans to break it.
  • Yeah, URL compatibility is important; right now, thereā€™s no active plans to change the URL structure.
4 Likes
8

Also note that when you see the ā€œLoadingā€ page on crates.io, Emberā€™s already fully loaded. The loading you see is the page contacting the crates.io API to load crate information.

3 Likes
9

For ā€œsearch-by-urlā€: basically, you can take a url stem and finish it with an arbitrary query to make a valid search. For example, right now you can search the stable std docs by taking the stem https://doc.rust-lang.org/std/?search= and appending whatever query you want on the end, like btreeset or the like. That will take you immediately to the search results without having to navigate to the page first. Firefox custom search providers and DuckDuckGoā€™s !bangs work like this.

6 Likes
10

Thatā€™s what I thought, I just hadnā€™t heard that term specifically used.

Yeah, keeping this is :100:

2 Likes
11

Frontend JavaScript is great, and improves the experience. I know youā€™re using EmberJS, but Iā€™m not familiar with Ember. Itā€™s in the folder labeled frontend, but the instructions on your GitHub repo specifically reference npm and some other things, which makes me wonder why it requires nodeā€¦ if it runs in the browser. All Iā€™m concerned about is whether rustup is going to require node to be installed on every computer that wants to do Rust development and view docs. That would be hard to accept.

If node is just required for developing on rustdoc itself, I donā€™t have even the slightest issue with that.

1 Like
12

Node is basically a build tool here; it's not even a requirement for developing rustdoc, it's a requirement only if you're working on the rustdoc frontend. If you only want to work on the backend, you don't need any of that installed, and if you're using rustdoc, there's no dynamic dependencies at all; the HTML/CSS/JS files are built into the binary.

I certainly don't want people to depend on node to be able to read docs.

6 Likes
13

Could you clarify what you mean by "frontend" and "backend" in this context? For a tool like this I would expect the "frontend" to be the code that extracts documentation from .rs files, and the "backend" to be the code responsible for emitting HTML, but that doesn't jibe with what you said.

14

ā€œFront-endā€ is typically what the user sees. (here: HTML/JS in your browser) back-end is stuff that shuffles around data. (Typically on the server. But in this case itā€™s just being handled by the rustdoc binary.)

2 Likes
15

By "frontend" I mean "the HTML, CSS, and JavaScript", and by "backend" I mean "The Rust code that grabs all of the doc comments out of your code, generates some JSON, and writes out all of the files."

1 Like
16

yay! im glad this is being worked on. iā€™ve known more than one person who have bounced off rust specifically because the std docs were overwhelming. I definitely think traits and trait-derived methods could be a lot better presented then they are now, for one thing.

2 Likes
17

If you have concrete thoughts on a better presentation, I am all ears! Please don't hesitate to open an issue.

1 Like
18

Please, donā€™t make it a tool that outputs HTML or any front-end code!

Make it a tool that outputs machine-readable documentation, so that others can build various documentation-related tools on top of it.

1 Like
19

Again, thatā€™s the default output. You can already request only JSON (oops not quite only JSON yet, but you do get it), which is machine readable and lets other people build other tools on top.

12 Likes
20

I tried running it on a crate of mine with a few dependencies and it took an enormous amount of time, and ended up panicking in the end. Why does the save analysis take so long?