TLDR: writing good lib docs are tricky, especially with the current doc tools. I try to provide a little overview of the current situation and present a mockup of what the main page might look like if rustdoc was expanded to include better structure presentation
According to this other thread, documenting libs well is problematic. For one, the libs docs just tend to be a compilation of what makes the library convenient to use but this is problematic to explain. For one, the docs don't provide any structure outside of the inherent structure of the library (without delegating to say dummy modules for the purpose of exposition).
Even a decently documented lib such as regex has quite a detailed front page with a lot of detail but it is not easily consumed without just skimming the entire document. Another similar example would be std::collections. In contrast, serde does not have an extensive front page and so users are left to just browse at random trying to figure the library out on their own.
Furthermore, a library like regex might benefit from having separate categories for things like character class documentation and grouping annotation as opposed to matching examples and introduction text. Even if the library authors wanted to separate out these documents into separate docs, they'd be less likely to setup 3 different sites for doc hosting on top of setting up working doc systems that suits their needs. Rustdoc handling all these aspects would certainly be ideal.
As is, the left sidebar really isn't that helpful but I didn't discard it (it'd probably/possibly still be inside "libs"). The main idea is have section categories in the sidebar which stay with you as you read (like mozilla does with right sidebar here). You should always be able to see the categories but the TOC could possibly scroll I suppose. I also tried a mockup with tabs at the top but I don't like giving up vertical space to a floating top bar.
I'm aware this would probably require some rustdoc convention of having a "doc" folder or module for structure which wouldn't be well specified yet. I imagine it would also require some experimentation and testing to see how much, the regex crate for example, would benefit from something like this. Either way, I thought having a mockup might be helpful at least to have a concept of what it might look like.
Also, I have this picture as an *.svg from inkscape. I can upload if someone else wants to play with it.
I think this is an important point to focus on. I very very much like having the structure of the docs tied to the structure of the library, because it encourages API authors to come up with something that is easy to follow not just in terms of prose, but in terms of the public types and functions. I don't mean to state this as a firm rule that I think we should always adhere to, but I do think it is a good heuristic to follow.
With that said, I do like the idea of a ToC, and putting it in the sidebar seems like a pretty decent idea. I'm not as much a fan of having docs span multiple pages. Going back to my first point, if you need multiple pages, then perhaps your module has become too complex or too big---which might suggest splitting it into sub-modules, each with their own docs.
For a crate like regex, I very much like having one page for module docs that covers module wide usage, and then more detailed docs on the Regex type itself. I do agree that having a ToC would improve things.
Could you expand on this a bit? My hope is that the ToC would be generated from the current module docs.
By the way, I'm not exactly proposing this be done. I was actually just trying to explore to see if a reasonable interface could be found which would make it work without being horrible. I couldn't find any libs which actually try to combine lib docs with usage docs so it's the wild west I guess.
If the interface was decent, then it could be decided whether the result was actually worth it.
Well, suppose having docs span multiple pages was a good idea (I'm not sure if it is or not but for the time being, let's assume it is; this would probably be more important if examples started getting bundled with the libs). Then, rather than having an epic lib.rs file which has some scheme for enabling one file to be separated into a tree structure, you could have say, this:
Maybe it would generate the aforementioned picture where the intro was generated from the readme. Maybe the doc files would be *.md instead. Anyway, this would allow more complicated doc structure shoehorned into the regular format of the libs. The lib structure would be normal. The rest would follow the structure from the doc folder constituents.
Also, this isn't to say an epic lib.rs couldn't be used. But you could choose not to use it if you desired.
Supposed multi-page docs are bad. Then getting a TOC out of the libs could probably be generated in a straightforward fashion from the libs themselves as you suggest. This would probably be a good idea regardless.
Regarding libs, perhaps some things could be done to make them more approachable to newcomers. For example, in the regex crate, of the listed types on the front page, almost every single type is only used indirectly (only 3 out of 14 are useable directly). Constantly finding things you can't use directly makes browsing really difficult (maybe indirect types should only be shown by default in return signatures?).
There is also the problem that some projects have a bit of documentation in their Readme file and different bits in the crate files (and many just duplicate the Readme into the lib.rs docs). One solution for this is to introduce an attribute like #![include_doc("../Readme.md")]. Other would be to link to arbitrary Markdown pages, which rustdoc can already render.
Now this is something I find very enticing, and I've been frustrated by it myself. I absolutely agree that it would be much clearer if there were a way to highlight important or central types in the module. I mostly find the current breakdown between enum/struct kind of arbitrary for the purposes of conveying API documentation. For example, in many cases, whether a type is an enum or a struct is an implementation detail and not something that is exposed to consumers (for example, regex::Regex is an enum, but no user should give a hoot about that).
The question is... how to do it? One of the nice properties of today's docs is that they are manifest from the module structure itself. I don't think I would want to lose that property (which is also kind of why I'm not a fan of the multi-page approach). But I find it hard to reconcile that with highlighting important types in some way.
I think the issue goes deeper than this. Rustdoc's output is automatic, very easy to use, and mostly quite sensible. As a comprehensive API reference, it is almost perfect — if I'm caught up in using regex and need to know exactly what I can do with the result of this method, all the information is there. However, there is more to documenting a library than simply listing all types and modules, with an optional text for the module as a whole. First-time readers (or n-th readers who can't be bothered to keep every it all in mind) need to be guided, and that guidance is more effectively achieved when the module author can influence the entire presentation, including placement and order of the API documentation, subdivision of pages, and documents that don't map to any one module.
This fundamentally clashes with the desire for simple, complete, accurate, module-structure-mirroring API documentation. There is certainly and will always be a place for that. Writing Sphinx documentation is much more involved than using Rustdoc even when simply delegating everything to directives that automatically include docstrings from the source code.
But as I already said, I think there is more to great documentation than a plain list of all API items. rust-lang/rust has rustbook, which is at opposite end of the spectrum: No automatic content at all, just a heap of hand-written content. Despite my earlier comment about Sphinx, I think it occupies a nice spot in the conceptual space of documentation: It generates API documentation from the source code, but it does so where and when and if the author wants it, with additional flexibility of how to format it and what to include. It's easy to (for example) present certain types first, omit irrelevant information, and write free-form text that segue from one part of the API documentation to the next. Mirroring the module structure is of course a manual process in this world — but the capability to deviate from it is a feature.
Bolting that kind of capability onto Rustdoc is completely unrealistic though. Feature creep, anyone? Perhaps it should focus on generating API docs and grow a machine-readable output format, to provide API docs as a service to another, Sphinx-like tool that covers other use cases. Everyone still gets API docs for free, and those who need/want to can write additional docs that present some of the content in a different format, without removing the API docs.
rustdoc has an option to output JSON, but last time I tried, it didn't seem to work.
I someone was to spend some time on this and provide me with a way to get my crate docs as JSON I'm sure I would spend at least one weekend on an alternative interface (suitable for a platform for hosting docs because that's something I've been wanting to build).
I was thinking about this. There are routes you could take: dimming less important items, grouping into categories by importance, adding gold stars, and having a mode which hides things. Maybe others. Hiding seems the most useful I guess. Here's a mockup of what it might look like (ignore the sidebar):
It would definitely direct people to look at certain specific things easier than now. I would imagine an "overview" mode would be site-wide until it's disabled (+/- would still work normally but hidden types wouldn't appear until it was disabled). You'd probably have to tag the types in the libs with #[doc="overview"] or favor or something to make it work well.
I think this is a very well spoken point and I would be interested in seeing how it could be utilized if it was implemented.
Anyway, since I will not be implementing any of the ideas which were presented, I hope someone who implements such things finds these at least a little useful.
