I'm by no means the most educated or motivated on this one, but all the people who are better qualified to write this have better things to do, so here we go!
On The Potential Inadequacy of Markdown
Rustdoc currently uses "vanilla" markdown with some common extensions for all of our documentation. This is done by piping through a C library called Hoedown. As more proposals pop up to tweak or extend our documentation capabilities, this becomes an increasingly questionable position.
Performance Docs: Performance is a distinct issue to raw usage or behaviour. For this reason I believe it would be desirable to separate it out as something that can be programmatically identified and manipulated. Possible usecases include "perf only" filters and tables of operations for e.g. collection traits.
Safety/Failure Docs: Similarly to the performance issue, safety and failure are important issues that would be convenient to extract/manipulate.
Inter-Api Linking: Being able to specify a part of some Rust api, and have it automagically linked without knowing where it would appear in the generated page hierarchy
Context-Sensitive Header Levels: Markdown considers an
# to be an
<h1> and not simply "the highest level heading reasonable where this will be embedded", which seems to really be what we want.
Mathjax Support: It would be great to be able to properly render simple and complex snippets of math using Mathjax, but the underlying server-side text processor needs to cooperate.
These issues can be individually hacked around.
#'s can be migrated to
##, or we can (try) to string-replace
<h2>'s. Safety/Failure/Perf docs can be designated by "magic"
#performance, etc headers. Magic URLs can be post-processed to find relevant docs. Mathjax can probably be backdoored in with some exceptionally brittle regexes. But these are hacks that have a debt associated with them.
Hoedown itself probably can't be configured to provide these features, as it is exceptionally unconfigurable, as far as I can tell. Special strings/characters are all hardcoded. Features are just toggled by hardcoded magic bitflags. I don't think we want to take on supporting our own fork of it to provide these features. Hoedown being written in C also presents a potential security vulnerability (compared to the rest of Rustdoc, at least).
All of this piles up to the point where you start saying "maybe markdown isn't right for the job". Now of course, markdown isn't without merits. It's the defacto internet standard for internet communication. These posts, Github posts, Reddit posts, and various others are all written in markdown. If you're contributing to Rust, you probably know at least some markdown, and can therefore pick up and modify the docs easily. Markdown also excels at intuitiveness in my opinion. However, markdown also lacks a good standard. It's more like English than a formal language, in that various semi-compatible dialects exist.
However, most critically of all, thousands of words of Rust docs are already written in markdown.
reStructuredText is the language used by Python and Swift to write their documentation in. It is well-defined, extensible, and critically designed for exactly our usecase. Chris Morgan originally proposed and provided some arguments for this migration (as far as I know) in his talk here (at 33 mins -- although Steve's talk before it is also worth reading).
I don't have any familiarity with this language myself, so maybe those of you who do can chime in. However if you skim the quick guide you can see that the common usecase of just making prose is pretty straightforward and easy. However it's definitely more complex than markdown.
To use reStructuredText, we can try to hook into Sphinx, the tool built for Python's docs, or write our own native parser/handler. I haven't done enough research into Sphinx to determine the relative merits of these positions.