Rustdoc: Asciidoctor vs Markdown

[DanielKeep]

I can’t be the only one who realises that we’re never going to make everyone equally happy by picking one and only one markup language… this argument is old, and is just getting silly at this point: there is too much extant documentation out there to ever switch to anything that isn’t a strict superset of “the Markdown dialect implemented by hoedown”.

Given that we’re going to be getting stable compiler plugins in the form of Macros 1.1 soonish, why not just define a plugin interface for markup processors and let everyone use whatever markup is most appropriate for the task at hand?

Simple Markdown as a default, whatever else for people with more complex needs. Plus, that way, if someone needs some feature that doesn’t exist, they can just implement it themselves, rather than having to convince everyone that, no, Markdown is not sufficient, and then having to get rustdoc updated, and then wait for those changes to filter down to stable, etc., etc.

[ghisvail]

I can’t be the only one who realises that we’re never going to make everyone equally happy by picking one and only one markup language

You are not indeed. Some people might be coming from Python and prefer ReST for instance, whilst Markdown / CommonMark does the job for simple documentation.

why not just define a plugin interface for markup processors

Simple Markdown as a default, whatever else for people with more complex needs.

Sounds sensible if that could be made possible.

[DanielKeep]

Within rustdoc itself, markdown is mostly just a case of calling a function with the contents of a doc comment, and dumping the result into the output. From a (cursory) look, the hardest part would probably be modifying Cargo to support rustdoc plugins.

[djc]

@steveklabnik, any comments/ideas/feedback?

[steveklabnik]

I’m not totally opposed to some sort of pluggable renderer for rustdoc, but I’m not sure who’s gonna be willing to do the work, or how it’d work.

[djc]

So I think we should somewhat ignore the question of who might be able to do the work (I am willing, but not sure I can find the time), but if we start by hashing out the design that might also make clear how much work it is. Maybe this starts with a high-level description of how rustdoc works, do we have something like that?

Then, let’s say the crate level would be fine-grained enough, so let’s say there’s some kind of switch in Cargo.toml, maybe like this:

[rustdoc]
render = markdown

Then, I’d envision you could for example cargo install rustdoc-rest which would enable rustdoc building docs for crates that have render = rest.

[DanielKeep]

I’d much rather see rustdoc plugins function like Just Another Dependency, just like rustc plugins do. You’d add the markup processor as a dev-dependency (you shouldn’t need it to build the code), then use a #[doc(markup="rst")] annotation. I’m not a fan of blunt, global switches; it makes it harder to change existing code to use a different markup, or integrate code from another source (or perhaps even just another author). If incremental conversion can be enabled, I think it’s worth aiming for.

Simplest-to-implement with that property would be to require it on every item being documented (with the existing hoedown processor perhaps aliased as one of rustdoc, hoedown, or legacy). Simpler-to-use would be to make the annotation “stick” to child items (i.e. set it on a module, and all the things inside of it get the same processor by default).

I actually started poking around using external processes (named rustdoc-mp-X), but abandoned that when I realised that "cargo install and check PATH" would be utterly lousy for (flogs self for using a term he hates) UX. Plus, it seemed like it’d be a hassle for docs.rs, and I do not want anything that would make docs.rs harder to operate. I got rather discouraged at that point, because that was when I realised I’d have to modify Cargo, and the last time I went there I didn’t have any fun whatsoever.

Aside: while I definitely want reStructuredText, I also feel like using a markup processor shouldn’t affect build times for dependencies at all, nor should it blow out doc generation times too much, or require users to contort their system configuration. I think any such change should come with a very strong community expectation that you do not depend on external binaries or libraries that might not be available, or aren’t easy to build automatically on all platforms.

In other words, someone’s gonna have to re-implement rST in Rust

[skade]

I’ve got one issue with all this: the biggest advantage with the current state of things is, that the following works:

$ cargo init
$ vim Cargo.toml
$ vim src/codealittle.rs
$ cargo doc

And I get docs for everything. If the documentation parser suddenly gets the ability to use arbitrary tools not shipped with the rust toolchain, I see this breaking.

I don’t want to see it break. It’s an extremely important property, especially for beginners.

[DanielKeep]

If the plugins are supported by Cargo, then this is no more likely to break than any other package not compiling for whatever reason. What’s more, if the markup processor is complicated to get working, crate authors might find themselves with nothing on docs.rs, which will hopefully be a strong incentive to not let that happen.

[skade]

which will hopefully be a strong incentive to not let that happen.

And so begin sad stories.

Here’s a thing: asciidoctor, the software we’re talking about here, is already “not simple to get working”. It needs a completely different stack (Ruby, etc…). The problem there is that for many library authors, they have multiple stacks installed quite fine. But if you start setting up things, you run into a horrible mess - just to see the type documentation.

I think cargo doc should work when cargo build works - all the time, on any supported platform.

See how this is different from using something like sphinx or asciidoctor for your project page or long-form docs. They happens at a different time and will probably only be run if your project is the top-level project.

We’re currently talking about giving arbitrary crates in the dependency chain the ability to break this guarantee.

[DanielKeep]

I feel like you’re just ignoring what I’ve been saying:

Aside: while I definitely want reStructuredText, I also feel like using a markup processor shouldn’t affect build times for dependencies at all, nor should it blow out doc generation times too much, or require users to contort their system configuration. I think any such change should come with a very strong community expectation that you do not depend on external binaries or libraries that might not be available, or aren’t easy to build automatically on all platforms.

In other words, someone’s gonna have to re-implement rST in Rust

I know this might happen. But Markdown is rubbish, and I’m tired of not being able to add diagrams, or TOCs, or math. But I don’t want rustdoc to have to support every markup under the sun because I know what I want is not what others will want, not to mention the ludicrous maintenance burden that’d entail. To put it another way: there’s no way solving this isn’t going to at least bruise a few eggs somewhere, and I’d rather this be addressed than not.

Also, keep in mind: there are times I’ve been seriously tempted to just bypass rustdoc entirely because I was getting tired of its limitations. If I do that, it wouldn’t matter how complicated the doc processing step is, because you wouldn’t be able to cargo doc it anyway. Integrating this into Cargo and setting a strong community expectation has a good chance of heading this off completely.

Plus, I’m on Windows, so random crates not working because “all computars r unex” is just a thing that happens. I wouldn’t view a crate that requires Ruby or Python any differently to a crate that requires Bash on Windows: badly written and should be fixed.

[skade]

I know this might happen. But Markdown is rubbish, and I’m tired of not being able to add diagrams, or TOCs, or math. But I don’t want rustdoc to have to support every markup under the sun because I know what I want is not what others will want, not to mention the ludicrous maintenance burden that’d entail. To put it another way: there’s no way solving this isn’t going to at least bruise a few eggs somewhere, and I’d rather this be addressed than not.

First of all, this is highly specific to the markdown implementation rustdoc uses. Second, the maintenance burden is there anyways: breakage of independent programs is still breakage.

I agree that the problem does need to be addressed, but the number of people aggressive about rustdoc is far larger then the number of people willing to work on rustdoc, the markdown implementation (or a successor?).

What I’m opposing to is the “we don’t want to provide implementations, so let’s just find the easiest way to hack around”.

I don’t want to see the immediacy of cargo doc break.

Also, keep in mind: there are times I’ve been seriously tempted to just bypass rustdoc entirely because I was getting tired of its limitations. If I do that, it wouldn’t matter how complicated the doc processing step is, because you wouldn’t be able to cargo doc it anyway.

I could still doc all the type information and standard docs if you feel to inclined to use that.

(btw. I’m taking that route, through tango, because a lot of documentation doesn’t fit the “annotate an item” approach, IMHO)

Plus, I’m on Windows, so random crates not working because “all computars r unex” is just a thing that happens. I wouldn’t view a crate that requires Ruby or Python any differently to a crate that requires Bash on Windows: badly written and should be fixed.

a) This is an issue unrelated to docs. b) badly written will happen and drive of new people, especially when things considered to basic as documenting break.

[getreu]

We discussed this matter on our Asciidoctor forum. I forward you "johncarl81"s advice:

"If RustDoc is anything like JavaDoc, I’d heavily suggest checking out our Asciidoclet project:

asciidoctor/asciidoclet

Write Javadocs in AsciiDoc with Asciidoclet | Asciidoctor

Asciidoclet 1.5.0 released! | Asciidoctor

This was a super easy adaptation of Asciidoctor which has full access to the rich ecosystem of plugins. I love the ability to write inline UML and formatted code snippets with the minimal syntax requirements of Asciidoctor."

[skade]

The huge advantage of a doclet is that you can just ship jruby along with it, which is a perfectly usual solution in the Java world.

We’d have to use something that can be linked as a library.

[jan_hudec]

Both Ruby and Python can be linked as libraries.

But that’s not the right requirement anyway. What matters is whether cargo can install it. And I think it could be taught to install “alien” libraries, by calling to appropriate package manager (python and ruby both have one) and eventually possibly even installing a “portable” version of python/ruby/….

[skade]

Sure, everything can be done, somehow.

What you describe is quite different from shipping a bundled interpreter that runs on all platforms the language supports just by including it in the maven file.

[jan_hudec]

Well, documentation generator does not have to run on all target platforms, only all platforms that host the compiler. And there is only three of those — Windows, Linux and MacOS (times 32/64-bit variants, but the 64-bit ones can run 32-bit interpreters if needed).

For maintenance, library in Rust is easier. But if it has to be written while suitable tool already exists in some other language, creating a suitable installer to pull it in if needed is less work up front.

(for the installer route, I wouldn’t go for anything less than pandoc).

[susurrus]

I’ve been following this issue since the originaly 80-comment thread in the start and I think that supporting markdown in the default case and optionally another format, whether it’s asciidoc or ReST (the two most popular proposed alternatives) makes sense. A common refrain is also that the tool should be written in Rust. Given that an implementation of ReST exists in Rust, I’d like to move forward with actually implementing support for ReST in combo with Markdown. I’m not certain if rust-rst is ready for prime time, but let’s see what happens.

To this there are two questions though:

How do we want to support multiple formats?

Current proposals are:

1. Add a dev-dependency in Cargo.toml like #[doc(markup="rst")] or specifying the renderer to rustdoc like [rustdoc] render = rst
2. Add a setting above every doc comment
3. Add a special string (like {{ReST}] as the first line in a doc comment

I’m not a fan of any of these because it adds so much boilerplate. I’d like to move towards a solution where the syntax can be auto-detected, say by running both formatting in parallel and using ReST if it succeeds otherwise Markdown. I’d also like to add in my own of running both in parallel and using the ReST output if it succeeds and otherwise falling back to the markdown version. There is a severe overlap in syntax between both of them, so maybe this would work? Only problem is you might be unpredictable output depending on the renderers used, so maybe combo this with 1?

Proposal for implementation in rustdoc

Does rustdoc follow the same RFC guidelines as that used by rustc? Should I start writnig up a pre-RFC for this issue?

[Michael-F-Bryan]

I like the #![doc(markup="rst")] syntax which was mentioned earlier. You could add a top level #! attribute which declares your crate uses rst for documentation by default, then you can individually annotate bits that are in asciidoc or markdown.

The way you implement this would probably be as a compiler plugin which you add as a dependency to your crate.

As an intermediate measure (more a hack really), because markdown allows embedded html, if your compiler plugin was to translate doc comments to html before rustdoc goes over the crate then it’d mean the transition is fairly seamless.

A system like this gives you the pluggable behaviour that you are looking for, meaning if people are fine with using markdown they don’t have to change anything. But say you want to work in asciidoc or ReST, then it’s just a matter of including the right plugin.

You still need to have a Rust implementation of a ReST or asciidoc parser, however there are already implementations in other languages to refer to, and parser libraries like nom exist on crates.io already.

[getreu]

I opted for Asciidoc 5 years ago and hadn’t touched ReST since. Yesterday I had a closer look into ReST and it has improved a lot! There is a proper specification now and at first glance it seems I can find everything I was missing 5 years ago. I especially like how it handles links and I decided to give ReST a second chance.