What to do about pulldown and commonmark?


I think part of the problem is that we don’t have such a clear policy around changes to things like rustdoc output, which do not affect your ability to compile code or the behavior of the resulting code. In contrast, we’ve provided explicit guidance for API evolution and language evolution, which for practical reasons do permit certain forms of limited breakage. On the other hand, the policy for our tools (cargo, rustup, rustdoc etc) is far less clear. We should fix that!

From my perspective, the change to warn when new tests would be run (but not run them) takes care of the primary stability concern, which is code suddenly failing to compile upon upgrade. But I do understand the point that silent changes to rendering behavior could damage our stability stability story.

There’s a principles vs pragmatics issue here as well: the time we spend trying to more thoroughly examine and warn about the (probably minor) changes here is potentially time we’re not spending improving rustdoc itself. (That could be a false dichotomy; with OSS, it’s always hard to know.) From IRC discussion yesterday, it sounded like @Snorge already had done a lot of work on a tool that could effectively diff the output. Is that right? Do we think we’re not so far of from being able to gather some more data here?

A final point: I know that many have been waiting a long time to land this change, but in the big scheme of things, there’s not so much urgency to push it out in this particular release cycle. The conservative option is always to delay. But I think, if we head that direction, we need to have a very clear story about what steps need to be taken before we do land the change, and e.g. what we will do if we are unable to gather useful data, and all stakeholders need to be on board.


A couple concerns here:

  • How confident are we that we can get enough people to opt-in and discover/fix problems? Maybe making it prominent in the release announcement is enough. But people may need some kind of diff tool to actually find changes.

  • Usually with “permissible breakage” we try to set things up so that you can “straddle” versions. That is, ideally you could change your docs so that they render correctly in both hoedown and pulldown-cmark. Is that possible? If not, how do we imagine the opt-in working?


I only see these concerns as relevant right this moment if we decide we’re not reverting for beta… we can always choose to revert for beta and then go forward with exactly what’s been done now in the next cycle.

Is there a chance we can come to a decision on what to do for beta first?


Oh, definitely, and if we don’t reach a resolution in the near future that’s what we should do. But I’m operating under the assumption that we have at least a couple more days to try to reach consensus before landing a revert.


To add on to what @aturon said, we constantly change rustdoc’s output. We fix bugs, tweak CSS, add new things.




Even without textual changes, there’s extra links, the removal of libc docs entirely, some other style things.

It is not possible to have stable rendering without completely abandoning any and all improvements.

What we could do, and one of the things I’d like to do, is make rustdoc’s rendering stuff pluggable, such that if you wanted to ensure things always stayed the same, you could use the exact same renderer to generate your docs, never getting any of those fixes or improvements. Effectively, that API would be what’s considered stable.

Yes, this 10000%. There are real, actual bugs that we keep in rustdoc because we cannot fix them without these changes. And it’s already been years of this delay, doing it Yet Again is extremely demoralizing, especially given that we mostly have theoretical concerns here.

If the community had gotten up in arms when we first announced this change, I’d have felt very differently. But only a few people are upset, and it’s mostly around concerns that others will be concerned in the future, which I can appreciate but also are extremely hard to hear after years and years of delay after delay and obstruction after obstruction.

If the people who actually do the work are empowered to actually have control over their work, then waiting for that day to come gets easier. It still stings, but it’s easier. I do believe that we’ll get there, and am kinda sad that it took this situation to actually make it finally happen, maybe.


In order to help everyone understand the type of breakage that will occur, I will share some more or less representative examples from crates.io crates. I found these all by compiling the clap crate, turning the docs into just trees of tags, and diffing for differences in them (ignoring tag content).

Here is an example of breakage in a crates.io crate from current stable to current nightly:

Crate: ansi-term, 0.9.0, last release 27 Aug 2016

Breakage: link reference and definition, lines 401-402

Stable rustdoc (hoedown) output:

<p>It might make more sense to look at a <a href="https://upload.wikimedia.org/wikipedia/en/1/15/Xterm_256color_chart.svg">colour chart</a>.</p>

Nightly rustdoc (pulldown-cmark) output:

<p>It might make more sense to look at a [colour chart][cc].
[cc]: https://upload.wikimedia.org/wikipedia/en/1/15/Xterm_256color_chart.svg</p>

The crate was last published over half a year ago and crates.io reports it is downloaded about 2,000 times per day right now. Someone is probably generating documentation that includes this crate’s documentation.

The markup is perfectly fine and accepted in hoedown (and Markdown.pl, since hoedown is pretty faithful to that) as a link reference and definition. However, CommonMark requires a blank line between the end of a paragraph and a link reference definition, so pulldown-cmark correctly leaves it as a paragraph.

Here is another instance of the same issue, this time from the clap crate.

Breakage: three link reference definitions, lines 57-61

Stable rustdoc (hoedown) output:

<p>Various settings that apply to arguments and may be set, unset, and checked via getter/setter
methods <a href="./struct.Arg.html#method.set"><code>Arg::set</code></a>, <a href="./struct.Arg.html#method.unset"><code>Arg::unset</code></a>, and <a href="./struct.Arg.html#method.is_set"><code>Arg::is_set</code></a></p>

Nightly rustdoc (pulldown-cmark) output:

<p>Various settings that apply to arguments and may be set, unset, and checked via getter/setter
methods [<code>Arg::set</code>], [<code>Arg::unset</code>], and [<code>Arg::is_set</code>]
[<code>Arg::set</code>]: ./struct.Arg.html#method.set
[<code>Arg::unset</code>]: ./struct.Arg.html#method.unset
[<code>Arg::is_set</code>]: ./struct.Arg.html#method.is_set</p>

In this case, the hoedown one doesn’t even seem to agree with Markdown.pl. The pulldown-cmark one does agree with CommonMark, but of course that’s not the output we’d want to see.

There are a number of other differences just from the clap crate and the other docs that come along when its docs are generated. I won’t go into detail on every one here, but some other examples I found multiple occurrences of in this rather small collection of docs are:

  • “automatic links” in pulldown-cmark need <> around them, while hoedown didn’t require that
  • lists in pulldown-cmark can occur immediately after paragraphs, but hoedown won’t treat those as a list unless there was a blank line before (usually this one would be “fixed” by switching to pulldown-cmark)
  • emphasis marks around a word need a space after the closing one in commonmark ("**NOTE:**Words words" stays as-is in pulldown-cmark, but is “**NOTE:**Words words” in hoedown and Markdown.pl)

EDIT: The reasoning behind the final bullet point above regarding emphasis marks was incorrectly worded. If the text was **NOTE**Words words, it would be fine, but since it was **NOTE:**Words words (note the : between NOTE and the **), it didn’t work as expected. CommonMark does not count the second ** as a right-flanking delimiter run because it is preceded by a punctuation character and not followed by whitespace.


Thanks so much @Snorge for doing this investigation. Your findings suggest that the problems are more widespread and significant than what had previously been laid out (e.g. in the original post). That’s definitely enough to raise concerns about Rust’s stability if we ship it to stable, IMO. @steveklabnik, @GuillaumeGomez, thoughts?

How far along is your diffing tool, in terms of being something we could use in the compiler to warn about anticipated changes?


Sorry, I really don’t even know enough about Rust yet to make a tool to do this in an automated fashion. I’m still working my way through reading the book.

All I’ve done to find those results is:

  1. Ran cargo doc on the master branch of the clap crate with rust nightly-2017-03-28 and nightly-2017-04-08. (The day before #40338 landed and the day after #41112. Not sure if there would be better choices to use here.)
  2. Made slight modifications to this example script from html5ever to only output one tag per line, omitting all content, attributes, etc. (So we’re ignoring the actual text between tags, just looking at tags to get a diff with virtually no “noise”.)
  3. Ran both sets of docs through that script to get files of just tags
  4. Diffed the resulting directories

It could certainly be the case that this crate was an outlier. Most crates seemed to have almost no documentation at all, so I just went with the first one I found in random clicking around that actually appeared to have something to check on. There were almost 7000 HTML files in the directory but only 10 files had differences. Some of those 10 had multiple differences and maybe most of the 7000 files were autogenerated or something, I don’t know enough about the directories to know.

I suppose if there was a crate that came along with a lot of other crates’ docs I could try the same thing again, but I’m guessing someone else would be able to do something like that on a wide array of crates faster than I could figure it out.

I wanted to run the same thing on the standard library docs but I couldn’t figure out how to run two versions of x.py doc on one set of files.


I think it is better to let pulldown-cmark provide a hoedown-compatible mode, which generates output parses input like hoedown and report those incompatible issues to the driver (rustdoc), based on the diff. This should be easier to generate the warnings inside the crate instead of inspecting these from outside in the compiler.


The issue isn’t in output generation but parsing. pulldown-cmark separates these two tasks, but it is not at all capable of a “hoedown-compatible” parsing mode.

In its current state, it isn’t even easily modifiable to fix the 50 failing CommonMark spec tests, because it made certain assumptions that later CommonMark versions have clashed with. Raph seems to have more or less given up on the current approach and it’s likely the fixes won’t come until the inside is redone in a significant manner.

It would be nice to have pluggable support for other parsing modes (especially to allow multiple versions of CommonMark, especially since it is still pre-1.0), but doesn’t seem to be anywhere near that point.

Edit: I see you updated your post. The output/rendering side of pulldown-cmark seems (again, I don’t know enough about Rust to say this with confidence) to be very flexible and easy to use in neat ways. Modifying the input/parsing side on the other hand is not in any way smooth and simple. It is fast, but on the messy side. Raph has shared a prototype branch with a possible new approach that actually builds a tree in parsing, but it was incomplete and still wasn’t built in a “pluggable” manner where one format could be swapped for another easily. This is probably part of the reason that the package is still at version 0.x.



  • Most of the time, there’s a pretty obvious way to write your syntax so that both render it correctly. The comment I’m writing in here is rendered identically by every implementation listed in babelmark
  • If there isn’t, you can always fall back on inline HTML.


I’m just afraid that if the hoedown check is performed in rustdoc/rustc, it would mean rustdoc/rustc needs to implement a half-baked markdown parser to do so.


I suppose I’m less concerned with how this first transition happens and more concerned with the fact that changing from hoedown (which I assume is stable and doesn’t break itself regularly) to pulldown-cmark and therefore commonmark (which seems to break itself regularly) will result in this happening frequently, if not every release.

If the proposal was: Let’s move from a library that is stable to another library that is stable, it would be a one time transition.

But the proposal as I see it is: Let’s move from a library that is stable to another library that is not stable. That implies arbitrarily many transitions of arbitrary complexity until it becomes stable. Even given the lack of specificity in the stability RFC about rust tools, it seems that this may be a bridge too far, even if just a mostly theoretical concern?


Here’s what I think we should do (and sorry for being a bit terse; I have way more thoughts on this but I’ll keep this brief).

Revert the change from 1.18 beta. Quality is super important to Rust and randomly breaking things is something to avoid. Use the time of the next train to gather empirical data. The most important thing is a prioritized list of diffs of breakage from hoedown. I can try to hit the top N, but more than that I’d like to mentor the community. Working in the codebase is not that hard, there’s no rocket science. It just takes some patience and persistence.

The low test scores are a red herring. We were at 100% for a snapshot a year or so ago, and things haven’t broken, just haven’t kept up with spec changes. If anything, it’s a reflection of the instability of the commonmark draft. It seems extremely unlikely to me that any of the breakage is people relying diffs in the spec from the point where pulldown hit it. I’m pretty sure almost all of it is the completely spec-free extensions. The newlines after footnote (https://github.com/google/pulldown-cmark/pull/21) is a classic example. That’s a PR to make the behavior more like hoedown, but I didn’t merge it because I thought the existing behavior was more consistent with the commonmark spec (ie likely that there would be some future spec covering footnotes that would be consistent with the current behavior). The evidence that lots of existing doc is a compelling reason to merge the PR, so I almost certainly will.

Going forward, it’s unfortunately impossible to provide a strong stability guarantee. This is true no matter what library is used. No complete Markdown spec is close to a 1.0. CommonMark is excellent work, but doesn’t address the extensions at all. As the spec changes (whether it ever hits 1.0 or not) it has the potential to break existing doc. If and when a 1.0 ever lands, compatibility with it will be a valid reason to break existing doc, and then we can reasonably ask people to upgrade.

An interesting question: how much breakage would there be if we adopted GFM? That seems likely to me have a usable spec at some point in the future. I’m not suggesting we do this, but I think having the empirical data would be very useful.

Don’t worry too much about the dev branch. That has the potential for significant speedup, fixing extreme inline markup nesting edge cases (which will never happen in real rustdoc), and being a cleaner codebase to work in. But I don’t think it blocks anything.

Thanks @Snorge for investigating. In the absence of a 1.0 spec, empirical data rules.


My thoughts, which are admittedly maybe a bit cynical, are that most people will view this like @SimonSapin: https://www.reddit.com/r/rust/comments/66g0d4/attention_crate_authorsmaintainers/dgianvs/?st=j1qjniaz&sh=beb66cd9

I don’t plan to double-check everything because it seems high-cost and low-benefit:

  • Checking this manually sounds very time consuming. In fact it sounds like a good job for computers. A tool could render every doc-comment with both renderers, compare, and flag for human review those that have significant differences. (Where “significant” needs to be defined but includes things like other than collapsible HTML whitespace.)
  • What’s the worst that could happen? I expect it to be very rare that pulldown-cmark silently drops content that hoedown does not. As to content formatted not-quite-right, as long as it’s still readable…

There are many crates that I (co-)maintain, but the most prominent one might be url.

I think we could spend a lot of time and effort into doing this transition, using this tooling, and I don’t think many people will even use it. Remember, most people didn’t even care about this change in any way until it caused some test failures, even though we explicitly told everyone “hey your docs might render strangely.” I expect pretty much more of the same.

You’re only considering one aspect of this change here. That is, there are other gains which are significant here:

  1. Since CommonMark has a spec, we can actually determine if rendering is a bug or not at all. Hoedown has no such thing, and so any bug fix we make to it can break this kind of rendering.
  2. Hoedown is in C, pulldown-cmark is in Rust. This moves us closer and closer to a pure-Rust distribution.
  3. Hoedown blocks things like this RFC, which does two primary things: fixes long-standing bugs in our doc generation that continue to confuse developers to this day, and improves the lives of doc writers by removing a lot of manual boilerplate.

CommonMark is pretty close to having a 1.0 spec release; that is, in my understanding there are 8 outstanding issues, and

With your help, we plan to announce a finalized 1.0 spec and test suite in 2017.


GFM is being re-done as extensions to CommonMark, so this move is already a move towards GFM.


If you break something once and then promise not to do it again, people might grudgingly tolerate it.

Once you break something twice, that establishes a trend, and people no longer trust you not to break it, regardless of what you promise.

If we suspect that future developments to pulldown-cmark or future changes to CommonMark will result in us having to go through this whole process again, then we should hold off on this change until we can promise that no more breakage will occur (at least to the same practical extent as our normal stability guarantee).


To be clear, I don’t think that changes to CommonMark are going to cause actual breakage, it sounds like they’re actually close to a 1.0 now (good to hear!). I’m more worried about the extensions. Switching to pulldown-cmark is actually a win for stability there because we get to control any changes we make.

The point I was making is that the level of guarantee we’re physically able to make for markdown rendering is a lot lower than the 1.0 Rust language guarantee, because of limitations in the spec standardization process.


Thanks, Raph, especially for this offer to use this as an opportunity to grow the community!

We’ve basically run out of time to discuss this further before beta is cut, and I think that @snorge’s findings suggest that we have more work to do in analyzing where we are, as @raphlinus is also advocating for. In particular, as I said above, it seems plausible that problems are much more widespread than we thought, and rolling out a change that breaks linking across the ecosystem seems very risky.

Docs and core team: I propose we quickly land a change that reverts to using hoedown (we can keep pulldown-cmark around), and try hard to formulate a clear plan on the next steps.


Note: I asked about reverting on the docs team channel. While @steveklabnik is away, at least @GuillaumeGomez agrees that reverting makes sense at this point and is preparing a PR to do so.