Feedback request: proposal to drop Rdiscount and Redcarpet support

[benbalter]

As we're preparing to upgrade to Jekyll 3.0.x, we're looking for ways to make things even easier for new users to get started. One point of unnecessary cognitive burden is choosing a markdown interpreter.

Kramdown? Rdiscount? Redcarpet? What's the difference? How do I choose? Why can't you just pick the best for me? When I fire up WordPress or Microsoft Word or my phone, I don't choose the rendering engine. I trust the publisher to make that choice for me. Using Jekyll should be no different.

There used to be a lot of difference between the various Markdown rendering engines. Some supported some features, others implemented different features in different ways. Today, they've largely converged on a standard feature set, with Kramdown implementing most, if not all features (maybe not options), across all renderers.

My question

Other than speed, is there anything Rdiscount or Redcarpet does that Kramdown doesn't? Would you miss them terribly, or would your site fail to render if GitHub Pages dropped support for any markdown renderer other than Kramdown?

If there are any specific examples of features or syntaxes that RedCarpet or Rdiscount supports that Kramdown does not (rather than personal experience or preference), it'd go a long way to help inform our decision.

Rdiscount and Redcarpet breakdown:

• Public sites explicitly using Redcarpet: ~24,000 (~2%)
• Public sites explicitly using Rdiscount: ~18,000 (~2%)

As a baseline:

• Public sites still asking for Maruku, even though it no longer works: ~1,200. (~0%)
• Public sites explicitly asking for Kramdown, even though its the default: ~50,000 (~5%)
• Public sites that specify any markdown generator: ~90,000 (~9%)
• Sites just using the default with no markdown engine specified: 90%+

For open source projects I normally like to optimize for the 80% use case (not edge cases or power users), but here this means that 95%+ of users are using Kramdown (and 90%+ don't care enough to specify). For these numbers, you can assume 1M+ public sites.

To note, this proposed change would only affect GitHub Pages. Users would be free to use any rendering engine they'd like when running Jekyll locally or on their own server.

[Skyost]

I am using the default markdown engine for my blog so I won't miss them.
Anyway, what is the approximative upgrade date ?

[envygeeks]

The most obvious problem here is backticks. Kramdown does not support the officially sanctioned Github way of doing things with backticks and this has come up a few times in our own tickets when users select Kramdown and the decision was made to not explicitly work around the lack of flexibility in processors so we never added support for them.

[geraldb]

FYI: You can use:

kramdown:
   input: GFM

To add support for backticks. Might be a great default (In my humble opinion - I also prefer backticks for code blocks). Cheers.

[sevenseacat]

Also consider that there are private repos also running blogs via GitHub pages, like my own. The percentages are probably fairly similar, but still.

[damncabbage]

I tried switching over to Kramdown (with GFM as above), but:

1. Haven't gotten syntax-highlighting working at all on GitHub Pages itself. I've tried Coderay, Rouge and Pygments (what I used with Redcarpet), and have come up with nothing.
2. Locally, Pygments doesn't seem to work with Kramdown, and Coderay doesn't seem to know how to detect language based on whatever I give after the start of the code-fence; the highlighting looks rubbish with YAML, for example.

I've seen some mention of a blessed "Pretty Lights" highlighter that's not been released. I'm a bit in the dark here. What highlighter are we supposed to be using with Kramdown?

[Phlow]

I like Kramdown. I think to reduce the clutter is a good idea.

[jeremenichelli]

I prefer RedCarpet for Markdown syntax. In case it's deprecated is there a way I can configure Kramdown so I don't have to go and change all my posts's syntax? I don't want to wake up one day and see my site does not render as expected.

[milanaryal]

👍 Kramdown

[r-brown]

The numbers as of 2015-11-28 may be slightly different:

• All public sites build using Jekyll 419,532 (100%)
• Public sites explicitly using Redcarpet 23,759 (~5.66%)
• Public sites explicitly using Rdiscount 18,337 (~4.37%)

We are also using kramdown on our sites, so this won't hurt us so much.
But other almost 50K (10%) users will get definitely a bad experience with this stack.

IMO this support shouldn't be simply switched off but the users should rather be notified in advance.

Best,

[benbalter]

Haven't gotten syntax-highlighting working at all on GitHub Pages itself... Locally, Pygments doesn't seem to work with Kramdown

I can confirm syntax highlighting and Pygments both work Kramdown (and are actually handled at the Jekyll level, via the highlight tag).

I prefer RedCarpet for Markdown syntax.

@jeremenichelli can you provide some specific examples of where Kramdown doesn't support a syntax that RedCarpet does?

All public sites build using Jekyll 419,532 (100%)

@r-brown to note, a site doesn't need a _config.yml file to be a Jekyll site (the number of Jekyll sites is roughly double that, by my records).

[benbalter]

One last call, if there are any specific examples of features or syntaxes that RedCarpet or Rdiscount supports that Kramdown does not (rather than personal experience or preference), it'd go a long way to help inform our decision.

[waldoj]

Nope, no problem for my Jekyll sites.

[benbalter]

IMO this support shouldn't be simply switched off but the users should rather be notified in advance.

We still need to discuss things internally from a product/support perspective, but right now I'm hoping that when we move to Jekyll 3.0.x we can also announce a six-month sunset for Rdiscount and Redcarpet support (with warnings on push, similar to what we did with Maruku).

[jlord]

I would be sad!

Kramdown does not support the officially sanctioned Github way of doing things

☝️ This.

I use Pages to host the site and documentation for atom/electron. This site uses the markdown documentation as it is in the GitHub repository. The problem is that GitHub.com renders markdown different than Pages. Redcarpet is the most similar, though it's still does not have feature parity.

If Pages switches to Kramdown I'm guessing none of the code blocks in the Electron documentation will render correctly on the site? I think the ultimate (and kind of obvious) goal should be to have markdown on Pages and GitHub.com be the same.

[benbalter]

The problem is that GitHub.com renders markdown different than Pages.

@jlord as @geraldb suggested above, have you tested putting Kramdown into GitHub-flavored Markdown mode?

kramdown:
   input: GFM

Edit: Kramdown-GFM docs

[thexavismith]

👍 I like Kramdown and don't find myself needing the others.

[holman]

I can confirm syntax highlighting and Pygments both work Kramdown (and are actually handled at the Jekyll level, via the highlight tag).

From a "imma write some Markdown now" perspective, the highlight tag never made much sense to me, since you have to leave Markdown to use it. (Although the fenced code syntax we chose itself was a misstep too, in hindsight, but I digress). +1 for definitely using the GFM setting as a default in Jekyll; the interface should just be Markdown, and Markdown's flavor differences are confusing enough. GitHub should at least put a consistent foot forward.

[mattr-]

Let's flip the question around a little bit, if we can.

From a product/support perspective, what (if any) additional burden does supporting Redcarpet/Rdiscount cause Pages? I understand that it's not for free, but what harm are we causing by not removing it?

[envygeeks]

I would think there would be less burden on their build servers by allowing the C equivs.

[geraldb]

FYI: I've put together a simple test page for kramdown using the gfm mode (see testgfm.html and the source testgfm.md using these settings:

markdown: kramdown

kramdown:
  input: GFM
  hard_wrap: false

It works as expected (if you turn off the hard_wrap) :-)

[jlord]

have you tested putting Kramdown into GitHub-flavored Markdown mode?

@benbalter Hmm, I feel like if that had worked I wouldn't be in the situation I am now with the site and wouldn't have had so much back and forth with you over the past few months re: markdown/Pages. But it has been months now and memory is fuzzy so I will give it a go again. If there is true parity between GitHub.com and this then ✨

[envygeeks]

@geraldb mind filing a ticket on Jekyll with those settings so I can add them to the Kramdown rewrite as default options? I'll forget by the time I get about to working on it again but I think those would be sane defaults for Jekyll

[geraldb]

@envygeeks Sorry I'm not at GitHub issue expert (do not know how to reference the ticket). Find it at jekyll/issues/4202. Cheers. PS: Thanks for your great work. Keep it up.

[jeremenichelli]

I've tried this configuration locally:

highlighter: pygments
markdown: kramdown
kramdown:
  input: GFM
  hard_wrap: false

And the result is that it works good except that inside the pre code tags I don't have the tokenization I had previously identifying comments for example so I can style them.

[envygeeks]

Yeah Kramdown doesn't support Pygments, it only supports Rouge and Coderay because both of those are pure Ruby and it's a pure Ruby Markdown processor.

[jeremenichelli]

If there's no native alternative for highlighting I'm against removing all Markdown engines except for Kramdown since you're removing a feature from Jekyll.

[envygeeks]

@jeremenichelli well... technically Pygments isn't a feature of Jekyll anymore, in 3.0 we switched to Rouge ourselves as the default processor, Pygments is an optional now.

[jeremenichelli]

@envygeeks well, but it's available, that's what I meant. My point is that I'm using Jekyll locally and I'm able to use highlighting, GitHub Pages at least would need to offer an alternative if Kramdown will be the only Markdown engine present.

[geraldb]

@jeremenichelli If you changed the _config have you tried to restart the server? For more tips, see How to install Rogue in Jekyll. Cheers.

[jeremenichelli]

@geraldb yes, I restarted the server. Though I did not install Rouge since I thought it was already supported in Jekyll. If that's not the case then my suggestion would be to propose Rouge to be present in Jekyll as Kramdown is, and then GitHub Pages can shut down other makrdown engines and leave Kramdown only, at least that's my humble opinion around this.

[gettalong]

@jeremenichelli If you want to test kramdown with rouge, you can do the following (from a POSIX-Shell):

$ echo -e "~~~ ruby\nclass New\nend\n~~~" | kramdown --syntax-highlighter rouge

Output with rouge:

<div class="highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">New</span>
<span class="k">end</span>
</code></pre>
</div>

If this works, then kramdown with rouge works. So if the output with Jekyll is not correct, then there is probably a problem with Jekyll.

Also note that if the rouge library is not installed, kramdown will fall back to not doing any highlighting. It doesn't currently emit a warning in such a case but this can be changed if needed.

Fallback output when rouge is not available:

<pre><code class="language-ruby">class New
end
</code></pre>

[gettalong]

@jeremenichelli Also note that you need kramdown >= 1.5.0 for rouge support.

[gettalong]

@jeremenichelli Yes, you need to have rouge installed locally if you build locally.

As for Github Pages, if the information on https://pages.github.com/versions/ is correct, kramdown is at version 1.5.0 but they don't have rouge installed. So it seems that Github Pages currently doesn't support rouge?

[jeremenichelli]

Well, this is already a bad sign, I imagine this can cause some fragmentation between Jekyll and GitHub Pages and that's not good in my opinion. I prefer features to be aligned across them.

[parkr]

it seems that Github Pages currently doesn't support rouge?

That would potentially change. We would go the pure-Ruby route of kramdown + Rouge.

[jeremenichelli]

@parkr if that happens, then, no fragmentation across GitHubPages gem and highlighting engines. GitHub staff can potentially let everyone know in advance and that's a thumbs up to me.

[benbalter]

Sounds like we have a path forward. Thanks everyone for the feedback. The additional context is extremely helpful. Stay tuned for more information about GitHub Pages upgrading to Jekyll 3.0.x.! 😄

[envygeeks]

@benbalter @gettalong I'd slow down on that "path forward", today we tried to convert our largest site to Kramdown and ran into a problem, GFM doesn't actually enable fenced codeblocks the way they work on Github, ```sh does not behave like ~~~sh actually we get no highlighting at all with the former.

BTW this is on Jekyll 3.1.alpha through: https://github.com/jekyll/jekyll/blob/master/lib/jekyll/configuration.rb#L71-L72

I don't know if this is intended but it's certainly how it's behaving for us on Kramdown 1.9, for us it's not a big deal, we don't process with Github so we worked around it by pre-parsing our source before Jekyll get's to it, but it might affect others

[gettalong]

@envygeeks I get the following results when running on the command line (just kramdown):

Input:

```sh
bash test
```

Output with kramdown -i GFM:

<div class="highlighter-coderay"><div class="CodeRay">
  <div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>bash test
</pre></div>
</div>
</div>

Output with kramdown -i GFM --syntax-highlighter rouge:

<div class="highlighter-rouge"><pre class="highlight"><code>bash <span class="nb">test</span>
</code></pre>
</div>

Seems about right to me.

[envygeeks]

@gettalong Given this snippet (taken from https://github.com/jekyll/docker):

## Running
***If you do not provide a command then it will default to `jekyll s`.***
### Native Docker
```sh
# Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`
docker run --rm --label=jekyll --volume=$(pwd):/srv/jekyll \
  -it -p 127.0.0.1:4000:4000 jekyll/jekyll
```
### Docker-Machine
~~~sh
# Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`
docker run --rm --label=jekyll --volume=$(pwd):/srv/jekyll \
  -it -p $(docker-machine ip `docker-machine active`):4000:4000 \
    jekyll/jekyll
~~~

Inside of "index.md" with a _config.yml that states markdown: kramdown with the default options (of which I linked to earlier) in 3.1.alpha on our builder server I get the following results:

<p><code class="highlighter-rouge">sh
# Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`
docker run --rm --label=jekyll --volume=$(pwd):/srv/jekyll \
  -it -p 127.0.0.1:4000:4000 jekyll/jekyll
</code></p>

<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`</span>
docker run --rm --label<span class="o">=</span>jekyll --volume<span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>:/srv/jekyll <span class="se">\</span>
  -it -p <span class="k">$(</span>docker-machine ip <span class="sb">`</span>docker-machine active<span class="sb">`</span><span class="k">)</span>:4000:4000 <span class="se">\</span>
    jekyll/jekyll
</code></pre>
</div>

Only the second one (the ~~~) is properly highlighted.

[envygeeks]

The problem is probably the tildes in the tildes.

[gettalong]

On the command line I get the following output using kramdown -i GFM --syntax-highlighter rouge --no-hard-wrap:

<h2 id="running">Running</h2>

<p><strong><em>If you do not provide a command then it will default to <code class="highlighter-rouge">jekyll s</code>.</em></strong></p>

<h3 id="native-docker">Native Docker</h3>

<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`</span>
docker run --rm --label<span class="o">=</span>jekyll --volume<span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>:/srv/jekyll <span class="se">\</span>
  -it -p 127.0.0.1:4000:4000 jekyll/jekyll
</code></pre>
</div>

<h3 id="docker-machine">Docker-Machine</h3>

<div class="highlighter-rouge"><pre class="highlight"><code><span class="c"># Switch to 80:80 or 4000:80 if you wish to use only Nginx with `jekyll build`</span>
docker run --rm --label<span class="o">=</span>jekyll --volume<span class="o">=</span><span class="k">$(</span><span class="nb">pwd</span><span class="k">)</span>:/srv/jekyll <span class="se">\</span>
  -it -p <span class="k">$(</span>docker-machine ip <span class="sb">`</span>docker-machine active<span class="sb">`</span><span class="k">)</span>:4000:4000 <span class="se">\</span>
    jekyll/jekyll
</code></pre>
</div>

So kramdown seems to do the correct thing in both cases. Maybe you can find out what is actually passed to kramdown as input?

[parkr]

@gettalong What is the Ruby code that the CLI produces? Jekyll uses the kramdown Ruby API, and it may be missing a configuration option or two that the kramdown CLI sets.

[envygeeks]

@gettalong @parkr I get the same broken result when running the command shown above with the source file, this is on Ubuntu 14.04 x64 Ruby 2.2.3p137, if you need more information:

ruby -r rbconfig -e 'puts RbConfig::CONFIG["configure_args"]'
 '--disable-install-doc' '--enable-shared' '--prefix=/usr/local'

ii  libffi-dev:amd64                                      3.1~rc1+r3.0.13-12ubuntu0.1
ii  libreadline6-dev:amd64                                6.3-4ubuntu2
ii  libssl-dev:amd64                                      1.0.1f-1ubuntu2.16
ii  libxml2-dev:amd64                                     2.9.1+dfsg1-3ubuntu4.5
ii  libxslt1-dev:amd64                                    1.1.28-2build1
ii  libyaml-dev:amd64                                     0.1.4-3ubuntu3.1

The result also happens on Alpine Linux 3.3 x64

[envygeeks]

@gettalong @parkr after I finish working on this Kramdown+Rouge segfault problem that keeps happening I'll jump into Pry and hit up our Kramdown processor and see exactly what content is passed, we did recently update it but... if it was a problem there I wouldn't get the same results when hitting Kramdown without Jekyll.

[gettalong]

@parkr See https://github.com/gettalong/kramdown/blob/master/bin/kramdown#L59, nothing really special.

@envygeeks I'm using kramdown 1.9.0 and ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-linux] on Ubuntu 15.04.

[gettalong]

@envygeeks Maybe you could try the following in an irb console:

require 'kramdown/document'
Kramdown::Document.new(File.read('/tmp/README.md'), input: :GFM, hard_wrap: false).to_html;
$".grep(/kramdown/).sort

The last line outputs all loaded kramdown source files. May this shows that some files are loaded from a wrong directory or something.

[envygeeks]

@parkr @gettalong I have feeling this is related to the bug in Rouge I'm currently working out, because it started working suddenly and then stopped working again, and stopped working again and then started again. At this point I'm going to assume they are related.

[benbalter]

For those interested, I posted some additional context behind our decision in this blog post.