[CS2] Literate CoffeeScript without dependencies (#4509)

* Reimplement `invertLiterate` without any dependency, tracking indentation levels (including inside lists); update literate test files to also check that no tests are skipped

* Drop Literate CoffeeScript’s support for executable code blocks inside list items (and also for second paragraphs or blockquotes in list items)

* Update Literate CoffeeScript docs to reflect current supported syntax

Cakefile

@@ -122,14 +122,6 @@ task 'build:browser', 'merge the built scripts into a single file for use in a b
     return #{fs.readFileSync "./package.json"};
   })();
   """
-  for {name, src} in [{name: 'markdown-it', src: 'dist/markdown-it.min.js'}]
-    code += """
-      require['#{name}'] = (function() {
-        var exports = {}, module = {exports: exports};
-        #{fs.readFileSync "node_modules/#{name}/#{src}"}
-        return module.exports;
-      })();
-    """
   for name in ['helpers', 'rewriter', 'lexer', 'parser', 'scope', 'nodes', 'sourcemap', 'coffeescript', 'browser']
     code += """
       require['./#{name}'] = (function() {

docs/v2/index.html

@@ -3009,8 +3009,14 @@ <h3><code>get</code> and <code>set</code> Keyword Shorthand Syntax</h3>
       </section>
       <section id="literate">
         <h2>Literate CoffeeScript</h2>
-<p>Besides being used as an ordinary programming language, CoffeeScript may also be written in “literate” mode. If you name your file with a <code>.litcoffee</code> extension, you can write it as a Markdown document — a document that also happens to be executable CoffeeScript code. The compiler will treat any indented blocks (Markdown’s way of indicating source code) as code, and ignore the rest as comments.</p>
+<p>Besides being used as an ordinary programming language, CoffeeScript may also be written in “literate” mode. If you name your file with a <code>.litcoffee</code> extension, you can write it as a Markdown document — a document that also happens to be executable CoffeeScript code. The compiler will treat any indented blocks (Markdown’s way of indicating source code) as executable code, and ignore the rest as comments. Code blocks must also be separated from comments by at least one blank line.</p>
 <p>Just for kicks, a little bit of the compiler is currently implemented in this fashion: See it <a href="https://gist.github.com/jashkenas/3fc3c1a8b1009c00d9df">as a document</a>, <a href="https://raw.githubusercontent.com/jashkenas/coffeescript/master/src/scope.litcoffee">raw</a>, and <a href="http://cl.ly/LxEu">properly highlighted in a text editor</a>.</p>
+<p>A few caveats:</p>
+<ul>
+<li>Code blocks need to maintain consistent indentation relative to each other. When the compiler parses your Literate CoffeeScript file, it first discards all the non-code block lines and then parses the remainder as a regular CoffeeScript file. Therefore the code blocks need to be written as if the comment lines don’t exist, with consistent indentation (including whether they are indented with tabs or spaces).</li>
+<li>Along those lines, code blocks within list items or blockquotes are not treated as executable code. Since list items and blockquotes imply their own indentation, it would be ambiguous how to treat indentation between successive code blocks when some are within these other blocks and some are not.</li>
+<li>List items can be at most only one paragraph long. The second paragraph of a list item would be indented after a blank line, and therefore indistinguishable from a code block.</li>
+</ul>
 
       </section>
       <section id="source-maps">

documentation/sections/literate.md

@@ -1,5 +1,11 @@
 ## Literate CoffeeScript
 
-Besides being used as an ordinary programming language, CoffeeScript may also be written in “literate” mode. If you name your file with a `.litcoffee` extension, you can write it as a Markdown document — a document that also happens to be executable CoffeeScript code. The compiler will treat any indented blocks (Markdown’s way of indicating source code) as code, and ignore the rest as comments.
+Besides being used as an ordinary programming language, CoffeeScript may also be written in “literate” mode. If you name your file with a `.litcoffee` extension, you can write it as a Markdown document — a document that also happens to be executable CoffeeScript code. The compiler will treat any indented blocks (Markdown’s way of indicating source code) as executable code, and ignore the rest as comments. Code blocks must also be separated from comments by at least one blank line.
 
 Just for kicks, a little bit of the compiler is currently implemented in this fashion: See it [as a document](https://gist.github.com/jashkenas/3fc3c1a8b1009c00d9df), [raw](https://raw.githubusercontent.com/jashkenas/coffeescript/master/src/scope.litcoffee), and [properly highlighted in a text editor](http://cl.ly/LxEu).
+
+A few caveats:
+
+* Code blocks need to maintain consistent indentation relative to each other. When the compiler parses your Literate CoffeeScript file, it first discards all the non-code block lines and then parses the remainder as a regular CoffeeScript file. Therefore the code blocks need to be written as if the comment lines don’t exist, with consistent indentation (including whether they are indented with tabs or spaces).
+* Along those lines, code blocks within list items or blockquotes are not treated as executable code. Since list items and blockquotes imply their own indentation, it would be ambiguous how to treat indentation between successive code blocks when some are within these other blocks and some are not.
+* List items can be at most only one paragraph long. The second paragraph of a list item would be indented after a blank line, and therefore indistinguishable from a code block.

package.json

@@ -43,9 +43,8 @@
     "google-closure-compiler-js": "^20170409.0.0",
     "highlight.js": "~9.10.0",
     "jison": ">=0.4.17",
+    "markdown-it": "^8.3.1",
     "underscore": "~1.8.3"
   },
-  "dependencies": {
-    "markdown-it": "^8.3.1"
-  }
+  "dependencies": {}
 }

src/helpers.coffee

@@ -2,8 +2,6 @@
 # the **Lexer**, **Rewriter**, and the **Nodes**. Merge objects, flatten
 # arrays, count characters, that sort of thing.
 
-md = require('markdown-it')()
-
 # Peek at the beginning of a given string to see if it matches a sequence.
 exports.starts = (string, literal, start) ->
   literal is string.substr start, literal.length
@@ -69,20 +67,34 @@ exports.some = Array::some ? (fn) ->
   return true for e in this when fn e
   false
 
-# Simple function for extracting code from Literate CoffeeScript by stripping
+# Helper function for extracting code from Literate CoffeeScript by stripping
 # out all non-code blocks, producing a string of CoffeeScript code that can
-# be compiled “normally.” Uses [MarkdownIt](https://markdown-it.github.io/)
-# to tell the difference between Markdown and code blocks.
+# be compiled “normally.”
 exports.invertLiterate = (code) ->
   out = []
-  md.renderer.rules =
-    # Delete all other rules, since all we want are the code blocks.
-    code_block: (tokens, idx, options, env, slf) ->
-      startLine = tokens[idx].map[0]
-      lines = tokens[idx].content.split '\n'
-      for line, i in lines
-        out[startLine + i] = line
-  md.render code
+  blankLine = /^\s*$/
+  indented = /^[\t ]/
+  listItemStart = /// ^
+    (?:\t?|\ {0,3})   # Up to one tab, or up to three spaces, or neither;
+    (?:
+      [\*\-\+] |      # followed by `*`, `-` or `+`;
+      [0-9]{1,9}\.    # or by an integer up to 9 digits long, followed by a period;
+    )
+    [\ \t]            # followed by a space or a tab.
+  ///
+  insideComment = no
+  for line in code.split('\n')
+    if blankLine.test(line)
+      insideComment = no
+      out.push line
+    else if insideComment or listItemStart.test(line)
+      insideComment = yes
+      out.push "# #{line}"
+    else if not insideComment and indented.test(line)
+      out.push line
+    else
+      insideComment = yes
+      out.push "# #{line}"
   out.join '\n'
 
 # Merge two jison-style location data objects together.

test/literate.litcoffee

@@ -2,8 +2,11 @@
 
 comment comment
 
+    testsCount = 0 # Track the number of tests run in this file, to make sure they all run
+
     test "basic literate CoffeeScript parsing", ->
       ok yes
+      testsCount++
 
 now with a...
 
@@ -16,6 +19,7 @@ now with a...
 ... nested block.
 
         ok yes
+        testsCount++
 
 Code must be separated from text by a blank line.
 
@@ -25,6 +29,7 @@ The next line is part of the text and will not be executed.
       fail()
 
       ok yes
+      testsCount++
 
 Code in `backticks is not parsed` and...
 
@@ -38,6 +43,7 @@ Code in `backticks is not parsed` and...
           ###
 
           ok yes
+          testsCount++
 
 Regular [Markdown](http://example.com/markdown) features, like links
 and unordered lists, are fine:
@@ -50,11 +56,6 @@ and unordered lists, are fine:
 
   * List
 
-Tabs work too:
-
-	test "tabbed code", ->
-		ok yes
-
 ---
 
     # keep track of whether code blocks are executed or not
@@ -69,13 +70,15 @@ if true
 
     test "should ignore code blocks inside HTML", ->
       eq executed, false
+      testsCount++
 
 ---
 
-*   A list item with a code block:
+*   A list item followed by a code block:
 
-        test "basic literate CoffeeScript parsing", ->
-          ok yes
+    test "basic literate CoffeeScript parsing", ->
+      ok yes
+      testsCount++
 
 ---
 
@@ -88,37 +91,6 @@ if true
 
 ---
 
-1.  This is a list item with two paragraphs. Lorem ipsum dolor
-    sit amet, consectetuer adipiscing elit. Aliquam hendrerit
-    mi posuere lectus.
-
-    Vestibulum enim wisi, viverra nec, fringilla in, laoreet
-    vitae, risus. Donec sit amet nisl. Aliquam semper ipsum
-    sit amet velit.
-
-2.  Suspendisse id sem consectetuer libero luctus adipiscing.
-
----
-
-1.  This is a list item with two paragraphs. Lorem ipsum dolor
-    sit amet, consectetuer adipiscing elit. Aliquam hendrerit
-    mi posuere lectus.
-
-    Vestibulum enim wisi, viverra nec, fringilla in, laoreet
-    vitae, risus. Donec sit amet nisl. Aliquam semper ipsum
-    sit amet velit.
-
-2.  Suspendisse id sem consectetuer libero luctus adipiscing.
-
----
-
-*   A list item with a blockquote:
-
-    > This is a blockquote
-    > inside a list item.
-
----
-
 This is [an example][id] reference-style link.
 [id]: http://example.com/  "Optional Title Here"
 
@@ -128,12 +100,13 @@ This is [an example][id] reference-style link.
 
 1986. What a great season.
 
-          executed = yes
+    executed = yes
 
 and test...
 
     test "should recognize indented code blocks in lists with empty line as separator", ->
       ok executed
+      testsCount++
 
 ---
 
@@ -146,6 +119,7 @@ and test...
 
     test "should ignore indented code in escaped list like number", ->
       eq executed, no
+      testsCount++
 
 one last test!
 
@@ -155,3 +129,9 @@ one last test!
            and bar!
       '''
       eq quote, 'foo\n   and bar!'
+      testsCount++
+
+and finally, how did we do?
+
+    test "all spaced literate CoffeeScript tests executed", ->
+      eq testsCount, 9