Compile error with fresh MiKTeX installation

[Yggdrasil128]

Hello, I've recently ran into a problem that causes markdown to fail with the following error message:

...a/Local/Programs/MiKTeX/tex/luatex/markdown/markdown.lua:3237: bad argument #1 to 'open' (string expected, got nil)
stack traceback:
	[C]: in function 'io.open'
	...a/Local/Programs/MiKTeX/tex/luatex/markdown/markdown.lua:3237: in function <...a/Local/Programs/MiKTeX/tex/luatex/markdown/markdown.lua:3235>
	...a/Local/Programs/MiKTeX/tex/luatex/markdown/markdown.lua:3235: in main chunk
	[C]: in function 'require'
	[\directlua]:1: in main chunk.
\lua_now:e #1->\__lua_now:n {#1}
                                
l.8 \end{markdown}
                
The lua interpreter ran into a problem, so the
remainder of this lua chunk will be ignored.

Here are my steps to reproduce:

1. Create a fresh Windows 11 VM

2. Install the most recent version of MiKTeX

3. Create file document.tex

   \documentclass{article}
   \usepackage{markdown}
   \begin{document}
   Hello World!
   
   \begin{markdown}
   _Hello,_ **Markdown**!
   \end{markdown}
   \end{document}

4. Run lualatex --interaction=nonstopmode --shell-escape --synctex=1 document.tex

5. The above error occurs.

In the resulting PDF, only the "Hello World" part is there, the "Hello Markdown" part is missing.
I have attached the full log file: document.log

Since everything is a fresh install, all packages are up-to-date.

Interestingly, on a different system where I haven't updated my MiKTeX packages in a while,
I still have markdown version 2.23.0-0-g0b22f91 installed, and everything works fine there.

Please let me know if I can help you with any additional information.

[Witiko]

Hello and thank you for the report. I don't have any fix for you at the moment but I will look into reproducing your issue.

[Witiko]

I installed Windows 11 in a VM. In it, I installed MikTeX by running file basic-miktex-24.1-x64.exe from the MikTeX web site:

Then, I updated MikTeX from the MikTeX console:

Next, I created a file document.tex with @Yggdrasil128's source code from #458 (comment) and I ran your command:

Note

The --shell-ecape part of the command is only needed with engines other than LuaTeX such as pdfTeX and XeTeX.

After running the command, I was prompted to download the following packages:

• Package lua-uni-algos due to using file lua-uni-case.lua
• Package markdown due to using file markdown.sty
• Package lt3luabridge due to using file lt3luabridge.tex
• Package paralist due to using file paralist.sty
• Package csvsimple due to using file csvsimple.sty
• Package fancyvrb due to using file fancyvrb.sty
• Package ltxcmds due to using file ltxcmds.sty
• Package gobble due to using file gobble.sty

After downloading the packages, the compilation failed with the error from your original post:

Here is the part of the code of the Markdown package on lines 2336–2338 from file markdown.lua that causes the error:

markdown/markdown.dtx

Lines 26294 to 26296 in a9804bd

	local pathname = kpse.lookup("UnicodeData.txt")
	local file = assert(io.open(pathname, "r"),
	[[Could not open file "UnicodeData.txt"]])

Apparently, we fail to locate the file UnicodeData.txt from the package unicode-data even though the package has been installed:

This appears to be an issue with the KPathSea library in MikTeX. To demonstrate, I created the following LuaTeX script find-unicode-data.lua that should locate the file UnicodeData.txt:

local kpse = require("kpse")
kpse.set_program_name("luatex")
local pathname = kpse.lookup("UnicodeData.txt")
if pathname == nil then
  print("Found no files.")
else
  print("Found file `" .. pathname .. "`.")
end

Then, I ran the command texlua find-unicode-data.lua on a recent TeX Live installation. This produced the following output:

Found file /usr/local/texlive/2023/texmf-dist/tex/generic/unicode-data/UnicodeData.txt.

Finally, I ran the same command on the MikTeX installation. This produced the following output:

Found no files.

This issue extends beyond just the Lua API. I also ran the command kpsewhich UnicodeData.txt on a recent TeX Live installation. This produced the following output:

/usr/local/texlive/2023/texmf-dist/tex/generic/unicode-data/UnicodeData.txt

By contrast, the same command produced no output on the MikTeX installation.

[Witiko]

@Yggdrasil128 After my review from #458 (comment), this issue seems unrelated to the Markdown package. In MiKTeX/miktex#1515, I escalated the bug report upstream to see what the MikTeX developers have to say about it.

[Yggdrasil128]

Very interesting, thank you for the detailed analysis and for opening the upstream bug report! I'll be watching that issue as well.

[Witiko]

Even without an upstream fix, we should still produce better error messages when KPathSea has failed to locate a file.

[Witiko]

@Yggdrasil128 In the meantime, I downloaded the file UnicodeData.txt and placed it in the directory with your example document document.tex. Then, I tried to compile the document. However, regardless of whether I compiled the document with LuaLaTeX or pdfLaTeX, the process crashed. I only have ~3G free RAM in the VM, so that could be a part of the issue. Since we don't run automated tests on Windows, it could also be related to the platform. Regardless, more investigation seems necessary. If you have the time and inclination, I will appreciate you trying this out as well.

[Yggdrasil128]

Of course, I'll give it a try and get back to you tomorrow.

[Yggdrasil128]

I have tested it again on a fresh Windows 11 VM, with the same steps as in my initial comment, but with the additional UnicodeData.txt file. Unfortunately I ran into the same problem you described yesterday. I took this screenshot right after the process crashed.

I have attached the log file: document.log

I then tested it again on anther VM with twice the assigned memory (16 GB total), but with the same result.
I'm fairly confident in saying that this is not an insufficient memory issue.

Please let me know if I can test additional things for you, I'm happy to help you investigate this problem.

[Witiko]

Please let me know if I can test additional things for you, I'm happy to help you investigate this problem.

Thanks! Here are some steps of preliminary analysis that I would take:

1. By default, we load many different LaTeX packages to provide the default look for many markdown elements. To exclude them as potential causes, I would first load the Markdown package with the option [plain]:

   \usepackage[plain]{markdown}

   If the issue is with the LaTeX packages, the compilation should now succeed. This seems unlikely to me.

2. If using option [plain] does not help, the issue is likely with Lua code. To confirm this, I would create an example Markdown document example.md, locate the script markdown-cli.lua, and execute the script. These would be the steps on Linux:

   $ cat > example.md <<EOF
   _Hello,_ **Markdown**!
   EOF

   $ kpsewhich markdown-cli.lua
   /usr/local/texlive/texmf-local/scripts/markdown/markdown-cli.lua

   $ texlua /usr/local/texlive/texmf-local/scripts/markdown/markdown-cli.lua example.md
   \markdownRendererDocumentBegin
   \markdownRendererEmphasis{Hello,} \markdownRendererStrongEmphasis{Markdown}!\markdownRendererDocumentEnd

   If the issue is with the Lua code, the execution should crash. This seems likely to me.

If you could try these steps and let me know what the results are, that would be helpful.

[Yggdrasil128]

With the [plain] option, I got the same result:

I then ran the markdown-cli.lua script with the example.md file. Afer a few seconds, the script exited with code -1073741571 and produced no output.

[Witiko]

Thanks! The next step would be to make a copy of file markdown-cli.lua and add print statements throughout method M.new() to locate the place where the script crashes. This is very likely to be the lines M.writer.new(options) (less likely) or M.reader.new(writer, options) (more likely) that initialize the entire parser. If this turns out to be the case, I would also add print statements throughout these methods to locate the exact place where the script crashes.

In summary, here are the next questions that would be helpful to answer:

• Does the script crash in function M.new()?
• If it does, where in function M.new() does the script crash?
• If in M.writer.new() or M.reader.new(), where in these functions does the script crash?

[Yggdrasil128]

I copied the markdown-cli.lua and markdown.lua files into the test directory, and added some print(debug.getinfo(1).short_src .. ":" .. debug.getinfo(1).currentline) statements here and there (they output the current filename and line number).

Looks like it doesn't crash in the M.new function, but in the convert function that it returns. I'm continuing to investigate where exactly the crash occurs.

[Witiko]

How interesting! We initialize most data structures in functions M.writer.new() and M.reader.new(). I can't imagine what could cause a crash during the conversion, especially in the absence of any identifiers that might need Unicode normalization.

[Witiko]

This may be quite difficult to debug, since LPEG processes the whole input at once. Unless the issue is with other steps taken by the conversion function, removing different parts of the Markdown grammar from the table walkable_grammar may be the simplest way to determine which part of the grammar causes the issues.

[Yggdrasil128]

Thanks for the pointers!
I found out that the crash has something to do with parsers.blocks, but I was stuck because that seems to be a LPEG table capture, which I am not familiar with.

However, with your approach, I determined that I only need to disable the Inline -> LinkAndEmph part of the grammar, and then it no longer crashes, although it obviously doesn't recognize the **Markdown** part as bold anymore.

[Witiko]

However, with your approach, I determined that I only need to disable the Inline -> LinkAndEmph part of the grammar, and then it no longer crashes, although it obviously doesn't recognize the **Markdown** part as bold anymore.

Interesting. The LinkAndEmph parser was added by @lostenderman. If you'd like to dive into it and try to zero in on the exact spot in the code that is causing the issue, they may have some ideas about where to look first. Independently, I'd like to try and reproduce the issue with the MikTeX docker image for Linux to see if the issue only manifests on Windows.

After we have pinpointed and fixed the issue, I would like to better document the installation and the usage of the Markdown package in MikTeX and add MikTeX (and perhaps Windows) to the continuous integration pipeline, so that we can ensure that such issues don't go unnoticed in the future. This may need some ingenuity, since we don't have many self-hosted runners and the full regression tests are slow on nodes with few CPU cores. Similarly, I would like to better document and test the Markdown package with ConTeXt standalone, as discussed with @andreiborisov in #402.

[Yggdrasil128]

I dug a bit deeper into this. I disabled various parsers (by setting them to P(false)) and worked my way upwards the whole parser chain to see what causes the crash. And I think I found it.

The crash occurs in the check_unicode_type function. Specifically, in the line that reads
if lpeg.match(parsers.punctuation[char_length], s, i+pos) then.

As you can see in the console output, line 4982 got hit, but neither did 4987 nor 4990.

Earlier in the code, the parsers.punctuation table is loaded from the UnicodeData.txt file. Could there be something wrong with this file or how it is read under Windows?
I downloaded the file from https://ctan.joethei.xyz/macros/generic/unicode-data/UnicodeData.txt and double-checked that it is intact.

[Witiko]

Independently, I'd like to try and reproduce the issue with the MikTeX docker image for Linux to see if the issue only manifests on Windows.

Could there be something wrong with this file or how it is read under Windows?

I confirmed that the issue is with MikTeX under Windows:

$ docker run -it -e MIKTEX_UID=`id -u` miktex/miktex:basic

miktex@9da1b10ba72b:/miktex/work$ curl https://mirrors.ctan.org/macros/generic/unicode-data/UnicodeData.txt

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>307 Temporary Redirect</title>
</head><body>
<h1>Temporary Redirect</h1>
<p>The document has moved <a href="https://mirrors.nic.cz/tex-archive/macros/generic/unicode-data/UnicodeData.txt">here</a>.</p>
</body></html>

miktex@9da1b10ba72b:/miktex/work$ curl https://mirrors.nic.cz/tex-archive/macros/generic/unicode-data/UnicodeData.txt > UnicodeData.txt

miktex@9da1b10ba72b:/miktex/work$ cat > example.md <<EOF
_Hello,_ **Markdown**!
EOF

miktex@9da1b10ba72b:/miktex/work$ texlua `kpsewhich markdown-cli.lua` example.md

\markdownRendererDocumentBegin
\markdownRendererEmphasis{Hello,} \markdownRendererStrongEmphasis{Markdown}!\markdownRendererDocumentEnd

I will now look into whether the issue is also present with TeX Live under Windows.

[Witiko]

I will now look into whether the issue is also present with TeX Live under Windows.

I failed to reproduce the issue with TeX Live 2024 on the same Windows VM:

This indicates that the issue is specific to just MikTeX under Windows and therefore lower-priority than if the issue affected Windows in general. However, the issue is intriguing and we are close to discovering the exact cause thanks to @Yggdrasil128's detailed analysis in #458 (comment).

[Witiko]

Specifically, in the line that reads
if lpeg.match(parsers.punctuation[char_length], s, i+pos) then.

More specifically, the issue seems to be with the parsers parsers.punctuation[char_length], where char_length >= 3:

To see if there is a difference in the table parsers.punctuation between MikTeX on Windows and elsewhere, I updated the code that produces the table, so that it also produces the files parsers.punctuation.N.txt with a text representation of the parsers parsers.punctuation[N] where N = 1, 2, 3, 4:

Then, I produced the files by running the command texlua markdown-cli.lua example.md:

MikTeX on Windows

parsers.punctuation.1.txt
parsers.punctuation.2.txt
parsers.punctuation.3.txt
parsers.punctuation.4.txt

TeX Live 2024 on Linux

parsers.punctuation.1.txt
parsers.punctuation.2.txt
parsers.punctuation.3.txt
parsers.punctuation.4.txt

After replacing the Windows end-of-line characters \r\n with the Unix end-of-line characters \n using the dos2unix program, the only difference between the produced files was a couple extra codepoints in parsers.punctuation.3.txt, likely due to a never version of file UnicodeData.txt that I used with MikTeX:

$ dos2unix windows/parsers.punctuation.*.txt

dos2unix: converting file windows/parsers.punctuation.1.txt to Unix format...
dos2unix: converting file windows/parsers.punctuation.2.txt to Unix format...
dos2unix: converting file windows/parsers.punctuation.3.txt to Unix format...
dos2unix: converting file windows/parsers.punctuation.4.txt to Unix format...

$ diff linux/parsers.punctuation.1.txt windows/parsers.punctuation.1.txt
$ diff linux/parsers.punctuation.2.txt windows/parsers.punctuation.2.txt
$ diff linux/parsers.punctuation.3.txt windows/parsers.punctuation.3.txt

3390a3391,3394
> U+2FFC = parsers.succeed * S(E2) * S(BF) * S(BC)
> U+2FFD = parsers.succeed * S(E2) * S(BF) * S(BD)
> U+2FFE = parsers.succeed * S(E2) * S(BF) * S(BE)
> U+2FFF = parsers.succeed * S(E2) * S(BF) * S(BF)
3477a3482
> U+31EF = parsers.succeed * S(E3) * S(87) * S(AF)

$ diff linux/parsers.punctuation.4.txt windows/parsers.punctuation.4.txt

At this point, it seems to me that LuaTeX in MikTeX just has trouble with lpeg.S(byte), where byte = string.char(tonumber(byte_text, 16)) for byte_text larger than DF (223).

Now, DFFF is the highest (surrogate) code point that can be encoded in UTF-16, which is the internal encoding of wide text in Windows, so we may be hitting some internal platform-specific limitation of LPEG, although it's still unclear to me why, since we are explicitly working with bytes and UTF-8. 🤷 Regardless, LPEG in the Windows version of TeX Live does not seem to have the same issue, so this seems as a case of a badly compiled LuaTeX. Let's create a minimal working example (MWE) and open a ticket upstream, unless you want to diagnose this in more detail, @Yggdrasil128?

[Witiko]

At this point, it seems to me that LuaTeX in MikTeX just has trouble with lpeg.S(byte), where byte = string.char(tonumber(byte_text, 16)) for byte_text larger than DF (223).

[...] Let's create a minimal working example (MWE) and open a ticket upstream, unless you want to diagnose this in more detail, @Yggdrasil128?

The issue seems to be somewhat more specific, though, since I can't seem to come up with an MWE that would crash LuaTeX:

It could also just be the size of the parser. Whereas parsers.punctuation.1.txt has just 1K in size and parsers.punctuation.2.txt has just 5K in size, parsers.punctuation.3.txt has 206K in size and parsers.punctuation.4.txt has 234K in size. If this is the issue, then, in addition to opening a ticket upsteeam, we could easily work around it by chunking the parsers into several smaller parsers. This would incur some speed penalty, hopefully insignificant.

[Witiko]

It could also just be the size of the parser. Whereas parsers.punctuation.1.txt has just 1K in size and parsers.punctuation.2.txt has just 5K in size, parsers.punctuation.3.txt has 206K in size and parsers.punctuation.4.txt has 234K in size.

parsers.punctuation.4.txt contains 4,143 lines, i.e. the corresponding parser concatenates ca 16,000 lpeg.S() parsers. I updated the MWE to try and reproduce the issue but I failed:

However, it took a second or two time to concatenate 16,000 lpeg.S() parsers, so we may have a strong contender for the cause of the slowdown described in ticket #474.

[Witiko]

@Yggdrasil128: If you have the time and inclination, I will appreciate help with trying to construct a minimal working example that reproduces the issue. I tried in #458 (comment) and #458 (comment) but so far, I failed to reproduce the issue.

[Yggdrasil128]

Sorry for the long radio silence, I'm currently a bit busy, but I will look into it again in the next days.

[Yggdrasil128]

I've created a smaller reproducing file that just loads the punctuation parsers from the UnicodeData.txt file and then applies the last one (#code == 4) to the string _Hello,_ **Markdown**!:

local lpeg = require("lpeg")
local unicode = require("unicode")

print(debug.getinfo(1).short_src .. ":" .. debug.getinfo(1).currentline)

parsers = {}
parsers.punctuation = {}
parsers.punctuation_size = {}

(function()
  local file = assert(io.open("UnicodeData.txt", "r"),
    [[Could not open file "UnicodeData.txt"]])
  local lineNumber = 0
  for line in file:lines() do
    lineNumber = lineNumber + 1
    --if lineNumber % 2 == 0 then goto continue end
    
    local codepoint, major_category = line:match("^(%x+);[^;]*;(%a)")
    if major_category == "P" or major_category == "S" then
      local code = unicode.utf8.char(tonumber(codepoint, 16))
      if parsers.punctuation[#code] == nil then
        parsers.punctuation[#code] = lpeg.P(false)
      end
      
      --[-[
      local code_parser = lpeg.P(true)
      for i = 1, #code do
        local byte = code:sub(i, i)
        local byte_parser = lpeg.S(byte)
        code_parser = code_parser
                    * byte_parser
      end
      --]]
      -- Does this replacement work?
      --local code_parser = lpeg.P(code)
      
      parsers.punctuation[#code] = parsers.punctuation[#code] + code_parser
      
      parsers.punctuation_size[#code] = (parsers.punctuation_size[#code] or 0) + 1
    end
    
    ::continue::
  end
  assert(file:close())
end)()

print(debug.getinfo(1).short_src .. ":" .. debug.getinfo(1).currentline)

print("Parser sizes, " .. "1: " .. parsers.punctuation_size[1] .. ", 2: " .. parsers.punctuation_size[2] .. ", 3: " .. parsers.punctuation_size[3] .. ", 4: " .. parsers.punctuation_size[4])

local s = "_Hello,_ **Markdown**!"
local result = lpeg.match(parsers.punctuation[4], s, 1)

print(debug.getinfo(1).short_src .. ":" .. debug.getinfo(1).currentline)
print("result:")
print(result)

And it crashes at lpeg.match:

C:\Users\user\Documents\test>texlua test.lua
test.lua:4
test.lua:47
Parser sizes, 1: 32, 2: 133, 3: 4309, 4: 4143

But if I uncomment line 16, which causes all even-numbered lines from the UnicodeData.txt file to be skipped, it works:

C:\Users\user\Documents\test>texlua test.lua
test.lua:4
test.lua:47
Parser sizes, 1: 16, 2: 60, 3: 2151, 4: 2065
test.lua:54
result:
nil

And if I change the 0 to a 1, so that odd-numbered lines are skipped instead, it also works:

C:\Users\user\Documents\test>texlua test.lua
test.lua:4
test.lua:47
Parser sizes, 1: 16, 2: 73, 3: 2158, 4: 2078
test.lua:54
result:
nil

This leads me to believe that the issue is not with any particular one of these parsers, but with the sheer number of parsers that are concatenated together.
I also tried setting lpeg.setmaxstack(1000000) but that didn't help.

Side note
As I've already said, my knowledge/experience regarding LPEG is very limited, so please correct me if I'm wrong, but shouldn't it be possible to replace this part of the code

local code_parser = lpeg.P(true)
for i = 1, #code do
  local byte = code:sub(i, i)
  local byte_parser = lpeg.S(byte)
  code_parser = code_parser
              * byte_parser
end

with just local code_parser = lpeg.P(code)? Or is there some gotcha that I'm missing?

Anyways, I then looked at your PR #482 and noticed that you build the parsers a bit differently there, so this doesn't apply anymore.

I copied your changes into the script and ran it again:

And not only did it work right out of the box, it was also significantly faster.

Finally, I plugged your changes into my copy of markdown.lua and ran texlua markdown-cli.lua example.md again:

Which worked as well!

So thank you very much, I think this highly intriguing issue is solved with your PR. :)

Well, apart from MiKTeX/miktex#1515 that is.
But for that issue, at least there is a clear and easy workaround.

[Witiko]

Thanks for your analysis. I hoped that #482 would also help here besides improving the speed.

So thank you very much, I think this highly intriguing issue is solved with your PR. :)

Please note that #482 is not yet finished, see the failing CI. For some reason, the parser does not match some punctuation symbols such as "€", likely due to some obscure typo in my depth-first search algorithm.

Well, apart from MiKTeX/miktex#1515 that is.
But for that issue, at least there is a clear and easy workaround.

As also discussed in #482, I would like to remove the dependency on UnicodeData.txt and precompile the parser into a Lua file markdown-unicode-data.lua that would be distributed together with the Markdown package. That way, we should be fully compatible with MikTeX.

[Witiko]

Furthermore, this would also make the package easier to use with other distros that may not contain UnicodeData.txt such as ConTeXt Standalone, as discussed with @andreiborisov in #402.

[Witiko]

I expect to close this ticket in the next month's release after closing ticket #486.

[Witiko]

As also discussed in #482, I would like to remove the dependency on UnicodeData.txt and precompile the parser into a Lua file markdown-unicode-data.lua that would be distributed together with the Markdown package. That way, we should be fully compatible with MikTeX.

Important

In version 3.7.1 of the Markdown package, to be released by the end of September, there is a new file markdown-unicode-data.lua with a Lua parser of Unicode punctuation, see #499. This file removes the dependency on UnicodeData.txt, which makes life easier for users of distributions such as MikTeX (see also MiKTeX/miktex#1515) and ConTeXt standalone (see also #402). The file must be installed with the rest of the Markdown package, see user manual.