Do not end paragraph before ::: in fenced divs

[Witiko]

Closes #407.

Tasks

• Reapply commit dda4a43.
• Review the fix attempted in Do not end paragraph before ::: in fenced divs #427 and the fenced div parser using the approach of @Omikhleia:
  • Fancy lists, task list and bunch of pandoc extensions jgm/lunamark#36
  • Fenced div blocks nesting jgm/lunamark#44
  • refactor: fenced divs can be nested jgm/lunamark#46
• Update the parser, so that continuous integration passes.
• Update CHANGES.md.

[Witiko]

I pressed on using the original approach of @Omikhleia, which relied on buffering the contents of a fenced div before parsing them. However, I realized that buffering the body of a fenced div would have so many different corner cases just to properly handle nested fenced divs and fenced code blocks that this approach is not worth the effort. I stashed my changes here.

Instead, I mapped out the viable approach that we should implement. Our issue is with the handling of trailing div fences when the blankBeforeFencedDiv option has been disabled, which it is by default:

markdown/markdown.dtx

Lines 30960 to 30981 in 415379f

	% If the `blank_before_div_fence` parameter is `false`, we will have the
	% closing div at the beginning of a line break the current paragraph if
	% we are currently nested in a div.
	%
	% \end{markdown}
	% \begin{macrocode}
	local function check_div_level(s, i, current_level) -- luacheck: ignore s i
	current_level = tonumber(current_level)
	return current_level > 0
	end
	local is_inside_div = Cmt(Cb("div_level"), check_div_level)
	local fencestart = is_inside_div * fenced_div_end
	if not blank_before_div_fence then
	self.update_rule("EndlineExceptions", function(previous_pattern)
	if previous_pattern == nil then
	previous_pattern = parsers.EndlineExceptions
	end
	return previous_pattern + fencestart
	end)
	end

Namely, the pattern is_inside_div is not sufficient to decide whether we should break a paragraph before a line that looks like the trailing div fence. This is because the line may be nested in a block-level element and will not end a fenced div, see the example from #403 (comment):

::: {#some-identifier}
This is the beginning of a div

> This is a blockquote that contains three colons:
> :::

This is the end of a div
:::

Here, the line > ::: will break the paragraph but it will not end the fenced div. This causes us to incorrectly produce a paragraph break before the three colons rather than the soft line break that we would expect and that is produced when we set the option blankBeforeFencedDiv to true.

In #427, @lostenderman followed my suggestion and tried to fix the issue by tracking whether we are on the top level of a fenced div or nested in a block-level element using a named capture group is_top_level. This allowed us to strengthen the condition for breaking a paragraph by requiring that not only are we in a fenced div but we are on the top level of a fenced div.

However, this approach does not work well with nested fenced divs, which would require that the named capture group is a stack rather than a scalar, so that we can know whether we are still at the top level of a parent fenced div when we have emerged from a nested fenced div. More crucially, this approach does not fix the issue for trailing lines of blockquotes and other elements such as list items, which should be able to close a div but are technically still nested in a block-level element:

::: {#some-identifier}
> This is a blockquote followed by the end of a div:
:::

Therefore, we need a different approach.

A more relaxed condition for the closing div fence is that it has the same indent level as the opening div fence. Currently, we use the function increment_div_level() to store the current nesting level inside the named capture group div_level. This is awkward, because we keep converting between a string and an integer, and also limiting, because we can only store a scalar value efficiently. Instead, we may use a similar hack as @lostenderman used in #226 for the named capture group indent_info that carries an anonymous Lua table instead of a string. We can use the same technique to make the named capture group div_level from a string that stores an integer into an array of opening indent levels for each level of fenced div nesting. Then, we would only break a paragraph when the closing div fence would have the same indent level. We should be able to tease this information out of the named capture group indent_info.

[Witiko]

@lostenderman This feels like the conceptually correct approach. However, num_current_indents is currently still wrong for e.g. lazy continuation lines:

::: {#an-identifier}
> a
:::  <!-- Here, `num_current_indents` is still 1 but `num_opening_indents` is 0. -->

This is also corroborated by the CI, which fails on the single test case in fenced-divs.test that involves what-would-be lazy continuation lines if the fenced div syntax extension were disabled:

markdown/tests/testfiles/lunamark-markdown/fenced-divs.test

Lines 89 to 92 in 415379f

	::: {#another-identifier}
	> This is a blockquote immediately in a
	> div
	:::

I will appreciate your advice on refining the current approach. In the Endline pattern, we call the parsers.check_minimal_indent and parsers.check_optional_indent patterns before the EnclineExceptions pattern and these update indent_table.trail with information about the current line. Perhaps we could use indent_table.trail to better determine num_current_indents? I am still not comfortable around the indent-info / trail-info objects and how to interpret their properties and will appreciate a nudge in the right direction (or a commit).

[Witiko]

@lostenderman If you are busy, I will likely figure it out eventually at this point. However, you would likely be much more efficient at this, since the indent-info / trail-info objects are your code. I will appreciate your insight.

[Witiko]

Suggested change

	% Initialize a named group named `fenced_div_level` for tracking how deep
	% Initialize a named group `fenced_div_level` for tracking how deep