Markdown parser gets multiple-underscores-inside-italics wrong

[ljw1004]

The markdown parser gets underscores-inside-italics wrong. For instance,

A *asterisk_one_two* B

On both GitHub and CommonMark, it's rendered as an italic with underscores:

A asterisk_one_two B

But FSharp.Formatting.Markdown parses it as a nested emphasis:

Emphasis [ Literal "asterisk", Emphasis [ Literal "one"], Literal "two"]

(Actually, to be honest, I don't know what is the correct behavior. I'm assuming that CommonMark is correct here.)

Here are some more examples:

A *asterisk_one_two* B

A *asterisk_one_two_three* B

A *asterisk_one_two_three_four* B

A _underscore_one_two_ B

A _underscore_one_two_three_ B

A _underscore_one_two_three_four_ B

A asterisk_one_two B

A asterisk_one_two_three B

A asterisk_one_two_three_four B

A underscore_one_two B

A underscore_one_two_three B

A underscore_one_two_three_four B

[ljw1004]

I wonder if this is the same as #376 ?

[matthid]

thanks for reporting, yes we have tests for the full spec (Im pretty sure for both issues you reported). If you need a near/mid term fix feel free to enable and fix them. There is not much that can go wrong.

[matthid]

If you need any help/pointers feel free to ask.

[tpetricek]

I think the behaviour was a bit unspecified before CommonMark.

With CommonMark, I think the rules are that there should be spaces if you want nesting:

this *is_just_underscores*
this *is _nested_ emphasis*

[ljw1004]

This might be related to #376

[tpetricek]

Yes, it is. In the previous issue, @amazingant points to the relevant bits of the CommonMark spec:

I can't find a part of the CommonMark spec that explicitly states it rather than just being implied by the left/right flanking delimiter rules, but emphasis with the underscore like some_words_here should be translated as some_words_here, but come out of FSharp.Markdown as some<em>words</em>here. Spec has a few examples of this; makes snake-case value names output wrong 😬

[deviousasti]

Has there been any resolution to this?
Working around this is a bit ugly.

Does DelimitedMarkdown need to be rewritten?
It looks like the spec has a lot of corner cases.

[abelbraaksma]

This is a rather unfortunate bug and as issue #2411 shows, leads to "real world" issues. In that particular scenario for Fantomas, the headers are the names of configuration tokens, which all have underscores, making the file as a whole hard to read.

I don't mind taking a stab at this sometime soon (no promises), perhaps @dsyme or @tpetricek know off-hand where things like this get parsed/rendered? Seems to me that the parser needs some look-ahead/behind to determine that _ has spaces around.

[nhirschey]

I believe it’s here

FSharp.Formatting/src/FSharp.Formatting.Markdown/MarkdownParser.fs

Lines 165 to 181 in a55c29b

	/// Recognizes some form of emphasis using `**bold**` or `*italic*`
	/// (both can be also marked using underscore).
	/// TODO: This does not handle nested emphasis well.
	let (|Emphasised|_|) =
	function
	| (('_'
	| '*') :: _tail) as input ->
	match input with
	| DelimitedMarkdown [ '_'; '_'; '_' ] (body, rest)
	| DelimitedMarkdown [ '*'; '*'; '*' ] (body, rest) ->
	Some(body, Emphasis >> List.singleton >> (fun s -> Strong(s, None)), rest)
	| DelimitedMarkdown [ '_'; '_' ] (body, rest)
	| DelimitedMarkdown [ '*'; '*' ] (body, rest) -> Some(body, Strong, rest)
	| DelimitedMarkdown [ '_' ] (body, rest)
	| DelimitedMarkdown [ '*' ] (body, rest) -> Some(body, Emphasis, rest)
	| _ -> None
	| _ -> None

[abelbraaksma]

Thanks for fixing this @nojaf and fr merging it in @dsyme!