FoLiA-txt fails on control character

[proycon]

It seems FoLiA-txt manages to retain some control characters from the input and propagates it to the output text, leading to invalid XML. (reported by @martinreynaert in #45)

[proycon]

I'm adjusting this in libfolia itself, it will do stripping of all non-allowed control characters. This does technically violate the principle of not manipulating the incoming data, but I think it's justifiable in this case. Ending up with invalid XML as was the current situation would be worse. (the alternative would a hard failure, which is what foliapy does upon serialisation).

[kosloot]

IMHO this libfolia patch to exterminate control characters is using a sledgehammer to crack a nut.
A less intrusive solution would have been to adapt the 'normalize_spaces()' function to see a FORM_FEED as a space too.
for instance by using ICU::u_isWhitespace() instead of u_isspace()

NOTE: as such, Control-Characters are not forbidden in XML, provided they are encoded correctly. So isn't this a signal that something else is wrong?

[proycon]

I did adapt normalize_spaces(), that was my first try... but that function is only invoked as part of text validation, we don't force space normalization on all text input (and that's a good thing), only this control-char normalization is something we want to force.

NOTE: as such, Control-Characters are not forbidden in XML, provided they are encoded correctly. So isn't this a signal that something else is wrong?

They are? That wouldn't make much sense, I tried: <a>&#x0c;</a> and xmllint rightly complains:

/tmp/test.xml:1: parser error : xmlParseCharRef: invalid xmlChar value 12
<a>&#x0c;</a>

[kosloot]

Of course , i stand corrected. this only accepted for xml version 1.1 and upward.

You state that removing control characters is a forced thing, so I assume that NOT to be the default?
That is NOT what you have implemented atm? Which breaks the API, and in theory might break existing programs.

[proycon]

it IS the default now (because not removing them would lead to invalid xml). settext() calls strip_control_chars(). It shouldn't really effect the API much, other than that the String.empty() check prior to settext() should now ideally use is_norm_empty(). If not, there could be situations where things now break with a NoSuchText error if the original input consists of only control characters. But even then raising such an error is preferable to generating invalid XML anyway.