RTF/CRE rework + small C extension to tokenize RTF

[benoit-pierre]

Summary of changes

This add a C extension to tokenize RTF (the parsing is still done in Python), this would bring the speed gains (with #1362, same dictionary) to ~5.3.

The problem: no more universal wheel for Plover! No universal wheel, mean installing from source unless a wheel for the correct platform / interpreter version is available, which means PyQt5 must be installed for building the UI. This would make installing Plover more difficult on things like a Raspberry Pi.

Possible fix: move the RTF/CRE support to a dedicated external plugin, like we did with Treal.

[user202729]

There remains an option to provide a pure Python plugin, and a C extension plugin. But that's more maintenance work.

Otherwise, I assume that you would remove the Python implementation (before the import)?

[benoit-pierre]

Or just make a small standalone rtf_tokenize package and depend on that (like for plover_stroke in #1362).

[benoit-pierre]

Updated to use a dedicated rtf_tokenize package with the C extension.

[gdwaynewarner]

Is this what caused my RTF dictionaries to break? I had to go back to an older version <https://github.com/openstenoproject/plover/releases/tag/v4.0.0.dev10> and use a backup of my RTF dictionaries. Here's the alerts that appeared during that incident: [image: plover_rtf_errors_09OCT21.png] It appears that something decided to remove the slash between the two strokes, leaving me with a lot of outlines that suddenly quit working. I meant to submit a bug report but wasn't sure what exactly happened.

[benoit-pierre]

Common @gdwaynewarner, you know the drill: create a dedicated PR with the relevant information (image is missing).

[gdwaynewarner]

Now that I have a bit more of a handle on what happened, I will do that here shortly!