[Question] How to break text into words?

[saona-raimundo]

First of all, thanks for the crate!!

I am looking into coding something like this ICU demo for computing ICU segments using icu4x in Rust, but I could not find how to use the icu::segmenter::WordBreakSegmenter in a locale-aware fashion.

Using unicode_segmentation::UnicodeSegmentation is pretty straight forward, but this only implements the Default Word Boundary Specification. I am looking into doing a similar job being locale-aware.

I know that icu::segmenter::WordBreakSegmenter is behind the experimental flag, but I was wondering if it is possible to do it today(?) Its constructor does not seem to be locale-aware (but maybe I am wrong)

As a benchmark, I would like to break this text in Primitive Irish (locale pgl, using the Ogham alphabet)

ᚁᚔ ᚇᚔᚂᚔᚄ ᚇᚒᚔᚈ ᚃᚓᚔᚅ

into words ᚁᚔ, ᚇᚔᚂᚔᚄ, ᚇᚒᚔᚈ, and ᚃᚓᚔᚅ.

Thanks beforehand!

[sffc]

Like ICU4C, ICU4X has dictionary-based segmentation for Chinese, Japanese, Korean, Thai, Burmese, Lao, and Khmer. It also has machine-learning-based segmentation for Thai, Burmese, Lao, and Khmer.

@makotokato @aethanyc Do we support pgl-Ogam segmentation?

[aethanyc]

@makotokato @aethanyc Do we support pgl-Ogam segmentation?

No, we don't. We currently only have the same set of dictionaries as ICU4C, so for pgl-Ogam segmentation, we'll use Default Word Boundary Specification in UAX29.