Adding decodeWithByteRanges: string_views into translated-text

[jerinphilip]

Attempting to solve browsermt/bergamot-translator#25. Might be useful for marian-nmt#775 (?)

Changes test to accommodate the decoded string_views as well. The square-brackets indicate annotations using string_views, constructed by src/tests/sentencepiece_norm.cpp. The test previously for encodeWithByteRanges is modified to include decodeWithByteRanges as well. Once the expected is set to the output, this regression-tests can be modified as well.

Input Text

Äffin
Henry IV
Ç
가
ℌℍ
①
︷
i⁹
¼

Output

Original: [Ä][ff][in]
Decoded: [Ä][ff][in]
Original: [Henry][ IV]
Decoded: [Henry][ IV]
Original: [Ç]
Decoded: [Ç]
Original: [][가]
Decoded: [][ ⁇ ]
Original: [ℌ][ℍ]
Decoded: [H][H]
Original: [①]
Decoded: [1]
Original: [][︷]
Decoded: [][ ⁇ ]
Original: [i][⁹]
Decoded: [i][9]
Original: [][¼]
Decoded: [][1⁄4]

Requesting review.

[kpu]

Is int the correct type here? Prefer the actual typedef.

[kpu]

This should be WordIndex

[kpu]

Ok you are right, it's std::vector in the SentencePiece source, sorry.