Set marian::bergamot::Response to work with token-ranges instead of sentence-ranges

[jerinphilip]

Source and target word byte ranges are required for Alignment, which seems to be important to the browser people. Alignment is proposed to be constructed from source byte-ranges to target-byte ranges.

For source, we construct sentence byte-ranges from source-token byte-ranges (ie, we already have source-word byte-ranges). However, translated text due to lack of API in marian-dev's vocabulary is stuck with operating at sentence level. The current source sets marian::bergamot::Response's operations to token-ranges capability, but is glued together with external browser API by assuming a long-translated sentence as a single-token, given the inability to extract token-level views at translation.

bergamot-translator/src/translator/response.cpp

Lines 58 to 66 in 45a8309

	// TODO(@jerinphilip):
	// Currently considers target tokens as whole text. Needs
	// to be further enhanced in marian-dev to extract alignments.
	for (auto &range : translationRanges) {
	std::vector<string_view> targetMappings;
	const char *begin = &translation_[range.first];
	targetMappings.emplace_back(begin, range.second);
	targetRanges_.push_back(std::move(targetMappings));
	}

An alternative to this is perhaps keeping sentence encoded and using the spaces to extract units, but this won't generalize to other vocabs which is are place in marian.