Tokenizer plugins

[wjones127]

It won't be sustainable to statically compile tokenizers into Lance. It also would be annoying to require them all to have feature flags in Lance.

We currently use Tantivy for tokenizer, so it might make sense for us to re-use their API. There is some discussion on their side for using a plugin API. It might make sense for us to help with that and re-use that same mechanism. That way tokenizer providers don't have to do anything extra to support Lance if they are supporting tantivy.

Tantivy has a stable API for tokenizeres here: https://github.com/quickwit-oss/tantivy/blob/main/tokenizer-api/src/lib.rs

(It's a Rust API though. We might want to create a stable C API, which will be more cross compatible between compilers.)

quickwit-oss/tantivy-py#358
quickwit-oss/tantivy-py#200

[chenkovsky]

@wjones127 I created a C API draft. https://gist.github.com/chenkovsky/40a5e6078fe7857ba3a2c1b2c624b4b6 . if you think it's feasible. I will create a PR. I have done a PoC on this API.