Multilang splitter

[oleksandrlukashov]

No description provided.

[urchade]

LGTM

[Ingvarstep]

Good job overall, I have a few suggestions on how to make it more flexible and efficient:

• I think it would be better to create a dictionary of splitters. If a new language appears, we can just add a new splitter;
• By default, the splitter for other languages is WhitespaceTokenSplitter, I propose to make it flexible, it can be spacy with language key - xx.
• Ideally, all tokenizer dependencies make optional, please, see how it was done for onnx-gpu.