Package: sentencepiece 0.2.5

sentencepiece: Text Tokenization using Byte Pair Encoding and Unigram Modelling

Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library <https://github.com/google/sentencepiece> which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) <doi:10.18653/v1/D18-2012>. Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) <http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf>.

Authors:Jan Wijffels [aut, cre, cph], BNOSAC [cph], Google Inc. [ctb, cph], Kenton Varda [ctb, cph], Sanjay Ghemawat [ctb, cph], Jeff Dean [ctb, cph], Laszlo Csomor [ctb, cph], Wink Saville [ctb, cph], Jim Meehan [ctb, cph], Chris Atenasio [ctb, cph], Jason Hsueh [ctb, cph], Anton Carver [ctb, cph], Maxim Lifantsev [ctb, cph], Susumu Yata [ctb, cph], Yuta Mori [ctb, cph], Benjamin Heinzerling [ctb, cph]