Package: sentencepiece 0.2.3

Jan Wijffels

sentencepiece: Text Tokenization using Byte Pair Encoding and Unigram Modelling

Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library <https://github.com/google/sentencepiece> which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) <doi:10.18653/v1/D18-2012>. Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) <http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf>.

Authors:Jan Wijffels [aut, cre, cph], BNOSAC [cph], Google Inc. [ctb, cph], Kenton Varda [ctb, cph], Sanjay Ghemawat [ctb, cph], Jeff Dean [ctb, cph], Laszlo Csomor [ctb, cph], Wink Saville [ctb, cph], Jim Meehan [ctb, cph], Chris Atenasio [ctb, cph], Jason Hsueh [ctb, cph], Anton Carver [ctb, cph], Maxim Lifantsev [ctb, cph], Susumu Yata [ctb, cph], Yuta Mori [ctb, cph], Benjamin Heinzerling [ctb, cph]

sentencepiece_0.2.3.tar.gz
sentencepiece_0.2.3.zip(r-4.5)sentencepiece_0.2.3.zip(r-4.4)sentencepiece_0.2.3.zip(r-4.3)
sentencepiece_0.2.3.tgz(r-4.4-x86_64)sentencepiece_0.2.3.tgz(r-4.4-arm64)sentencepiece_0.2.3.tgz(r-4.3-x86_64)sentencepiece_0.2.3.tgz(r-4.3-arm64)
sentencepiece_0.2.3.tar.gz(r-4.5-noble)sentencepiece_0.2.3.tar.gz(r-4.4-noble)
sentencepiece_0.2.3.tgz(r-4.4-emscripten)sentencepiece_0.2.3.tgz(r-4.3-emscripten)
sentencepiece.pdf |sentencepiece.html
sentencepiece/json (API)
NEWS

# Install 'sentencepiece' in R:
install.packages('sentencepiece', repos = c('https://bnosac.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/bnosac/sentencepiece/issues

Uses libs:
  • c++– GNU Standard C++ Library v3

On CRAN:

bytenatural-language-processingsentencepieceword-segmentation

4.10 score 25 stars 8 scripts 302 downloads 10 exports 1 dependencies

Last updated 2 years agofrom:95f86c692b. Checks:OK: 1 NOTE: 8. Indexed: yes.

TargetResultDate
Doc / VignettesOKNov 06 2024
R-4.5-win-x86_64NOTENov 06 2024
R-4.5-linux-x86_64NOTENov 06 2024
R-4.4-win-x86_64NOTENov 06 2024
R-4.4-mac-x86_64NOTENov 06 2024
R-4.4-mac-aarch64NOTENov 06 2024
R-4.3-win-x86_64NOTENov 06 2024
R-4.3-mac-x86_64NOTENov 06 2024
R-4.3-mac-aarch64NOTENov 06 2024

Exports:BPEembedBPEembedderread_word2vecsentencepiecesentencepiece_decodesentencepiece_download_modelsentencepiece_encodesentencepiece_load_modeltxt_remove_wordpiece_encode

Dependencies:Rcpp