NEWS

---

sentencepiece 0.2.3 (2022-11-13)

• fix R CMD check warning due to change in version 0.2.2.
  • in third_party/protobuf-lite/strutil.cc:506:33: warning: argument to ‘sizeof’ in ‘int snprintf(char*, size_t, const char*, ...)’ call is the same expression as the destination; did you mean to provide an explicit length? [-Wsizeof-pointer-memaccess]
  • this part of third_party/protobuf-lite/strutil.cc was not used in sentencepiece

sentencepiece 0.2.2 (2022-11-09)

• use snprintf instead of sprintf to handle the R CMD check deprecating note on M1mac

sentencepiece 0.2.1 (2021-12-21)

• Fix for clang-UBSAN error

sentencepiece 0.2 (2021-12-14)

• Fix wordpiece bug for 1-character words. (@jonthegeek, #4)
• Upgraded to sentencepiece release v0.1.96

sentencepiece 0.1.3

• Fix wordpiece bug for 1-character words. (@jonthegeek, #4)
• Fix Solaris installation issue related to incorrect usage of pointer as a function
• Also download the binary model in sentencepiece_download_model as it can be loaded with word2vec::read.wordvectors
• read_word2vec now uses word2vec::read.wordvectors from word2vec >= 0.2.0
• added BPEembed and predict.BPEembed
• allow subword regularisation by adding nbest and alpha option in sentencepiece_encode and changed sentencepiece_decode accordingly
• Added txt_remove_
• Upgrade sentencepiece to release v0.1.91 commit a32d7dc6ce6f383a65ad6e1cbe1983f94ab11932 which has subword regularisation for BPE

sentencepiece 0.1.2 (2020-06-08)

• Fix Solaris installation issue which used log of uint64 which is not defined on Solaris

sentencepiece 0.1.1 (2020-06-04)

• Added verbose argument in sentencepiece

sentencepiece 0.1.0

• Initial package based on https://github.com/google/sentencepiece release v0.1.84 commit 2424d82d396b43b2556203c592e48a621ef10f3c
• Third-party code from https://github.com/google/sentencepiece/tree/master/third_party is put in src/absl, src/esaxx, src/darts_clone, src/protobuf-lite