Repository Issue Activity (beta)

#1303 Intermittent abort in test_trainer_with_normalizer under pytest --parallel-threads 4 (free-threaded 3.14, riscv64)

#1308 Breaking change in 0.2.2: rejects loading models with null-byte pieces without migration path

#1250 riscv64 distribution?

#967 High frequency token segmented into letter sequence when input is a tsv file

#1297 2,303 unreachable "dead-code" vocabulary pieces in `spiece.model` (~7.2% embedding overhead)

#1290 OSS-Fuzz issue 536054465

#1238 Use upb instead of protobuf

#1288 Decode() with empty input crashes for return_type='proto' and returns str instead of bytes for bytes/'serialized_proto'

#1228 Deprecate differential privacy features.

#1257 Derepecates unused methods

#1281 OSS-Fuzz issue 532422553

#1009 Tokenization for phonetic languages

#1278 OSS-Fuzz issue 528684769

#366 can we train by Parallel Computing or Multithreading or multi-Progress

#1256 OSS-Fuzz issue 518026690

#866 Tokens Chunking to respect Language Word Boundaries

#1269 [Security] Out-of-bounds read when loading a malicious tokenizer model via the precompiled_charsmap trie

#1268 Migrate from SWIG to pybind11

#649 user defined char set

#1194 sentencepiece sample_encode_fuzzer: sanitize nbest_size/alpha to reduce degenerate paths and timeouts

#881 Duplicate tokens in BPE vocabulary

#1253 add --split_digits mode

#1264 Use Abseil in the public API.

#1229 Support parallel encoding

#1030 No typings in Python package