Repository Issue Activity (beta)

#1663 Inconsistent behaviour of `PreTrainedTokenizerFast`s on diacritics marked texts

#2045 Publish error in npm repository, Node.js binary bindings

#1784 Add cp314t tests, mark extension modules as compatible, and ship free-threaded wheels

#1975 [RFC] Korean Tokenization: Jamo Decomposition as Pre-tokenizer (A Blueprint for Compositional Scripts)

#2334 `Precompiled` normalizer diverges from SentencePiece: shortest-prefix match + discarded remainder silently drops codepoints

#2320 BpeTrainer emits a merge for a pair that occurs zero times (negative pair_counts sign-extended by `as u64`)

#1581 [building on windows] onig_sys/oniguruma two or more data types in declaration specifiers

#1973 Make huggingface-hub an optional dependency (extras)

#2236 [BUG] | The docs listing page for tokenizers is throwing 404

#2225 [Feature/Security] Support trusted and untrusted special-token spans

#1791 SyntaxWarning: invalid escape sequence '\w'

#2210 EncodingVisualizer: data-* attributes built from token text are not HTML-escaped (XSS gap left by #1937)

#1407 How to add byte_fallback tokens?

#1902 Guide: Compiling `tokenizers` on Android/Termux

#2198 BPE `build()` panics on load for a crafted `tokenizer.json` (merge concatenation exceeds the scratch buffer)

#1817 Inconsistent encoding of unknown tokens when they have special tokens as prefixes in the WordLevel Tokenizer

#1877 encode bytes directly

#2109 Proposal: emit UserWarning from Split.pre_tokenize_str() when it produces zero matches (catches silent str vs Regex confusion)

#1954 Constructive criticism for Tokenizers library.

#2116 Precompiled normalizer uses grapheme-level shortest-prefix matching and diverges from SentencePiece on NFD input

#1369 `BPE` tokenization model does not respect custom `RegEx` via `Split` pre-tokenizer

#1918 do we have plan support java ，we need you

#1793 `end_of_word_suffix` is ignored

#1839 `encode_special_tokens` is not saved for loading

#2091 Release 0.23.1 introduces breaking change in Encoding.overflowing