Repository Issue Activity (beta)

#57 Support tokenizing with and without special tokens using the same `gl.Tokenizer` instance.

#56 Support converting from an `tiktoken.Encoding` instance

#51 Cargo.toml cannot be parsed on current nightly due to `profile.profiling.rustflags`

#47 Your project is on StackMap — a curated map of the AI stack

#55 byte_fallback=true with no <0xHH> pieces rejects a ByteLevel BPE tokenizer

#54 Less memmove in `gigatoken_rs::bpe_train::update_word`

#36 Unbounded cache growth

#53 Make Python bindings optional for Rust consumers

#52 Remove hardcoded data paths

#49 POWER8/ppc64le: mask scanner falls back to scalar; VSX tier in my fork gets 3.4x single thread

#50 discussion: exploring possible collaboration between gigatoken and ffbpe

#48 (accidental api probe, please ignore)

#44 HFCompat uses _token_to_id whilst Huggingface Tokenizers uses token_to_id

#33 Improve caveat at bottom of README

#45 Security: unsafe Hub cache paths and unchecked UTF-8 in native APIs

#35 Token-order mismatch vs HuggingFace on Qwen2.5-7B tokenizer for Unicode combining-mark ranges

#43 Better report issues due to Unicode version differences (HF outdated)

#40 Bug: pretokenizer selection in from_tiktoken always selects r50k

#38 Eliminate remaining allocator overhead for large documents

#39 Contribution access

#37 Unicode property version mismatch with HuggingFace tokenizers

#32 Suggestion: Add Fastokens in benchmark

#34 Standalone pretokenization for all supported schemes from Python (implementation ready)

#31 Fix o200k special-token handling in the tiktoken loader

#30 Publish the Rust core crate on crates.io with an MSRV