tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Language models process text as sequences of numbers called tokens rather than as raw characters. Byte pair encoding is a method for converting text into these tokens while maintaining reversibility and lossless compression. tiktoken solves the problem of efficiently tokenising text for OpenAI models by implementing BPE as a compiled tokeniser that achieves between 3 and 6 times faster performance than comparable open source alternatives. The tool works on arbitrary text, including text not seen during training, and typically compresses text such that each token represents about 4 bytes on average.
Developers should choose tiktoken when working directly with OpenAI's models and need to count tokens, split text into token sequences, or understand how their input will be processed. The tool is particularly suited to applications that require accurate token accounting for cost estimation or context window management. The project includes an educational submodule for learning about BPE mechanics and supports extending the tokeniser with custom encodings through both direct instantiation and a plugin mechanism.
The project maintains a substantial base of adopters who report real-world issues through the tracker, indicating active production use. Maintainers typically respond to new issues and pull requests within a few days.
