Text Embeddings Inference is a toolkit for deploying and serving text embeddings and sequence classification models.
The tool addresses the need for high-performance inference when extracting embeddings from text. It achieves speed through optimized transformer implementations using Flash Attention, Candle, and cuBLASLt, combined with token-based dynamic batching. The service exposes embeddings via a REST API with Swagger documentation and also supports gRPC. It loads model weights from Safetensors or ONNX formats without requiring a model graph compilation step.
The project supports a range of embedding architectures including BERT, RoBERTa, JinaBERT, Mistral, GTE, Qwen, MPNet, ModernBERT, and Gemma3 models. It runs on NVIDIA GPUs, AMD Instinct GPUs via ROCm, and Apple Silicon via Metal, making it suitable for both cloud deployments and local development. The tool produces small Docker images with fast boot times, positioning it for serverless environments. It also supports re-ranker and sequence classification models alongside embeddings. For production use, it includes distributed tracing with Open Telemetry and Prometheus metrics.
The project maintains active development with regular updates to model support and infrastructure. The codebase is written in Rust, providing performance benefits for the inference workload. Documentation is comprehensive, covering deployment scenarios from local installation to air-gapped environments and private model access. The tool is actively maintained with examples and clear guidance for common use cases.
