KAITO is a Kubernetes operator suite written in Go that automates the deployment and management of large language model inference, fine-tuning, and retrieval augmented generation services within Kubernetes clusters. The project provides a simplified interface for running vLLM-supported HuggingFace models on GPU infrastructure without requiring users to manually configure complex deployment parameters.
The core architecture centers on three primary Custom Resource Definitions. The Workspace CRD serves as the fundamental building block, allowing users to specify only a GPU instance type and HuggingFace model ID while the controller automatically estimates GPU memory requirements, calculates necessary GPU counts, and provisions nodes through integration with Karpenter APIs. The InferenceSet CRD manages multiple replicas of workspace instances for the same model and enables autoscaling based on inference request load through integration with KEDA autoscaler and vLLM metrics. The InferencePool CRD integrates with the Gateway API Inference Extension to support LLM-based routing and KVCache-aware request distribution.
KAITO distinguishes itself through several key technical features. It removes the need for detailed deployment parameters by providing optimized preset configurations for inference engine scheduling parameters including pipeline parallelism, data parallelism, tensor parallelism, and maximum model length. The node auto provisioner accurately estimates model memory requirements to determine optimal node counts for distributed inference. The system leverages GPU node built-in local NVMe storage for model storage, eliminating the need for external storage infrastructure. Support extends to any vLLM-supported HuggingFace model, with LoRA adapters and KVCache offloading enabled by default.
The project also includes a RAGEngine operator that streamlines retrieval augmented generation service management. This operator manages LLM endpoints, embedding services, and vector databases. It supports a built-in FAISS in-memory vector database by default, with optional persistent database support through Qdrant or Milvus. The RAGService component leverages LlamaIndex orchestration and provides APIs for document indexing, LLM call interception with automatic context retrieval, and hybrid search combining BM25 sparse retrieval with vector dense retrieval using the Reciprocal Rank Fusion algorithm.
KAITO shares contributors with related projects including openyurtio/openyurt, llm-d/llm-d, and istio/istio, indicating integration within a broader ecosystem of Kubernetes and AI infrastructure tools.
The project was first released on November 15th, 2023 as version 0.1.0, with the latest release being version 0.11.0 on July 1st, 2026. Installation is supported through both Helm and Terraform, with comprehensive documentation covering quick start guidance, autoscaling configuration, custom model deployment, CPU model support, and advanced features like prefill/decode disaggregation. The project welcomes contributions under the CNCF Contributor License Agreement and maintains an active community with weekly meetings and a dedicated Slack channel.
