KAI Scheduler is an open source Kubernetes-native scheduler written in Go that specializes in optimizing GPU resource allocation for AI and machine learning workloads at large scale. It is designed to manage large-scale GPU clusters containing thousands of nodes while handling high-throughput workloads, making it suitable for extensive and demanding environments. The scheduler allows Kubernetes cluster administrators to dynamically allocate GPU resources to workloads and can run alongside other schedulers already installed on a cluster.
The project supports the entire AI lifecycle, from small interactive jobs requiring minimal resources to large training and inference workloads, all within the same cluster. It ensures optimal resource allocation while maintaining resource fairness between different consumers.
KAI Scheduler includes comprehensive features for managing AI workloads. Batch scheduling ensures all pods in a group are scheduled simultaneously or not at all. The scheduler supports both bin packing and spread scheduling strategies to optimize node usage. It provides hierarchical queues with two-level queue hierarchies for flexible organizational control, along with workload priority management and separation of workload priority from preemptibility as independent policies. Resource distribution can be customized through quotas, over-quota weights, limits, and priorities per queue. Fairness policies implement Dominant Resource Fairness and resource reclamation across queues, while time-based fairshare ensures fair resource usage over time considering historical usage and time decay parameters.
Additional capabilities include workload consolidation to reduce fragmentation, elastic workloads that dynamically scale within defined thresholds, and GPU sharing that allows multiple workloads to efficiently share single or multiple GPUs. The scheduler supports Dynamic Resource Allocation through Kubernetes ResourceClaims for vendor-specific hardware like NVIDIA and AMD GPUs. Recent major features released in version 0.10.0 include Topology-Aware Scheduling for optimized placement, Hierarchical PodGroups for gang scheduling of multi-level workloads, and Time-based Fairshare. The scheduler is fully compatible with dynamic cloud infrastructures including auto-scalers like Karpenter as well as static on-premise deployments.
Recent integrations include native support for Ray workloads on Kubernetes and integration with Grove and Dynamo for orchestrating complex multi-component workloads like disaggregated serving and agentic pipelines. The repository maintains active development with enhancement requests being the dominant issue category, indicating ongoing feature development and community-driven improvements to the scheduler's capabilities.
