KAI Scheduler is an open source Kubernetes-native scheduler written in Go that specializes in optimizing GPU resource allocation for AI and machine learning workloads at large scale. It is designed to manage large-scale GPU clusters containing thousands of nodes while handling high-throughput workloads, making it suitable for extensive and demanding environments. The scheduler allows Kubernetes cluster administrators to dynamically allocate GPU resources to workloads and can run alongside other schedulers already installed on a cluster.
The project supports the entire AI lifecycle, from small interactive jobs requiring minimal resources to large training and inference workloads, all within the same cluster. It ensures optimal resource allocation while maintaining resource fairness between different consumers. According to GitGenius activity tracking, the repository has demonstrated strong community engagement with a median issue and pull request response latency of 0.0 hours and a mean response time of 9.3 hours across 280 tracked items. The most active contributors tracked by GitGenius are enoodle with 272 events, davidLif with 209 events, and romanbaron with 190 events. Enhancement requests represent the most active issue label with 106 items, followed by bug reports with 65 items.
KAI Scheduler includes comprehensive features for managing AI workloads. Batch scheduling ensures all pods in a group are scheduled simultaneously or not at all. The scheduler supports both bin packing and spread scheduling strategies to optimize node usage. It provides hierarchical queues with two-level queue hierarchies for flexible organizational control, along with workload priority management and separation of workload priority from preemptibility as independent policies. Resource distribution can be customized through quotas, over-quota weights, limits, and priorities per queue. Fairness policies implement Dominant Resource Fairness and resource reclamation across queues, while time-based fairshare ensures fair resource usage over time considering historical usage and time decay parameters.
Additional capabilities include workload consolidation to reduce fragmentation, elastic workloads that dynamically scale within defined thresholds, and GPU sharing that allows multiple workloads to efficiently share single or multiple GPUs. The scheduler supports Dynamic Resource Allocation through Kubernetes ResourceClaims for vendor-specific hardware like NVIDIA and AMD GPUs. Recent major features released in version 0.10.0 include Topology-Aware Scheduling for optimized placement, Hierarchical PodGroups for gang scheduling of multi-level workloads, and Time-based Fairshare. The scheduler is fully compatible with dynamic cloud infrastructures including auto-scalers like Karpenter as well as static on-premise deployments.
The project is built on the foundation of kube-batch and has established connections with related Kubernetes projects including kubernetes-sigs/kubespray, kubernetes-sigs/kueue, and kubernetes/kubernetes through overlapping contributors. Recent integrations include native support for Ray workloads on Kubernetes and integration with Grove and Dynamo for orchestrating complex multi-component workloads like disaggregated serving and agentic pipelines. The repository maintains active development with enhancement requests being the dominant issue category, indicating ongoing feature development and community-driven improvements to the scheduler's capabilities.
