From manual negotiation to automated scheduling: How AI21 manages its GPU fleet with Kueue

AI21 faced challenges managing its large, shared GPU fleet of around 10,000 GPUs across multiple teams. Initial management through manual negotiation in messaging channels proved inefficient, unfair, and unable to scale as utilization approached 100%. The company transitioned to Kueue, a Kubernetes-native job queueing system, to automate workload scheduling, prioritization, and preemption on their GKE clusters. This new system addressed key principles of fairness, order, and efficiency, significantly reducing manual intervention and resource starvation for critical jobs. The implementation and real-world feedback from AI21 also contributed to the development of new features within the open-source Kueue project.