Netflix replaces in-house batch job system with open-source Kubernetes-native Kueue

Netflix’s decision to replace its homegrown Compute Managed Batch (CMB) with Kueue reflects a broader trend of adopting open-source, Kubernetes-native tools for batch job management. CMB, built in 2018, initially addressed Netflix’s need for tenant hierarchies and multi-cluster federation, but over time, its integration with Kubernetes became a limitation. Kueue, as a Kubernetes-native solution, aligns more closely with the ecosystem’s evolution, reducing the engineering effort required to maintain and extend the system. This shift also allows Netflix to leverage community-driven innovation, which can outpace the development of proprietary solutions in both features and scalability.

The migration process prioritized continuity for existing users, a critical factor for large-scale platforms. By maintaining API parity with CMB, Netflix minimized disruption, allowing tenants to transition without changes to their workflows. Engineers mapped CMB’s tenant hierarchies to Kueue’s Cohors, ClusterQueue, and LocalQueue resources, ensuring that capacity requirements and isolation policies were preserved. This approach derisked the project, as it enabled gradual adoption and easy rollback if issues arose. The decision to migrate the most complex use case first further validated Kueue’s capabilities, ensuring the system could handle Netflix’s production workloads before broader deployment.

Kueue’s features, such as preemption-based fair sharing, address a key challenge in batch job management: balancing resource utilization with reservation guarantees. Netflix’s implementation of this feature improved average resource utilization by allowing idle capacity to be temporarily allocated to other tenants while maintaining the ability to reclaim it when needed. This capability is particularly valuable in multi-tenant environments where workloads vary in priority and resource demands. The migration also highlights the importance of load testing in non-production environments, which helped fine-tune Kueue’s configuration to meet Netflix’s throughput requirements without compromising stability.

For engineers considering a similar migration, Netflix’s experience underscores the value of open-source tools in reducing long-term maintenance costs. Kueue’s adoption by Netflix, managing millions of batch workloads, demonstrates that open-source solutions can scale to meet the demands of large, complex systems. However, the migration also reveals the trade-offs involved: while Kueue offers advanced features like topology-aware scheduling, integrating it into an existing infrastructure requires careful planning to ensure compatibility with legacy systems and workflows. The project’s success hinged on maintaining API parity and prioritizing tenant transparency, lessons applicable to any large-scale system migration.