Netflix reports 58% Flink compute savings as it moves 30,000+ jobs to open-source autoscaler

Netflix is moving from its custom cluster-level autoscaler to the open-source Apache Flink Autoscaler for over 30,000 streaming jobs. The old system scaled all operators in a job together based on cluster telemetry, which was inefficient for stateful pipelines with branches and joins. The new autoscaler uses per-operator metrics to estimate true processing rate and adjusts parallelism for individual vertices. This is a shift from coarse-grained to fine-grained scaling.

Adopting the open-source autoscaler required significant integration work. Netflix integrated it with its internal control plane using a Spring Boot service and Temporal workflows, rather than deploying via the Flink Kubernetes Operator. They modified JobManager metric collection to support up to 3,000 subtasks and added server-side metric filtering. They also had to handle forward connected subgraphs and sink backpressure, which are not trivial to implement.

The reported benefits are substantial: one team reduced annualized Flink compute expenditure by 58%, saving about $1.1 million annually. Netflix uses a 0.45 utilization target, lower than Flink's 0.7 default, to avoid aggressive rescaling of large stateful jobs. The approach builds on research from the DS2 project and is described in FLIP-271. This suggests the open-source autoscaler is mature enough for production use.

The autoscaler has known limitations. The forward connection issue is a problem: changing parallelism across a FORWARD connection can require redistribution, and Netflix keeps forward connected operators together. An open FLINK-38538 issue highlights cases where busy operators can be affected by output ratio-based scaling decisions. Also, state recovery during rescaling is costly, and Netflix is investigating Flink 2's disaggregated state architecture to address that. So the autoscaler works well for many cases but has edge cases that need attention.