DATABASES Signal 329
Fixed cadence to seconds: making ClickHouse Cloud autoscaling more reactive
↗
ClickHouse Cloud replaced its fixed-interval autoscaling loop with an event-driven controller-runtime pipeline that can react to load spikes in seconds.
The previous timer-based approach could leave a service undersized for several minutes after a sudden workload increase, leading to slower queries or out-of-memory errors. By processing real-time signals through a fast path, resources are provisioned almost immediately, improving performance and reducing failure risk. The periodic sweep is retained for steady-state verification and graceful scale-down, preserving cost efficiency.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
A custom binary that ran on a timer was swapped for a controller-runtime based reconciler that consumes ClickHouse-derived metrics as events.
The new system deduplicates work, applies exponential backoff, and limits concurrency, eliminating the need to hand-craft these reliability mechanisms.
Scaling up now happens within seconds, while the original periodic pass continues to handle routine adjustments and scale-down.
THE READ
What elseif makes of it.
Earlier, ClickHouse Cloud’s autoscaling relied on a single recommendation service that woke up at fixed intervals, examined every service, and then slept until the next tick. This design guaranteed predictability but introduced latency whenever a service experienced a rapid surge in demand, because the next evaluation could be minutes away. The result was temporary undersizing, slower query response, and possible memory exhaustion.
To address the latency, the team rebuilt the orchestration layer using the controller-runtime library, which is commonly used for Kubernetes operators. This library provides an event-driven work queue that automatically deduplicates items, rate-limits retries, and caps concurrent processing. Real-time usage signals are read directly from a ClickHouse table, feeding the queue as soon as they appear.
With the fast path in place, a service that suddenly needs more CPU or memory receives a recommendation within seconds, allowing the platform to spin up additional capacity almost instantly. This reactive scaling reduces the window of degraded performance and mitigates out-of-memory events that previously had to wait for the next scheduled pass. The periodic sweep remains unchanged for routine verification and for scaling resources down when demand eases.
Adopting the new pipeline requires integrating controller-runtime into the existing codebase and ensuring that the ClickHouse signals table is populated with up-to-date metrics. Engineers must also configure the work-queue parameters, such as concurrency limits and backoff settings, to match the scale of their deployment. The change does not add new hardware costs, but it does introduce a dependency on the controller-runtime library and on the ClickHouse-based signal source.
The fast path is designed only for upward scaling; downward adjustments still depend on the original timer-driven pass, which runs on a predictable schedule. If many services trigger the fast path simultaneously, the bounded concurrency may delay some recommendations, though deduplication prevents duplicate work. Consequently, the system retains a safety net for non-urgent scaling while delivering rapid response for critical load spikes.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER