DATABASES Signal 129
Announcing ClickHouse On-Demand Compute: Instant compute for your most intensive workloads
↗
Users can request additional ClickHouse workers for individual queries, isolating heavy workloads from the main cluster.
It removes the need to overprovision clusters for peak query demand. Heavy or ad-hoc queries can run on a shared worker pool without waiting for autoscaling to react. This also enables direct querying of data lake formats like Iceberg and Delta on demand.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
On-Demand Compute relies on a new distributed query execution framework that uses ClickHouse multistage execution and a cost-based optimizer to pick efficient plans.
Each query leases workers from a shared pool for a minimum of 60 seconds, with leases renewed automatically and workers reused for subsequent queries while the lease is active.
If the worker pool cannot satisfy a request, the query runs with whatever workers are available, and concurrent queries share workers up to the largest request rather than summing them.
THE READ
What the cluster adds up to.
ClickHouse Cloud now offers On-Demand Compute, a feature that lets a query request extra workers from a managed pool instead of scaling the whole cluster. This changes the way intensive workloads are handled because the request is made per query rather than relying on metric-based autoscaling. The feature is currently in private preview and requires specific query settings to activate.
The underlying technology combines a new distributed query execution framework based on ClickHouse multistage execution with a cost-based optimizer that evaluates alternative plans. Users enable it with settings such as use_distributed_plan=1, distributed_plan_workers_num=<N>, and enable_parallel_replicas=0. These settings direct the query to draw workers from the shared pool.
When a query runs, it leases the requested number of workers for at least sixty seconds; the lease renews automatically if the query continues. After the query finishes the workers stay leased but inactive, allowing immediate reuse for later queries without a cold start. This reuse reduces latency compared to provisioning new nodes each time.
Concurrent queries do not receive isolated worker sets; they share the pool up to the size of the largest request. If the pool cannot meet a request, the query proceeds with whatever workers are currently available. During the preview the pool’s autoscaling is being tuned, so occasional shortages may occur.
On-Demand Compute is useful for ad-hoc exploration, moving selected read workloads off the main cluster, and querying data lake formats like SharedMergeTree, Iceberg, and Delta. It avoids the need to overprovision for peak loads and eliminates the wait for reactive autoscaling, though it is limited to the supported storage engines and requires the explicit query settings described.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER