OBSERVABILITY Signal 129
Ensuring reliable OpenTelemetry ingestion at scale
↗
ClickHouse Cloud now ingests 50 million OpenTelemetry events per second via agent-gateway collectors and local write-ahead logs that absorb backpressure without losing data or delaying fresh telemetry
Handling 50M events/sec shows how observability pipelines can meet massive telemetry volumes while staying cost-effective. The shift from pure in-memory queues to disk-backed write-ahead logs lets the system survive downstream outages without data loss. Engineers can apply the same pattern to buffer bursty workloads without over-provisioning for peak demand.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
ClickHouse Cloud’s OpenTelemetry pipeline now processes 50 million events per second, equivalent to 10 GBps of compressed data.
The architecture evolved from agent-gateway with in-memory queues to adding local write-ahead logs via the OTel file_storage extension to absorb backpressure.
Local write-ahead logs are per-pod, cannot be split, and draining a 1 TiB backlog takes about four hours, limiting horizontal scaling during outages.
THE CLUSTER