Shopify powers observability for global-scale commerce with ClickHouse

Shopify’s migration to ClickHouse for observability addresses a common pain point for large-scale engineering teams: fragmented tooling. By consolidating metrics, logs, traces, and exceptions onto a single open-source database, the company eliminated the complexity and cost of managing multiple proprietary vendors. This shift also removed the unpredictability of vendor pricing, which had previously scaled faster than the infrastructure itself. For engineers, this validates the feasibility of replacing specialized observability tools with a general-purpose columnar database, provided the workload aligns with ClickHouse’s strengths in high-throughput, time-series data.

The performance gains are substantial. Shopify reports up to 30x faster queries after moving to ClickHouse, with baseline improvements of 16x. This is critical for a platform where a minute of downtime during peak events like Black Friday, Cyber Monday can cost millions in lost transactions. The system ingests 100 million events per second at peak (110 GB/s of uncompressed telemetry) and makes data queryable in under a minute. However, these results come with trade-offs: ClickHouse required custom engineering to handle Shopify’s scale, and the team now manages its own infrastructure or considers ClickHouse Cloud for future deployments.

The choice of ClickHouse was driven by its open-source nature, scalability, and cost structure. Unlike proprietary vendors, ClickHouse’s lack of per-byte pricing allowed Shopify to control costs as its telemetry volume grew. The open-source model also enabled the team to contribute to the project and debug issues directly, which is valuable for a platform operating at Shopify’s scale. That said, the solution isn’t turnkey, Shopify’s engineering team had to build and maintain the platform internally, which may not be feasible for smaller teams without similar resources.

For engineers evaluating similar migrations, Shopify’s experience highlights both the benefits and challenges of self-hosting a high-performance database. The platform’s ability to handle 1.5 million pods across 500 Kubernetes clusters and serve 2.2 trillion edge requests during peak events demonstrates ClickHouse’s scalability. However, the decision to build an internal platform (Observe) rather than rely on off-the-shelf tools underscores the need for customization when operating at this scale. Teams considering this path should weigh the cost of self-hosting against the flexibility and control it provides.