DATABASES Signal 191
Let’s Encrypt builds new data warehouse using ClickHouse for improved analytics
↗
Comments
Let’s Encrypt has transitioned to using ClickHouse for their data warehouse, significantly improving their ability to analyze certificate issuance data. This change addresses previous inefficiencies in data processing and provides a scalable solution for future growth. The move highlights the importance of selecting the right database technology to meet both current and future analytical needs.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Let’s Encrypt chose ClickHouse for its cost-efficient storage and fast aggregation capabilities.
The new infrastructure allows for querying issuance records in milliseconds instead of hours.
Custom tools were developed to meet specific reporting needs, enhancing usability and efficiency.
THE READ
What the cluster adds up to.
Let’s Encrypt replaced its previous logging and reporting system with a data warehouse built on ClickHouse, which now handles large volumes of certificate issuance logs more efficiently. The new system is designed to process structured data and has a capacity of around 100TB, currently utilizing only 14% of that space. This allows for significant future data growth without immediate concerns about capacity.
The transition to ClickHouse resulted in drastic improvements in query performance. Tasks that previously took hours to complete can now be executed in under 10 seconds. This shift not only speeds up the data processing pipeline but also reduces the workload on engineers, allowing them to focus on more strategic tasks rather than manual data handling.
While ClickHouse provides robust capabilities for log ingestion and querying, Let’s Encrypt needed to develop custom solutions for scheduled reporting, indicating that some engineering effort is still required to fully tailor the system to their specific needs. This additional tool has improved report readability and streamlined processes, showcasing the importance of integrating custom solutions with off-the-shelf technologies.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER