DATABASES Signal 187
WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL
↗
WalShadow replicates Postgres data to ClickHouse with around 200 ms latency.
WalShadow introduces a new approach to data replication that minimizes latency and resource consumption. By leveraging the physical Write-Ahead Log (WAL), it supports complex schema changes while maintaining high performance. This can significantly improve the efficiency of analytics workflows for organizations using both Postgres and ClickHouse.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
WalShadow replicates data from Postgres to ClickHouse directly from the physical WAL stream.
It achieves approximately 200 ms commit-to-visible latency while handling 289K rows/sec.
The tool supports complex schema changes without the operational overhead of traditional logical replication.
THE READ
What the cluster adds up to.
WalShadow represents a significant advancement in the way Postgres integrates with ClickHouse by utilizing the physical WAL for replication. This method reduces latency and increases throughput compared to traditional Change Data Capture (CDC) systems, which rely on logical replication. By processing the WAL outside of the source database, it achieves a performance level closer to that of a physical standby.
The tool's ability to handle schema evolution, including complex changes like ADD COLUMN and RENAME COLUMN, enhances its usability in dynamic environments. This flexibility allows engineers to adapt the data structure without significant downtime or performance impact, a common drawback in traditional systems that use logical replication slots.
However, the effectiveness of WalShadow is contingent upon the availability of physical WAL, which is not supported by many managed Postgres services. This limitation means that organizations relying on such services will need to consider alternatives or look for managed solutions that provide this access. The integration with ClickHouse Managed Postgres aims to address this gap, paving the way for a more seamless user experience.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER