Fresh context: change data capture, not batch ETL

The core argument is architectural: batch ETL pipelines refresh downstream context stores on a fixed schedule, leaving agents to act on data that may be hours old. CDC inverts this model by continuously streaming individual change events, inserts, updates, and deletes, from a source database to downstream systems. The practical effect is that an agent's context store stays aligned with the source database as changes happen, rather than lagging behind until the next scheduled refresh.

The article distinguishes three CDC implementation strategies with materially different trade-offs. Log-based CDC reads the database transaction log directly and asynchronously, which keeps capture off the write path, captures deletes, and requires no schema modifications. PostgreSQL streams committed changes through logical decoding, and MySQL's row-based binlog records row events with before or after images. Trigger-based CDC fires logic within the source database's own write transactions, capturing all change types but adding latency to every write. Query-based polling repeatedly queries for new rows by timestamp or version, but typically misses deletes entirely and loses intermediate state changes after downtime.

The article positions Debezium as the primary production-oriented tool for log-based CDC. Built on Kafka Connect, Debezium's MySQL and PostgreSQL connectors can achieve millisecond-range capture delay, though end-to-end latency depends on workload and pipeline. Roughly 90% of Debezium users deployed it on Apache Kafka with one topic per captured table by default. This means adopting CDC in a typical configuration involves standing up or extending a Kafka infrastructure, not just installing a connector.

CDC carries concrete engineering costs that the article is explicit about. Log-based connectors require source-specific setup including replication slots, binlog configuration, permissions, and retention policies. Debezium's delivery guarantee is at-least-once, so a change event may be delivered more than once, and downstream consumers must implement deduplication logic. These are different in kind from the problems batch pipelines create, staleness and missed intermediate states, but they are not trivial. Teams evaluating CDC need to weigh connector configuration overhead and consumer-side idempotency against the cost of agents acting on stale context.

Only one feed carried this story, so the claims about CDC's superiority over batch ETL and the specific latency figures reflect a single vendor blog post. The Air Canada tribunal example is cited as motivation rather than as direct evidence of a batch ETL failure, and the article does not present benchmark data comparing CDC and batch pipelines under controlled conditions. The technical descriptions of log-based, trigger-based, and query-based CDC are consistent with standard database engineering practice, but the recommendation to adopt CDC is an argument, not a measured result.