brain

Change Data Capture

Properties7
statusseedling
typeconcept
created2026-09-24
updated2026-10-04
tagscdc, data-engineering, duckdb, parquet, olap, streaming
aliasesCDC, Change data capture
public

2 min read

Change Data Capture

Change Data Capture is the pattern of recording every change (insert/update/delete) that hits a source database as an ordered stream of events, instead of repeatedly asking the database for its full current state. CDC is the bridge between operational databases and analytics: the source stays untouched, and downstream consumers receive changes as they happen.

The classic shape is Source DB → CDC → Stream → Parquet landing → query engine. Debezium reads the database’s replication log (Postgres WAL, MySQL binlog), Kafka or Redpanda transports the change events, a small consumer writes Parquet micro-batches to object storage, and an engine like duckdb (OSS) queries the files. You can swap components, but the shape stays the same.

The interesting move in the captured article is treating the append-only Parquet lake as a log, not a table: never update files, only append change events, and reconstruct “current truth” at query time as the latest row per key. You trade heavier queries (latest-state reconstruction) for a much simpler ingestion path — writes are immutable, idempotent, and parallel-safe, which makes pipelines calmer. The operational price is compaction: append-only logs grow forever, so latest-per-key queries over months of files eventually need a periodic rewrite that collapses history.

Near-real-time is mostly about not rescanning the universe: keep a small watermark table (last processed ingest_ts or file name) in a persistent DuckDB database, each run reads only partitions newer than the watermark, and CHECKPOINT syncs the WAL for durability. DuckDB ends up being both the query engine and the lightweight state store.

  • Reference architecture + SQL patterns: DuckDB Streaming CDC (hjparmar1944, Medium) — captured 2026-09-24, clippings/medium-com-hjparmar1944-duckdb-streaming-cdc-near-real-time-analytics-from-append-only-parquet-95027f3f4af4.md (via freedium mirror)
  • DuckDB now ships inside dbt v2 (Rust Fusion engine, built-in adapter): @duckdb, 2026-09-24 — captured 2026-09-24, clippings/x-2102923598277255575.md
  • duckle (OSS) (1.4k★, Rust) — CDC as a checkbox in a self-hosted DuckDB-based ETL/ELT platform (385 components, reverse ETL, lineage, MCP for agents): the productization of this pattern; see also Embedded databases for the DuckDB-as-substrate thread

Growing

  • How does this compare to streaming tables / materialized views as the “current truth” layer?
  • Where does S3 object storage after MinIO fit as the Parquet landing zone — the article assumes object storage, and the MinIO alternatives research is adjacent.

Graph View

Backlinks