Change Data Capture (CDC)

• Real-time data — changes land at the destination in seconds, not minutes
• Deletes captured — when a row is deleted at the source, it’s deleted at the destination
• Low source load — no polling queries; the agent reads the log that the database is already writing

​

How CDC works

Source Database                    Agent                    Destination
─────────────                      ─────                    ───────────
INSERT row → written to log ──▶  reads log ──▶  INSERT at destination
UPDATE row → written to log ──▶  reads log ──▶  UPDATE at destination
DELETE row → written to log ──▶  reads log ──▶  DELETE at destination

​

Supported sources (live today)

​

Setting up CDC on PostgreSQL

​

1. Enable logical replication

wal_level = logical
max_replication_slots = 5
max_wal_senders = 5

​

2. Create a replication user

CREATE USER zwiron_cdc WITH REPLICATION LOGIN PASSWORD 'your_password';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO zwiron_cdc;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO zwiron_cdc;

​

3. Create a publication

CREATE PUBLICATION zwiron_pub FOR ALL TABLES;

​

4. Use CDC in Zwiron

​

Setting up CDC on MySQL

​

1. Enable binary logging

[mysqld]
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
binlog_row_image = FULL
expire_logs_days = 7

​

2. Create a replication user

CREATE USER 'zwiron_cdc'@'%' IDENTIFIED BY 'your_password';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'zwiron_cdc'@'%';
FLUSH PRIVILEGES;

​

Requirements for CDC

• Primary key required — every table you want to CDC must have a primary key. Zwiron uses this to identify which row to update or delete at the destination.
• Replication must be enabled — see the database-specific instructions above.
• Log retention — the database must retain the change log long enough for Zwiron to read it. If CDC falls too far behind (e.g. agent offline for a long time), it may need to perform a full re-sync.

​

CDC vs. Incremental

​

Checkpointing

​

What happens if CDC falls behind?