Full Refresh
Reads the entire source table every run and replaces the destination table.- Small tables (< 1 million rows)
- Lookup tables or reference data that changes infrequently
- You don’t have a reliable cursor column (no
updated_at, no auto-increment ID) - You want a guaranteed clean copy every run
- Slow on large tables — reads everything every time
- Puts more load on the source database
- Destination table is truncated and rewritten on each run (brief period where it may be empty)
Incremental
Only reads rows that are new or changed since the last run. Zwiron tracks a cursor column — typicallyupdated_at or an auto-increment id — and only fetches rows where the cursor is greater than the last recorded value.
- Large tables where copying everything each run is too slow or too expensive
- Your table has a reliable cursor column (
updated_at,created_at,id, sequence) - You want lower load on the source database
- Append — new rows are inserted without touching existing rows. Use this when your source rows are immutable (event logs, append-only tables). Do not use Append if source rows can be updated — re-reads will produce duplicate rows and can cause duplicate key errors at the destination.
- Upsert — Zwiron inserts new rows or updates existing ones using the source primary key. Requires a primary key on the source table. Works with SQL databases. Not suitable for SaaS APIs, MongoDB, or sources that don’t have a unique key per row.
- Requires a cursor column that increments reliably
- Does not capture deletes — if a row is deleted at the source, it stays in the destination
- Rows updated with an older timestamp won’t be re-synced
CDC — Change Data Capture
Streams every insert, update, and delete from the source in real time. Instead of querying the table, CDC reads directly from the database’s change log:- Postgres — logical replication slot (uses
pgoutputorwal2json) - MySQL — binary log (binlog)
- MongoDB — Change Streams
- You need near-real-time data (sub-minute latency)
- You need to capture deletes
- You want minimum load on the source (no polling queries)
- High-volume tables where incremental queries are slow
- CDC-capable source live today: Postgres, MySQL, or MongoDB
- Additional database permissions (replication user / change streams)
- Primary key on the source table for SQL sources (required to apply updates and deletes at destination)
- See the connector-specific guide for exact setup steps
- Requires database-level configuration (replication must be enabled)
- More complex to set up than Full Refresh or Incremental
- Some databases have limits on how long the change log is retained — if CDC falls behind, it may need to re-sync
More CDC sources are on demand — see More connectors on demand.
Comparison
Choosing the right mode
- Start with Full Refresh if your table is small or you’re unsure
- Switch to Incremental when your table grows and full refresh becomes too slow
- Use CDC when you need real-time data or need to capture deletes
Destination write modes
Each sync mode is paired with a destination write mode that controls how data lands in the destination table.
Overwrite truncates the destination table and replaces it entirely each run. It is the only safe option for Full Refresh — using Append with Full Refresh would duplicate every row on each run.
Append adds incoming rows without modifying existing rows. Safe for immutable sources (event logs, audit trails, SaaS activity feeds). Avoid for tables where rows are updated — use Upsert instead.
Upsert inserts new rows or updates existing ones based on the source primary key. Prevents duplicates when rows change. Requires a primary key — available for SQL databases only, not for SaaS APIs, MongoDB, or document stores.