Skip to main content
ETL (Extract, Transform, Load) is the core use case in Zwiron: read data from a source, optionally transform it in transit, and write it to a destination.

​
Creating an ETL pipeline

  1. Go to Jobs → New Job
  2. Select ETL as the job type
  3. Follow the 4-step wizard:
    • Source — choose the source connection and execution mode
    • Destination — choose the destination connection and schema
    • Tables — select which tables to sync; Zwiron auto-discovers them
    • Settings — choose sync mode, destination write mode, schedule, retries

​
Source sync modes

See Sync Modes for a full comparison.

​
Destination write modes

Choosing between Append and Upsert for Incremental / CDC:
  • Use Upsert when your source table has a primary key and rows can be updated. Upsert prevents duplicate rows when a row is re-read after being modified. Available for SQL databases.
  • Use Append only when source rows are immutable — event logs, audit trails, activity feeds, or sources without a primary key (SaaS APIs, MongoDB, document stores). Appending to a table where rows can be updated will produce duplicates and may cause primary key conflicts.
Sync mode and destination write mode are locked after a job is created. Choose carefully — if you need to change them, you’ll need to create a new job.

​
Table selection

When you choose a source connection, Zwiron connects through your agent and automatically discovers all tables and schemas. You can:
  • Select individual tables
  • Select entire schemas (all tables)
  • Search by table name
  • Manually enter table names if you prefer
Zwiron auto-selects all tables on first load. Deselect any you don’t want to sync.

​
Destination schema

When you create a job, Zwiron auto-generates the destination schema name from your source connection name. For example, if your source connection is called “Prod Postgres”, the destination schema will be prod_postgres. This keeps each source isolated in the destination — each source gets its own namespace so multiple jobs sharing the same destination never collide. You can override the schema name in the job’s advanced settings if needed. If the schema you choose overlaps with another active job’s tables, Zwiron shows a conflict warning — but it won’t block you. Tables are created automatically in the destination if they don’t exist. Column names and types are mapped from source to destination types automatically.

​
Parallel execution

Zwiron can read and write multiple tables in parallel. The number of parallel workers is configured per job (default: auto). For large tables, Zwiron also partitions reads — splitting the table into chunks and reading them concurrently for maximum throughput.

​
Retries

Each table sync is retried independently on failure. The default is 3 retries with exponential backoff. Configure the max retry count in the job’s advanced settings.

​
Performance tips

  • Use Incremental mode instead of Full Refresh for tables larger than ~500K rows
  • Enable CDC for real-time requirements or when you need to capture deletes
  • Add an index on your cursor column (updated_at, id) for fast incremental reads
  • For Snowflake/BigQuery destinations, Zwiron uses bulk load — no row-by-row inserts
Last modified on July 13, 2026