> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zwiron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# ETL Pipelines

> Extract, transform, and load from any live Zwiron source to destination — full refresh, incremental, or CDC.

**ETL (Extract, Transform, Load)** is the core use case in Zwiron: read data from a source, optionally transform it in transit, and write it to a destination.

***

## Creating an ETL pipeline

1. Go to **Jobs** → **New Job**
2. Select **ETL** as the job type
3. Follow the 4-step wizard:
   * **Source** — choose the source connection and execution mode
   * **Destination** — choose the destination connection and schema
   * **Tables** — select which tables to sync; Zwiron auto-discovers them
   * **Settings** — choose sync mode, destination write mode, schedule, retries

***

## Source sync modes

| Mode             | Description                                     | Best for                                              |
| ---------------- | ----------------------------------------------- | ----------------------------------------------------- |
| **Full Refresh** | Read the entire table every run                 | Small tables, no cursor column                        |
| **Incremental**  | Read only rows newer than the last cursor value | Large tables with `updated_at` or auto-increment `id` |
| **CDC**          | Stream changes from the database change log     | Real-time, captures deletes                           |

See [Sync Modes](/concepts/sync-modes) for a full comparison.

***

## Destination write modes

| Mode          | Behavior                                                 | Available with    |
| ------------- | -------------------------------------------------------- | ----------------- |
| **Overwrite** | Truncate the destination table and replace with new data | Full Refresh only |
| **Append**    | Insert new rows without touching existing rows           | Incremental, CDC  |
| **Upsert**    | Insert or update rows based on primary key               | Incremental, CDC  |

**Choosing between Append and Upsert for Incremental / CDC:**

* Use **Upsert** when your source table has a primary key and rows can be updated. Upsert prevents duplicate rows when a row is re-read after being modified. Available for SQL databases.
* Use **Append** only when source rows are immutable — event logs, audit trails, activity feeds, or sources without a primary key (SaaS APIs, MongoDB, document stores). Appending to a table where rows can be updated will produce duplicates and may cause primary key conflicts.

<Warning>
  Sync mode and destination write mode are **locked after a job is created**. Choose carefully — if you need to change them, you'll need to create a new job.
</Warning>

***

## Table selection

When you choose a source connection, Zwiron connects through your agent and automatically discovers all tables and schemas. You can:

* Select individual tables
* Select entire schemas (all tables)
* Search by table name
* Manually enter table names if you prefer

Zwiron auto-selects all tables on first load. Deselect any you don't want to sync.

***

## Destination schema

When you create a job, Zwiron auto-generates the destination schema name from your **source connection name**. For example, if your source connection is called "Prod Postgres", the destination schema will be `prod_postgres`.

This keeps each source isolated in the destination — each source gets its own namespace so multiple jobs sharing the same destination never collide.

| Source connection | Destination | dest\_schema     |
| ----------------- | ----------- | ---------------- |
| Prod Postgres     | Snowflake   | `prod_postgres`  |
| Salesforce CRM    | Snowflake   | `salesforce_crm` |
| MySQL Staging     | BigQuery    | `mysql_staging`  |

You can override the schema name in the job's advanced settings if needed. If the schema you choose overlaps with another active job's tables, Zwiron shows a **conflict warning** — but it won't block you.

Tables are created automatically in the destination if they don't exist. Column names and types are mapped from source to destination types automatically.

***

## Parallel execution

Zwiron can read and write multiple tables in parallel. The number of parallel workers is configured per job (default: auto). For large tables, Zwiron also partitions reads — splitting the table into chunks and reading them concurrently for maximum throughput.

***

## Retries

Each table sync is retried independently on failure. The default is 3 retries with exponential backoff. Configure the max retry count in the job's advanced settings.

***

## Performance tips

* Use **Incremental** mode instead of Full Refresh for tables larger than \~500K rows
* Enable **CDC** for real-time requirements or when you need to capture deletes
* Add an **index on your cursor column** (`updated_at`, `id`) for fast incremental reads
* For Snowflake/BigQuery destinations, Zwiron uses bulk load — no row-by-row inserts

***

## Related

* [Reverse ETL](/pipelines/reverse-etl) — push data from warehouse back to operational systems
* [Transforms](/concepts/transforms) — rename, filter, hash, and compute columns
* [Schema Evolution](/concepts/schema-evolution) — handle source schema changes gracefully
* [Scheduling](/concepts/scheduling) — configure when jobs run