> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zwiron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks Connector

> Connect Databricks as a source or destination in Zwiron — Unity Catalog, Volume staging, SQL warehouse.

Use Databricks as a **destination** (load data into Delta tables on Unity Catalog) or a **source** (read existing tables).

Zwiron connects through a **Databricks SQL warehouse** using the Databricks SQL driver (SQL warehouse / JDBC-style connectivity). Writes use **Unity Catalog Volumes** for Parquet staging, then `COPY INTO` — the lakehouse equivalent of Snowflake PUT + COPY.

***

## Supported sync modes

| As destination | Write modes             |
| -------------- | ----------------------- |
|                | Append, Upsert, Replace |

| As source | Full Refresh | Incremental |
| --------- | ------------ | ----------- |
|           | ✅            | ✅           |

When Databricks is the **destination**, source connectors that support CDC (PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, and others) can stream changes into Databricks tables.

***

## Connection details

| Field            | Required | Description                                                                                                                                                                           |
| ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Host**         | Yes      | Workspace server hostname (e.g. `dbc-xxxx.cloud.databricks.com` or `adb-xxx.azuredatabricks.net`)                                                                                     |
| **Warehouse ID** | Yes      | SQL warehouse ID (Zwiron builds `/sql/1.0/warehouses/{id}` unless you override HTTP path)                                                                                             |
| **Token**        | Yes      | Personal access token or service principal token                                                                                                                                      |
| **Catalog**      | No       | Unity Catalog name. Prefer a real UC catalog (e.g. `main`). Default in product: `hive_metastore` — **Volume staging requires Unity Catalog**; use a UC catalog for destination loads. |
| **Schema**       | No       | Target schema (default: `default`)                                                                                                                                                    |
| **HTTP path**    | No       | Override only if your warehouse uses a non-standard path                                                                                                                              |

***

## How to connect

### Step 1 — SQL warehouse connection details

1. In Databricks, open **SQL Warehouses**.
2. Select (or create) a warehouse and start it if it is stopped.
3. Open **Connection details**.
4. Copy **Server hostname**. The HTTP path usually includes the warehouse ID (e.g. `/sql/1.0/warehouses/<id>`).

### Step 2 — Access token

Create a token for a user or service principal that has the privileges below:

1. **Settings → Developer → Access tokens → Generate new token**, or
2. Create a **service principal** and generate a token for it (recommended for production).

Paste the token into Zwiron. Prefer a dedicated identity with least privilege — not a workspace admin personal token.

### Step 3 — Connect in Zwiron

1. Go to **Connections → Add Connection → Databricks**
2. Fill in host, warehouse ID, token, **Unity Catalog** name, and schema
3. Click **Test Connection**
4. Save the connection

***

## Privileges (Unity Catalog)

Grant the connecting identity at least:

| Privilege                      | On      | Purpose                                                   |
| ------------------------------ | ------- | --------------------------------------------------------- |
| `USE CATALOG`                  | Catalog | Resolve the three-level namespace                         |
| `USE SCHEMA`                   | Schema  | Read/write within the schema                              |
| `CREATE SCHEMA`                | Catalog | Optional — Zwiron can create the target schema if missing |
| `CREATE TABLE`                 | Schema  | Create destination tables                                 |
| `MODIFY`                       | Tables  | Insert / merge / replace                                  |
| `SELECT`                       | Tables  | Read for source syncs and validation                      |
| `CREATE VOLUME`                | Schema  | Create the staging volume `__zwiron_staging`              |
| `WRITE VOLUME` / `READ VOLUME` | Volume  | PUT Parquet files and `COPY INTO`                         |

Example (adjust names):

```sql theme={null}
GRANT USE CATALOG ON CATALOG main TO `zwiron-sp`;
GRANT USE SCHEMA ON SCHEMA main.analytics TO `zwiron-sp`;
GRANT CREATE TABLE, CREATE VOLUME ON SCHEMA main.analytics TO `zwiron-sp`;
-- After tables exist:
GRANT SELECT, MODIFY ON TABLE main.analytics.orders TO `zwiron-sp`;
```

***

## How Zwiron writes to Databricks

For destination loads, Zwiron:

1. Ensures the **catalog.schema** exists (`CREATE SCHEMA IF NOT EXISTS`)
2. Creates a Unity Catalog volume `__zwiron_staging` when needed
3. Writes **Parquet** files and **PUT**s them to `/Volumes/{catalog}/{schema}/__zwiron_staging/...`
4. Runs **`COPY INTO`** into the target table (or a temp table for upsert)
5. For **upsert**, merges into the target with primary keys, then cleans staging files

This path requires **Unity Catalog volumes**. Managed Delta tables in your catalog/schema are the default landing target.

***

## Schema mapping

| Source type           | Databricks / Delta type |
| --------------------- | ----------------------- |
| INTEGER / INT         | INT                     |
| BIGINT                | BIGINT                  |
| FLOAT                 | FLOAT                   |
| DOUBLE                | DOUBLE                  |
| DECIMAL / NUMERIC     | DECIMAL(p,s)            |
| VARCHAR / TEXT / JSON | STRING                  |
| BOOLEAN               | BOOLEAN                 |
| DATE                  | DATE                    |
| TIMESTAMP             | TIMESTAMP               |
| BYTES / BLOB          | BINARY                  |
| ARRAY / MAP / STRUCT  | STRING (JSON text)      |

Zwiron creates tables automatically when they do not exist and can evolve schemas (add/widen columns) according to job settings.

***

## Cloud coverage

The same connection shape works on Databricks workspaces hosted on **AWS**, **Azure**, and **GCP**. Use the hostname and SQL warehouse from that cloud’s workspace.

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Permission denied creating volume or COPY INTO">
    Confirm the identity has `CREATE VOLUME` on the schema and write access on `__zwiron_staging`. Legacy Hive-only setups without volume support cannot use the current staging path — use a Unity Catalog–enabled workspace.
  </Accordion>

  <Accordion title="Warehouse stopped / connection timeout">
    Start the SQL warehouse in Databricks, then retry **Test Connection**. Auto-stop warehouses must be running for the first sync.
  </Accordion>

  <Accordion title="Invalid access token">
    Regenerate the PAT or service principal token. Ensure the token belongs to an identity that can use the selected warehouse (`CAN USE`).
  </Accordion>

  <Accordion title="Catalog or schema not found">
    Create the catalog/schema in Unity Catalog, or grant `CREATE SCHEMA` so Zwiron can create the schema on connect. Check that **Catalog** in Zwiron matches the UC name exactly.
  </Accordion>
</AccordionGroup>

***

## Related

* [Connectors overview](/connectors/overview)
* [PostgreSQL](/connectors/databases/postgres) → Databricks pipelines
* [HubSpot](/connectors/hubspot) → warehouse sync
* [Network access / egress IPs](/concepts/network-access) for hosted mode