> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zwiron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# BigQuery Connector

> Load data into Google BigQuery from Postgres, MySQL, MongoDB, and more with hosted or agent sync in Zwiron.

<Info>
  **Live — enterprise connector.** Fully supported today (SAT-validated). One of Zwiron's five production connectors: PostgreSQL, MySQL, MongoDB, Snowflake, BigQuery.
</Info>

Google BigQuery is a serverless, highly scalable cloud data warehouse. Zwiron supports BigQuery as a destination for loading data from any source, and as a source for reading data into other systems.

<Note>
  BigQuery has achieved **GA (Hardened)** status — the highest conformance tier in Zwiron's Standard Acceptance Test suite. All three levels passed: basic correctness, production stress (100K–1M rows, failure/resume, schema evolution), and enterprise hardening (concurrency, performance gates).
</Note>

***

## Supported sync modes

| As destination | Write modes             |
| -------------- | ----------------------- |
|                | Append, Upsert, Replace |

| As source | Full Refresh | Incremental |
| --------- | ------------ | ----------- |
|           | ✅            | ✅           |

***

## Connection details

| Field      | Description                                                   |
| ---------- | ------------------------------------------------------------- |
| Project ID | Your Google Cloud project ID                                  |
| Dataset    | The BigQuery dataset to read from or write to                 |
| Auth       | **Sign in with Google** (OAuth) or a service account JSON key |
| Location   | Dataset location (e.g. `US`, `EU`, `us-central1`)             |

***

## Sign in with Google (OAuth)

The easiest way to connect BigQuery is **Sign in with Google** in the connection form.

1. Enter your **GCP project ID**
2. Click **Sign in with BigQuery** and approve access in the Google popup
3. Pick a **dataset** from the discovered list (and optionally set location)

Your Google account needs BigQuery IAM permissions on that project (for example `BigQuery Data Editor` + `BigQuery Job User`).

### Admin setup — Google OAuth client

A Zwiron admin must configure a Google Cloud OAuth client once:

1. In [Google Cloud Console → Credentials](https://console.cloud.google.com/apis/credentials), create an **OAuth client ID** (Web application)
2. Add authorized redirect URI: `https://yourdomain.com/v1/oauth/callback`
3. Enable the **BigQuery API** on the project that owns the OAuth client
4. Set these environment variables on Atlas:

```
OAUTH_BIGQUERY_CLIENT_ID=<your client ID>
OAUTH_BIGQUERY_CLIENT_SECRET=<your client secret>
```

Until these are set, the Sign in button is hidden and only service-account auth is offered.

***

## Setting up a service account

Use a service account when you prefer key-based auth (agents, CI, or shared automation identities).

### Step 1 — Create a service account

In the Google Cloud Console:

1. Go to **IAM & Admin → Service Accounts**
2. Click **Create Service Account**
3. Name it `zwiron-agent` and click **Create**
4. Skip optional steps and click **Done**

### Step 2 — Grant permissions

Assign these IAM roles to the service account:

| Role                         | Purpose                                         |
| ---------------------------- | ----------------------------------------------- |
| `BigQuery Data Editor`       | Read and write tables                           |
| `BigQuery Job User`          | Run queries and load jobs                       |
| `BigQuery Read Session User` | (Optional) For reading large tables efficiently |

### Step 3 — Create a key

1. Click the service account → **Keys → Add Key → Create new key**
2. Choose **JSON**
3. Download the key file

Paste the contents of the JSON key file into the **Service Account JSON** field in the Zwiron connection form.

***

## How Zwiron writes to BigQuery

For high-throughput writes, Zwiron uses the **BigQuery Storage Write API** (default) or `LOAD DATA` jobs:

* **Append mode** — rows are appended to the table
* **Upsert mode** — rows are loaded to a staging table, then merged using `MERGE INTO` based on the primary key
* **Replace mode** — table is truncated and replaced with new data

Zwiron creates tables automatically if they don't exist.

***

## Schema mapping

| Source type       | BigQuery type |
| ----------------- | ------------- |
| INTEGER / BIGINT  | INT64         |
| FLOAT / DOUBLE    | FLOAT64       |
| DECIMAL / NUMERIC | NUMERIC       |
| VARCHAR / TEXT    | STRING        |
| BOOLEAN           | BOOL          |
| DATE              | DATE          |
| TIMESTAMP         | TIMESTAMP     |
| JSON / JSONB      | JSON          |
| BYTES / BLOB      | BYTES         |

***

## Partitioning and clustering

When Zwiron creates a table in BigQuery, you can configure:

* **Partitioning** by a timestamp column (e.g. `created_at`) — reduces query cost significantly
* **Clustering** by a frequently-filtered column — improves query performance

Configure these in the job's advanced destination settings.

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Permission denied on dataset">
    Ensure the service account has `BigQuery Data Editor` on the specific dataset, or `BigQuery Admin` on the project. Dataset-level permissions are separate from project-level IAM.
  </Accordion>

  <Accordion title="Project not found">
    Double-check the Project ID (not the project name). The project ID is visible in the Cloud Console header and in the service account JSON file under `"project_id"`.
  </Accordion>

  <Accordion title="Quota exceeded">
    BigQuery has per-project and per-day quotas for load jobs. For very high-frequency pipelines, contact Google to increase quotas or use streaming inserts instead.
  </Accordion>
</AccordionGroup>