> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zwiron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Amazon S3 Connector

> Read files from or write files to Amazon S3 in Zwiron pipelines.

Amazon S3 is an object storage service. Zwiron can read files from S3 (as a source) or write files to S3 (as a destination). Supported file formats include CSV, JSON (newline-delimited), and Parquet.

***

## Supported sync modes

| As source | Full Refresh | Incremental                            | CDC |
| --------- | ------------ | -------------------------------------- | --- |
|           | ✅            | N/A (no native date-filtered list API) | N/A |

| As destination | Write modes     |
| -------------- | --------------- |
|                | Append, Replace |

***

## Connection details

| Field             | Required | Description                                                  |
| ----------------- | -------- | ------------------------------------------------------------ |
| Bucket            | Yes      | S3 bucket name                                               |
| Region            | Yes      | Region of the bucket (e.g. `ap-southeast-2`)                 |
| Access key ID     | Yes      | IAM access key (`AKIA…`)                                     |
| Secret access key | Yes      | IAM secret key                                               |
| Path prefix       | No       | Optional prefix to scope reads/writes (e.g. `data/exports/`) |
| Endpoint          | No       | Leave blank for AWS; set only for MinIO/R2-compatible stores |

***

## Setting up IAM permissions

Create an IAM policy with the minimum required permissions:

### For source (read-only)

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}
```

### For destination (read-write)

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}
```

***

## Reading from S3 (source)

### File discovery

Zwiron lists all files in the configured bucket and prefix. You can filter by:

* **File extension** — only read `.csv`, `.json`, or `.parquet` files
* **Path pattern** — glob-style pattern (e.g. `orders/2024-*/*.parquet`)

### Supported formats

| Format        | Notes                                               |
| ------------- | --------------------------------------------------- |
| CSV           | Configurable delimiter, header row, quote character |
| JSON (NDJSON) | One JSON object per line                            |
| Parquet       | Schema is read from file metadata                   |

***

## Writing to S3 (destination)

Zwiron writes data to S3 as Parquet files by default (configurable to CSV or JSON). Files are named with a timestamp and partition key to avoid collisions.

Output path pattern:

```
s3://your-bucket/path-prefix/<table-name>/<date>/<timestamp>.parquet
```

***

## Troubleshooting

<AccordionGroup>
  <Accordion title="Access Denied">
    Verify the IAM policy includes the correct bucket ARN. Note that `ListBucket` applies to the bucket ARN (`arn:aws:s3:::bucket-name`) while `GetObject`/`PutObject` apply to the object ARN (`arn:aws:s3:::bucket-name/*`).
  </Accordion>

  <Accordion title="Files not being detected">
    Check the path prefix — it must match exactly. S3 is case-sensitive. Also ensure the file extension matches the configured format filter.
  </Accordion>

  <Accordion title="Agent not in the same region as the bucket">
    Cross-region S3 access works but incurs data transfer costs. For best performance, run the agent in the same AWS region as your bucket.
  </Accordion>
</AccordionGroup>