Amazon S3 Connector

Amazon S3 is an object storage service. Zwiron can read files from S3 (as a source) or write files to S3 (as a destination). Supported file formats include CSV, JSON (newline-delimited), and Parquet.

Supported sync modes

As source	Full Refresh	Incremental	CDC
	✅	N/A (no native date-filtered list API)	N/A

As destination	Write modes
	Append, Replace

Connection details

Field	Required	Description
Bucket	Yes	S3 bucket name
Region	Yes	Region of the bucket (e.g. `ap-southeast-2`)
Access key ID	Yes	IAM access key (`AKIA…`)
Secret access key	Yes	IAM secret key
Path prefix	No	Optional prefix to scope reads/writes (e.g. `data/exports/`)
Endpoint	No	Leave blank for AWS; set only for MinIO/R2-compatible stores

Setting up IAM permissions

Create an IAM policy with the minimum required permissions:

For source (read-only)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

For destination (read-write)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Reading from S3 (source)

File discovery

Zwiron lists all files in the configured bucket and prefix. You can filter by:

File extension — only read .csv, .json, or .parquet files
Path pattern — glob-style pattern (e.g. orders/2024-*/*.parquet)

Supported formats

Format	Notes
CSV	Configurable delimiter, header row, quote character
JSON (NDJSON)	One JSON object per line
Parquet	Schema is read from file metadata

Writing to S3 (destination)

Zwiron writes data to S3 as Parquet files by default (configurable to CSV or JSON). Files are named with a timestamp and partition key to avoid collisions. Output path pattern:

s3://your-bucket/path-prefix/<table-name>/<date>/<timestamp>.parquet

Troubleshooting

Access Denied

Verify the IAM policy includes the correct bucket ARN. Note that ListBucket applies to the bucket ARN (arn:aws:s3:::bucket-name) while GetObject/PutObject apply to the object ARN (arn:aws:s3:::bucket-name/*).

Files not being detected

Check the path prefix — it must match exactly. S3 is case-sensitive. Also ensure the file extension matches the configured format filter.

Agent not in the same region as the bucket

Cross-region S3 access works but incurs data transfer costs. For best performance, run the agent in the same AWS region as your bucket.

Last modified on September 4, 2026

Getting Started

Guides

Core Concepts

Pipelines

Connectors

Platform

Support

Amazon S3 Connector

Supported sync modes

Connection details

Setting up IAM permissions

For source (read-only)

For destination (read-write)

Reading from S3 (source)

File discovery

Supported formats

Writing to S3 (destination)

Troubleshooting

​Supported sync modes

​Connection details

​Setting up IAM permissions

​For source (read-only)

​For destination (read-write)

​Reading from S3 (source)

​File discovery

​Supported formats

​Writing to S3 (destination)

​Troubleshooting

Supported sync modes

Connection details

Setting up IAM permissions

For source (read-only)

For destination (read-write)

Reading from S3 (source)

File discovery

Supported formats

Writing to S3 (destination)

Troubleshooting