> ## Documentation Index
> Fetch the complete documentation index at: https://docs.zwiron.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipeline Validation

> Run deep accuracy audits comparing source and destination by rows, content, columns, and custom assertions.

**Validation** runs a multi-dimensional comparison between a pipeline's source and destination. It goes beyond the automatic [row-count check](/concepts/schema-validation) that runs after every sync and produces a scored report across four accuracy dimensions.

***

## Accuracy dimensions

| Dimension               | Score of 100% means                                                  |
| ----------------------- | -------------------------------------------------------------------- |
| **Structural accuracy** | Every table has the same number of rows at source and destination    |
| **Content accuracy**    | Row-level values match across all sampled or checked tables          |
| **Schema fidelity**     | Column names, types, and counts match between source and destination |
| **Semantic accuracy**   | All custom assertion rules you defined returned zero rows            |
| **Overall accuracy**    | Weighted combination of all four dimensions                          |

***

## How validation differs from schema validation

|                    | [Schema Validation](/concepts/schema-validation) | Validation (this page)                        |
| ------------------ | ------------------------------------------------ | --------------------------------------------- |
| **Trigger**        | Automatic after every sync                       | Manual (on demand)                            |
| **What it checks** | Row counts only                                  | Row counts, content, schema, and custom rules |
| **Result**         | Pass / mismatch per table                        | Scored report with per-table breakdown        |

***

## Running a validation

1. Open a pipeline in **Jobs**
2. Click the **Data Quality** tab
3. Click **Run Validation**

Zwiron connects to both source and destination through the agent, executes the comparison, and displays the report when complete. Duration depends on table size and the selected mode.

### Validation options

| Option            | Default    | Description                                           |
| ----------------- | ---------- | ----------------------------------------------------- |
| `validation_mode` | `full`     | `full` checks every row; `sample` checks a percentage |
| `sample_percent`  | `10`       | Percentage of rows to check when using `sample` mode  |
| `timeout_seconds` | `300`      | Maximum time to wait before aborting                  |
| `tables`          | all tables | Limit validation to specific table names              |

***

## Reading the report

### Accuracy scores

The top of the report shows the five accuracy scores as percentages. A score of **100%** means the dimension passed completely. Any value below 100% indicates divergence — the details below tell you where.

### Per-table breakdown

Each table in the pipeline appears as a row:

| Column               | Description                                                     |
| -------------------- | --------------------------------------------------------------- |
| **Table**            | Table name                                                      |
| **Status**           | `match` ✅, `mismatch` ❌, or `error` ⚠️                          |
| **Source rows**      | Row count at source                                             |
| **Destination rows** | Row count at destination                                        |
| **Diff rows**        | Number of rows where content differed (content validation only) |

### Schema comparison

When schema fidelity is below 100%, the report lists:

* **Missing columns** — present in source, absent at destination
* **Type mismatches** — columns where the type differs between source and destination
* **Missing count** and **type mismatch count** totals

***

## Semantic rules

Semantic rules are SQL assertions you define per pipeline. Each rule must return **zero rows** to pass.

### Creating a semantic rule

In the **Data Quality** tab of a pipeline:

1. Scroll to **Semantic Rules**
2. Click **+ Add Rule**
3. Enter a **name** (e.g. `Orders must have a customer`)
4. Enter a **SQL expression** that returns rows only when data violates the rule:

```sql theme={null}
-- Fails if any order has no customer
SELECT * FROM orders WHERE customer_id IS NULL

-- Fails if any price is zero or negative
SELECT * FROM products WHERE price <= 0

-- Fails if a shipped order has no tracking number
SELECT * FROM orders
WHERE status = 'shipped' AND tracking_number IS NULL
```

5. Click **Save**

Rules are evaluated automatically every time validation runs for this pipeline.

### Rule results

| Result     | Meaning                                                  |
| ---------- | -------------------------------------------------------- |
| ✅ `passed` | Query returned zero rows                                 |
| ❌ `failed` | Query returned one or more rows — data violates the rule |

***

## Common reasons for mismatches

Not all divergence indicates a bug. Review these before investigating further:

| Cause                            | Explanation                                                                                  |
| -------------------------------- | -------------------------------------------------------------------------------------------- |
| **Row filters in transforms**    | Transforms that filter rows will produce fewer destination rows than source rows — by design |
| **Deduplication at destination** | MERGE or upsert strategies deduplicate rows; destination count can be lower                  |
| **CDC lag**                      | A CDC pipeline in progress hasn't fully caught up yet                                        |
| **Eventual consistency**         | Source count was taken mid-transaction; counts converge shortly after                        |

If none of these apply and content accuracy is significantly below 100%, check the [Events log](/pipelines/events) for errors during the sync.

***

## Validation vs. Data Quality tests

|                 | Validation                              | [Data Quality](/platform/data-quality) |
| --------------- | --------------------------------------- | -------------------------------------- |
| **Scope**       | Source ↔ destination comparison         | Rules on a single table's content      |
| **Purpose**     | Verify the sync was faithful            | Verify the data itself is correct      |
| **When to use** | After a sync, to confirm data integrity | Continuously, on a schedule            |

Both complement each other — validation confirms the pipeline moved data correctly; quality tests confirm the data inside meets business expectations.

***

## Related

* [Schema Validation](/concepts/schema-validation) — automatic post-sync row count check
* [Data Quality](/platform/data-quality) — automated test rules on individual tables
* [Events](/pipelines/events) — granular log for debugging sync failures