Pipeline Validation

Validation runs a multi-dimensional comparison between a pipeline’s source and destination. It goes beyond the automatic row-count check that runs after every sync and produces a scored report across four accuracy dimensions.

Accuracy dimensions

Dimension	Score of 100% means
Structural accuracy	Every table has the same number of rows at source and destination
Content accuracy	Row-level values match across all sampled or checked tables
Schema fidelity	Column names, types, and counts match between source and destination
Semantic accuracy	All custom assertion rules you defined returned zero rows
Overall accuracy	Weighted combination of all four dimensions

How validation differs from schema validation

	Schema Validation	Validation (this page)
Trigger	Automatic after every sync	Manual (on demand)
What it checks	Row counts only	Row counts, content, schema, and custom rules
Result	Pass / mismatch per table	Scored report with per-table breakdown

Running a validation

Open a pipeline in Jobs
Click the Data Quality tab
Click Run Validation

Zwiron connects to both source and destination through the agent, executes the comparison, and displays the report when complete. Duration depends on table size and the selected mode.

Validation options

Option	Default	Description
`validation_mode`	`full`	`full` checks every row; `sample` checks a percentage
`sample_percent`	`10`	Percentage of rows to check when using `sample` mode
`timeout_seconds`	`300`	Maximum time to wait before aborting
`tables`	all tables	Limit validation to specific table names

Reading the report

Accuracy scores

The top of the report shows the five accuracy scores as percentages. A score of 100% means the dimension passed completely. Any value below 100% indicates divergence — the details below tell you where.

Per-table breakdown

Each table in the pipeline appears as a row:

Column	Description
Table	Table name
Status	`match` ✅, `mismatch` ❌, or `error` ⚠️
Source rows	Row count at source
Destination rows	Row count at destination
Diff rows	Number of rows where content differed (content validation only)

Schema comparison

When schema fidelity is below 100%, the report lists:

Missing columns — present in source, absent at destination
Type mismatches — columns where the type differs between source and destination
Missing count and type mismatch count totals

Semantic rules

Semantic rules are SQL assertions you define per pipeline. Each rule must return zero rows to pass.

Creating a semantic rule

In the Data Quality tab of a pipeline:

Scroll to Semantic Rules
Click + Add Rule
Enter a name (e.g. Orders must have a customer)
Enter a SQL expression that returns rows only when data violates the rule:

-- Fails if any order has no customer
SELECT * FROM orders WHERE customer_id IS NULL

-- Fails if any price is zero or negative
SELECT * FROM products WHERE price <= 0

-- Fails if a shipped order has no tracking number
SELECT * FROM orders
WHERE status = 'shipped' AND tracking_number IS NULL

Click Save

Rules are evaluated automatically every time validation runs for this pipeline.

Rule results

Result	Meaning
✅ `passed`	Query returned zero rows
❌ `failed`	Query returned one or more rows — data violates the rule

Common reasons for mismatches

Not all divergence indicates a bug. Review these before investigating further:

Cause	Explanation
Row filters in transforms	Transforms that filter rows will produce fewer destination rows than source rows — by design
Deduplication at destination	MERGE or upsert strategies deduplicate rows; destination count can be lower
CDC lag	A CDC pipeline in progress hasn’t fully caught up yet
Eventual consistency	Source count was taken mid-transaction; counts converge shortly after

If none of these apply and content accuracy is significantly below 100%, check the Events log for errors during the sync.

Validation vs. Data Quality tests

	Validation	Data Quality
Scope	Source ↔ destination comparison	Rules on a single table’s content
Purpose	Verify the sync was faithful	Verify the data itself is correct
When to use	After a sync, to confirm data integrity	Continuously, on a schedule

Both complement each other — validation confirms the pipeline moved data correctly; quality tests confirm the data inside meets business expectations.

Schema Validation — automatic post-sync row count check
Data Quality — automated test rules on individual tables
Events — granular log for debugging sync failures

Getting Started

Guides

Core Concepts

Pipelines

Connectors

Platform

Support

Pipeline Validation

Accuracy dimensions

How validation differs from schema validation

Running a validation

Validation options

Reading the report

Accuracy scores

Per-table breakdown

Schema comparison

Semantic rules

Creating a semantic rule

Rule results

Common reasons for mismatches

Validation vs. Data Quality tests

​Accuracy dimensions

​How validation differs from schema validation

​Running a validation

​Validation options

​Reading the report

​Accuracy scores

​Per-table breakdown

​Schema comparison

​Semantic rules

​Creating a semantic rule

​Rule results

​Common reasons for mismatches

​Validation vs. Data Quality tests

​Related

Accuracy dimensions

How validation differs from schema validation

Running a validation

Validation options

Reading the report

Accuracy scores

Per-table breakdown

Schema comparison

Semantic rules

Creating a semantic rule

Rule results

Common reasons for mismatches

Validation vs. Data Quality tests

Related