Data Quality
Quality Rules / Assertions
Checks every row against your rules — required, not-empty, range, regex and in-set. Rows that break a rule leave on the Fail pin with the broken rules written in a reason column; the rest leave on Pass.
When to use it
Use it to quarantine bad records instead of stopping the whole run: send Fail rows to a file or a Logging node for someone to fix, and let the clean rows carry on.
In a pipeline
File InputQuality Rules / AssertionsExport CSV
Settings
| Setting | Type | Required | Default or hint |
|---|---|---|---|
| Rules (JSON) | textarea | Yes | [{"column":"ID","check":"required"},{"column":"DOB","check":"range","max":"TODAY"}] |
| Severity | select | — | error |
| Reason column name | text | — | Quality Reason |
Example
A blank ID or a birth date in the future
- Rules (JSON)=[{"column":"ID","check":"required"},{"column":"DOB","check":"range","max":"TODAY"}]
| ID | DOB |
|---|---|
| 1 | 1990-04-02 |
| — | 1985-11-20 |
| 3 | 2031-01-01 |
| ID | DOB | Quality Reason |
|---|---|---|
| — | 1985-11-20 | ID is blank |
| 3 | 2031-01-01 | DOB is over the maximum (TODAY) |
Worth knowing
- Checks: "required" / "not-empty" (no blank), "range" with "min" and/or "max" (numbers, dates, or TODAY / NOW), "regex" with a "pattern" (and optional "flags"), and "in-set" with "values", e.g. ["US","CA"].
- Range, regex and in-set skip blank cells — add a required rule for the same column to catch blanks too.
- Give a rule a "message" to choose its reason text. Several broken rules are joined with "; ".
- Severity Warn keeps every row on Pass (reasons still filled in) and only reports the counts.