Cleaning
Remove Duplicates
Drops rows whose key values were already seen, keeping the first occurrence of each.
When to use it
Use it after stacking files or merging branches, when the same record can arrive more than once and only one copy should continue.
In a pipeline
File InputTrim WhitespaceRemove DuplicatesExport CSV
Settings
| Setting | Type | Required | Default or hint |
|---|---|---|---|
| Compare Columns | select | — | the whole row (every column) |
Example
One row per email address
- Compare Columns=Email
| Order | |
|---|---|
| A-1 | mia@acme.com |
| A-2 | mia@acme.com |
| A-3 | leo@acme.com |
| Order | |
|---|---|
| A-1 | mia@acme.com |
| A-3 | leo@acme.com |
A-2 repeats an email already seen on A-1, so the first row is the one kept.
Worth knowing
- The first matching row wins, so sort upstream if a particular copy should survive.
- Pick each key column from its dropdown and use Add column for more; leaving every dropdown empty compares the whole row.
- Values are compared as text, so Acme and acme count as different keys.
- Run Trim Whitespace or Change Case first when the duplicates differ only by spacing or casing.