Unstructured Text
Remove Duplicate Lines
Keep only the first copy of each line in a list, without changing the order of what remains.
When to use it
Typical uses: Cleaning email or customer lists before importing them into a CRM; Removing repeated keywords from SEO research exports; De-duplicating URLs, product codes or tags.
In a pipeline
Text InputRemove Duplicate LinesText Output
Settings
| Setting | Type | Required | Default or hint |
|---|---|---|---|
| Text column | select | — | Text |
| Ignore upper/lowercase differences | checkbox | — | false |
| Ignore extra spaces when comparing | checkbox | — | false |
| Generate New Column | checkbox | — | false |
| New Column Name | text | — | Default: <column> Result |
| Insert After | select | — | After the column |
| When a value cannot be converted | select | — | stop |
Example
Remove Duplicate Lines: an example
- Ignore upper/lowercase differences=Ticked
- Ignore extra spaces when comparing=Ticked
| Text |
|---|
| apple banana Apple cherry banana cherry |
| Text |
|---|
| apple banana cherry |
With both options ticked:
Worth knowing
- Does the tool sort my list? No. The original order is kept. If you also want the result in alphabetical order, run it through Sort Lines A–Z afterwards.
- Are blank lines treated as duplicates? Yes. Only the first blank line is kept. To remove all blank lines, use Remove Empty Lines.
- How many lines can it handle? It comfortably handles hundreds of thousands of lines. The only limit is your device’s memory, because everything runs in your browser.
- Works on any text column, not only Text Input's: pick it under Text column. Tick Generate New Column to keep the original next to the result.