Clean a text file
Clean text, remove empty and duplicate lines, then download the result.
What does this template do?
Clean text, remove empty and duplicate lines, then download the result.
This template cleans an entire text file (TXT, CSV read as text, Markdown or log): spaces are normalised, empty and repeated lines are removed, then a new TXT file is downloaded. It saves you from opening a large file in an editor to fix it by hand.
Pipeline steps
This template automatically chains 5 steps in the Workspace. You can review every setting before running it.
- 1Import a text file
- 2Clean up spaces and lines
- 3Remove empty lines
- 4Remove duplicate lines
- 5Download as TXT
Why the steps come in this order
Space cleaning runs first so lines that only differ by spaces become identical. Removing empty lines also drops those that contained only spaces, then deduplication works on clean lines.
When should you use this template?
- Deduplicate a list exported as TXT before importing it into a tool.
- Slim down a notes file full of empty lines.
- Prepare a list of URLs or identifiers extracted from another system.
Local processing in your browser
When the selected modules support local processing, your files stay on your device while the Pipeline runs.
Settings to check
- Source file: TXT, CSV, Markdown or log, read line by line.
- Output file name: “resultat.txt” by default.
- The three cleaning steps have no settings.
Example
- Before
- 5,000-line file of identifiers, with 600 empty and 1,100 repeated lines.
- After
- resultat.txt with 3,300 unique identifiers, in their original order.
Limits to know
- In a log, two identical events at different times are merged: do not use this template to count occurrences.
- A CSV is treated as text: a repeated header line is removed, but columns are not analysed.
- The kept lines stay in file order, without sorting.