Remove duplicates

Remove duplicates from a list of text values while keeping the first occurrence of each one. The tool handles one entry per line and lets you decide whether capital letters and spaces at the edges of a line count in the comparison.

Each non-empty line is compared with the values already seen. When two lines are treated as identical, only the first is kept in the result. Two options, both on by default, ignore differences between upper and lower case and trim spaces at the start and end of each line before comparing; switch them off for a strict comparison. Empty lines are dropped. The counter of removed duplicates does not include empty lines: it only measures repeated occurrences. The tool does not correct spelling or recognise synonyms. For example, “Leeds” and “leeds” are grouped when case is ignored, but “Leeds” and “Leds” stay separate. Keep your original list if the repetitions carry statistical meaning. To decide whether two values should count as the same, look at the quality of the source. In a contact directory, differences in capitals rarely matter; in a list of technical codes, they may distinguish valid identifiers. Trimming edge spaces is handy after a copy and paste, but it never touches spaces in the middle of an expression. If a line appears three times, the tool keeps the first occurrence and removes the others according to the selected rules. It never matches values that are merely similar. For an audit, keep a copy of the original list and note the options you used so you can explain the result. Removing duplicates is not the same as counting occurrences: if you care about how often each value appears, use a value counter instead.