Remove a repeated import
Compare all columns to remove identical records after combining files, without confusing separate orders that happen to belong to the same customer or share another non-unique field.
CSV Deduplicator
Detect duplicate rows in a CSV using one or more selected columns, keep the useful records, and download a cleaned file.
- Clipboard
Data not sent
Leave empty to compare every column
Deduplicated CSV
Was this tool useful?
How does this tool work?
Two records may count as duplicates because every cell matches or because they share a key despite containing different information. CSV deduplication lets you choose that definition before creating a smaller file. Use it to remove repeated records after combining exports or to keep one version per identifier. Unlike a text-line duplicate analyser, it understands CSV fields, can compare several columns and returns the retained records. Deciding whether two customers or transactions should count as the same entity remains a business decision for you.
All columns are compared when the key field is blank. Otherwise, enter exact headings or positions starting at 1, separated by commas. Multiple key columns are combined, so a repeated record must match on every chosen field. A repeated heading selects its first occurrence; use positions for another occurrence or for a heading containing commas. Without a header, select positions. A position outside the table must be corrected before processing rather than treated as a missing field.
The delimiter is detected from commas, semicolons, tabs and vertical bars, or you can select it explicitly. The header setting starts enabled and keeps the first record outside deduplication. Reading supports quoted cells, internal delimiters and quoted line breaks. A leading BOM is removed. Surrounding spaces are ignored by default when constructing keys, whereas case remains significant. Enabling case-insensitive comparison groups A and a together. These settings affect matching, not the cell contents of the record eventually retained.
The first occurrence is retained by default, or the last when requested. In both modes, groups retain the order in which their keys first appeared. Keeping the last version therefore does not sort output by date or by that final record's position. Fields from discarded versions are not merged. Numbers are still compared as text, so 1 and 1.0 do not become equal. A missing cell and an empty cell both contribute an empty key value. The summary counts removed and retained data records, excluding the header.
How to use this tool
Confirm how the export is read
Paste or import the CSV, select its delimiter if necessary and confirm the header setting. Identify the fields that actually determine whether your records should be considered unique.
Set the key and retention rule
Leave key columns blank for complete-record comparison or enter the required columns. Choose first or last occurrence and decide whether surrounding spaces and letter case should distinguish values.
Review the versions being discarded
Run deduplication and compare the removal count with your expectation. Inspect retained values, especially fields that differ outside the key, before copying or exporting the resulting CSV.
Use cases
Keep one version of a reference
Choose the product identifier as the key and explicitly decide whether to retain its first or last record. The engine does not automatically identify which version is newest.
Use a composite business key
Combine reference and date when each product should occur once per day. Matching the reference alone is then insufficient to make two records duplicates and remove one of them.
Examples
Remove an entirely repeated record
Comparing all columns and keeping the first occurrence removes one record. Both A and Alice must match together: a different name associated with A would remain distinct in this mode. Two data records remain in the output.
Input
id;name
A;Alice
B;Ben
A;AliceResult
id;name
A;Alice
B;BenKeep the last name for an identifier
With id as the key and last occurrence selected, Alicia replaces Alice in group A. Group A remains before B because its key first appears earlier in the input. The engine does not reorder groups by their final occurrence.
Input
id;name
A;Alice
B;Ben
A;AliciaResult
id;name
A;Alicia
B;BenTips and best practices
- Do not use an empty identity field as a key without considering the consequences: several unknown records may be grouped and only one retained. Complete or isolate those records before deduplication.
- To retain the most recent record, first sort an appropriate date field in a comparable format, then keep the last occurrence. The last option does not search other cells for dates.
- Columns outside the key do not protect a record against removal. Keep the source data so you can investigate an important difference such as an amount, address or status in a discarded version.
Frequently asked questions
Does ignoring spaces also clean the exported cells?
No. Surrounding spaces are removed only while determining whether keys match. The retained record keeps its original values, including spaces. Apply an explicit cleanup afterwards if you want to change those cell contents.
Does first occurrence mean the oldest version?
Only when the input is already ordered chronologically. The tool follows input order without interpreting dates. First and last refer to record positions in the file, rather than to their age in your business process.
Can I retain information from every grouped record?
This operation keeps one complete record and discards the others. It does not fill empty cells from another version or add amounts together. Consolidating fields according to priority rules requires a different transformation.
Why do apparently similar numbers remain distinct?
Keys are textual: 1, 1.0 and 01 are different strings. Case-insensitive comparison does not change that. Normalise them only when they truly represent the same entity rather than distinct identifiers whose spelling must be preserved.
Related tools
Keep going with these tools
Tools similar or complementary to this one.
Similar tools
Find duplicate lines, count occurrences and export a clean summary.
Data & spreadsheetsUse this tool
Sort CSV rows by a selected column in ascending or descending order.
Data & spreadsheetsUse this tool
Remove duplicates, empty rows and unnecessary spaces.
Data & spreadsheetsUse this tool
Read next
GuideBest practicesBeginner
Compare two CSV or Excel files and identify the differences
Learn how to compare two versions of a table using a stable key and distinguish added, removed and modified records.
29 August 20262 min
GuideBest practicesBeginner
How to clean and prepare a CSV or Excel file before using it
Learn a practical order for removing empty rows, trimming values, standardizing headers, checking identifiers and validating a CSV or Excel dataset.
29 August 20263 min
GuideBest practicesBeginner
Merge CSV or Excel files without misaligning columns
Learn how to align headers, verify schemas and combine several CSV or Excel files while avoiding duplicated or shifted columns.
29 August 20262 min
GuideBest practicesBeginner
Remove duplicates and validate CSV data without deleting the wrong rows
Understand how to define a reliable duplicate key, distinguish exact and business duplicates, and validate a CSV before and after deduplication.
29 August 20262 min