About This Tool
CSV Deduplicator parses actual CSV before comparing field arrays. Use all fields as a key or combine one or more column indexes. It keeps the first record by default and excludes the header from deduplication.
How to Use It
- Paste CSV or choose a UTF-8 file, then check the delimiter and header settings.
- Choose whole-row matching or select fields using names labeled with column numbers.
- Click Remove Duplicates and check the original, remaining, and removed record counts.
- Review the limited table preview, then copy or download correctly escaped CSV.
Examples
- a,b / 1,2 / 1,2 / 2,3 retains two data records after whole-row deduplication.
- Matching id in id,name / 1,Tom / 1,Jack / 2,Amy retains 1,Tom and 2,Amy.
- Duplicate headers name (column 1) and name (column 2) can be selected independently.
Rules and Limits
The first row is a header by default. Deduplication keeps first occurrences in their original order. Keys use unambiguous serialization of field arrays, avoiding collisions from simply joining fields with a separator. Fields are compared exactly, without implicit trimming or numeric rewriting. The same Papa Parse core generates output, quoting fields containing delimiters, quotes, or line breaks. Input limits: 2 MiB, 100,000 records, 500 columns, and 500,000 cells. Preview limits: 500 rows, 100 columns, and 10,000 cells.
Common Uses
- Remove records repeated in data exports.
- Keep the first record for each ID, email address, or combination of fields.
- Clean CSV without overwriting duplicate headers.
Privacy
Text processing, file reading, conversion, and previews all run locally in this browser. Input is not uploaded or automatically saved. No backend, database, online API, or third-party CDN is used. Copying and downloading also happen locally.
FAQ
What does whole-row matching mean?
Every field and its position must match. Selected-column mode compares only the fields you choose.
Which duplicate record is kept?
The complete first occurrence is kept, including its other fields. Output order remains stable.
Can duplicate headers cause data loss?
No. Internally, columns are identified by index, and the interface includes column numbers rather than relying on unique names.
How are quotes handled in output?
Fields containing a delimiter, quote, or line break are quoted according to CSV rules. A quote inside a field becomes two quotes.
Does the tool modify formulas or field whitespace?
Fields are preserved as written, without automatic trimming or rewriting. When importing into a spreadsheet, select text or other column types appropriate to your data.