Remove Duplicate Lines

What this tool does

Remove Duplicate Lines cleans lists of emails, keywords, URLs or log entries. It compares lines with or without case and surrounding spaces, keeps the first or the last occurrence in original order, can list only the lines that never repeat or only those that do, and can append how often each line appeared.

Lists collected from several sources almost always contain repeats: the same email address exported from two forms, a keyword that appears in three research tools, or a URL logged hundreds of times. Sorting and scanning by eye misses near-duplicates such as “Apple” and “apple ” with a trailing space, and spreadsheet functions need the data in columns first.

This tool works line by line and keeps your original order, which matters for ranked lists and logs. Each line is turned into a comparison key, optionally trimmed, with repeated inner spaces collapsed and case folded, and only the first (or last) line with each key is kept. Because it uses a hash set, a list of a few hundred thousand lines is processed in well under a second, all in your browser.

How to use Remove Duplicate Lines

1. Paste your list into Lines, one item per line, or open a .txt or .csv file.
2. Under Keep, choose First occurrence, Last occurrence, Only lines that appear once, or Only lines that are repeated.
3. Turn Case-sensitive comparison on if “Apple” and “apple” are genuinely different items.
4. Leave “Ignore leading and trailing spaces” on to catch copies with stray spaces.
5. Switch on “Append how many times each line appeared” to see counts next to each line.
6. Check the Lines removed figure, then copy or download the cleaned list.

Tips for better results

• Choose Last occurrence for logs or changelogs where the most recent entry is the one that matters.
• “Only lines that are repeated” is a quick way to find which records were imported twice.
• Email addresses are case-insensitive in practice, so leave case sensitivity off when cleaning mailing lists.
• Run Sort Lines afterwards if you also want the unique list in alphabetical order.

Common use cases

• Merging subscriber lists from two newsletter platforms without sending anyone two copies.
• Cleaning keyword research exports before grouping them.
• Finding duplicate order numbers or IDs in a pasted column.
• Reducing a noisy log file to its distinct messages.

Common mistakes to avoid

• Deduplicating whole CSV rows when only one column should be unique. Extract that column first or use the CSV Deduplicator.
• Leaving case sensitivity on for data typed by people, which keeps “London” and “london” as separate entries.
• Expecting near matches such as “Ltd” and “Limited” to merge; only lines that are equal after the chosen normalisation are treated as duplicates.