Remove Duplicate Lines

Paste a list of emails, keywords or URLs and remove duplicates, blank lines and sort the result.

   

How the deduplication works

Each line is compared as a whole string against the lines already seen, keeping the first occurrence and discarding later ones. That preserves the original order, which matters when the list has meaning - a sequence of steps, or rows that must stay aligned with another column.

Two options change the behaviour. Case-insensitive matching treats 'Apple' and 'apple' as the same entry, which is usually what you want for email addresses and domains. Trimming spaces first removes hidden differences caused by trailing spaces, the single most common reason a 'duplicate' is not detected.

Common uses

Email lists: merge two exports and strip addresses that appear in both. Keyword lists: deduplicate after combining results from several research tools. URL lists: clean a crawl export before feeding it into a screaming-frog style audit.

SKUs and inventory: catch the same part number entered twice with different formatting. Log files: collapse repeated error lines to see how many distinct errors actually occurred, which is often far fewer than the raw line count suggests.

Keep the first or the last occurrence?

Keeping the first is right when earlier entries are more authoritative - the first time a record appears in a merge is often the original source. Keeping the last is right when later entries carry corrections, such as an updated price for the same SKU.

This tool keeps the first. If you need the last, reverse the lines before and after using the tool, or sort the list so the preferred version appears first.

Frequently asked questions

Is the duplicate check case-sensitive?

By default no - 'Apple' and 'apple' are treated as duplicates, which is correct for emails and domains since those are case-insensitive by definition. Untick the option if your data is case-sensitive, such as product codes or hashes.

Why are two identical-looking lines not being removed?

Almost always invisible whitespace: a trailing space, a tab, or a non-breaking space pasted from a web page. Enable 'trim spaces' and run it again. If they still survive, one of them contains a different Unicode character that looks identical on screen.

How many lines can it handle?

Tens of thousands comfortably, because the comparison runs in memory inside your browser. Beyond roughly 100,000 lines the tab will start to lag - at that point a command-line tool such as awk is the better instrument.

Can it remove duplicates across multiple columns?

No, it works line by line on plain text. For spreadsheets, use Data > Remove Duplicates in Excel, or the UNIQUE() function in Google Sheets, which can compare whole rows rather than single strings.

Is my list uploaded anywhere?

No. The entire operation happens in your browser and nothing is sent to a server. That matters for email lists and customer data, which should never be pasted into a tool that processes it remotely.

Everything runs inside your browser. Nothing you type is uploaded, there is no signup, and the page keeps working if you go offline.
Sponsored — ad slot reserved

More free tools

Related guides

Related searches

remove duplicate linessort linesclean text list