Duplicate Line Remover & Sorter
Remove duplicate lines, sort alphabetically or numerically, trim whitespace, reverse, number lines and clean up any text list instantly. Works great for IPs, domains, keywords, log entries, CSV columns and more.
What Happens to the Data You Paste In
Deduplication tools frequently get fed exactly the kind of data that shouldn't leave a laptop unexamined — internal IP ranges, VPN allowlists, customer email exports, or an incident's worth of masked-but-still-sensitive log lines. Everything here runs as plain JavaScript in the browser tab: the input never touches a network request, there is no server-side processing step, and closing the tab leaves nothing behind. That matters for a security-conscious workflow in a way that a random web-based "text tool" often doesn't guarantee — many similar-looking utilities online quietly proxy input through a backend for analytics or ad-targeting purposes, which is a real consideration when the input is a firewall allowlist or a list of internal hostnames.
The practical implication is that this tool is safe to use on data an engineer wouldn't want to paste into an arbitrary SaaS product, but it is not a substitute for handling genuinely regulated data (PII subject to compliance obligations, for instance) according to whatever policy governs that data class — "runs client-side" reduces exposure, it doesn't eliminate the need to follow a data-handling policy that may still apply to what's on screen.
An Old Idea With a New Interface
Removing duplicate lines from text predates this tool by decades — Unix's uniq utility, which only collapses adjacent duplicate lines (hence the near-universal habit of piping through sort first), has shipped with Unix-like systems since the 1970s. The underlying operation this tool performs is functionally the same job modernized: build a collection that can only hold one copy of each distinct value, then read the original list back out through it. JavaScript's native Set object is exactly that collection — insertion order is preserved, and adding a value that's already present is a silent no-op — which is what lets this tool dedupe a list without needing to sort it first, unlike the classic uniq command it's conceptually descended from.
When the Output Looks Wrong
The most common "why didn't this dedupe?" report turns out to be invisible whitespace: a trailing space or a stray carriage return character (common in CSV exports saved on Windows) makes two visually identical lines compare as distinct, so both survive deduplication untouched. Trimming whitespace before removing duplicates — not after — resolves this, and is why the trim option in this tool is applied ahead of the dedupe pass rather than as a cosmetic final step.
A second source of confusion is sort order on IP addresses or version strings: alphabetical sorting places 10.0.0.1 before 2.0.0.1 because it compares character by character rather than numeric value, which makes a manually reviewed CIDR list look subtly wrong even though nothing was actually lost. If a merged list needs octet-aware ordering, treat the alphabetical sort here as a starting point and expect to re-sort numerically downstream, or skip the sort and rely on a purpose-built CIDR tool for true numeric ordering.
The One-Liner Version
On any Unix-like system, the direct command-line equivalent of this tool's default mode is sort -u file.txt, which sorts and deduplicates in a single pass. To dedupe while preserving original order (rather than sorting), the idiom is awk '!seen[$0]++' file.txt, which is functionally identical to what this tool's default "preserve order" mode does under the hood, just expressed as an AWK associative-array trick instead of a JavaScript Set. Neither command gives you the delimiter conversion, line numbering, or one-click whitespace trimming this tool bundles together, which is the main reason to reach for a browser tool instead of a terminal when the input didn't originate in a file you already have local shell access to — a pasted Slack thread or a browser-copied table, for instance.
Where This Actually Gets Used
- Firewall and WAF rule consolidation: merging IP allowlists or blocklists sourced from multiple security vendors before loading the combined list into a firewall or security group.
- Multi-source
.envcleanup: combining environment files assembled from several teams or templates and removing duplicate variable definitions before they cause unpredictable "which value actually loaded" runtime bugs. - Incident log triage: collapsing a repetitive grep dump down to the distinct error messages actually present, so a responder isn't scrolling past the same line fifty times while trying to spot what's new.
- Dependency and image-tag normalisation: deduplicating package names or container tags gathered from several Dockerfiles or lockfiles before generating one consolidated dependency report.
Frequently Asked Questions
What operations does this tool support?
The tool supports removing duplicate lines, sorting the result alphabetically (A–Z or Z–A), sorting by line length, sorting numerically, trimming leading and trailing whitespace from each line, removing blank lines, reversing the line order, and numbering each output line. Input and output delimiters are configurable — you can paste a comma-separated list or a tab-separated export and receive newline-delimited output, or vice versa. These operations can be combined: for example, you can trim whitespace, remove duplicates, and then sort alphabetically in a single pass.
Can I process very large lists?
Yes. All processing runs entirely in your browser using JavaScript — no data is uploaded to any server, making it safe for sensitive lists such as internal IP ranges or customer email addresses. The tool handles tens of thousands of lines comfortably on modern devices. For very large inputs (hundreds of thousands of lines), operations like sort-by-length involve an O(n log n) comparison pass and may take a moment to complete; the tool recalculates on every input change so you may notice a brief delay. For extremely large datasets, the equivalent command-line tool is sort -u on Linux, which is optimised for large file processing.
Does duplicate removal preserve the original order of lines?
By default, duplicate removal keeps the first occurrence of each line and discards all subsequent duplicates, preserving the original insertion order of the list. This behaviour is important for use cases like ordered IP allowlists, ranked search results, or configuration files where line order is semantically significant — the cleaned output will have the same sequence as the input, just with repeats removed. If you enable a sort option, the order is replaced by the sort criterion. For cases where you want to find and count duplicates rather than remove them, check the "show duplicate count" option to see how many times each line appeared in the input.