Removing duplicate channels from an M3U playlist
Duplicates are easy to remove when the addresses match and hard to judge when they do not. The difference between those two cases is the whole problem, and it is why a button that removes everything suspicious is worse than one that removes almost nothing.
How a playlist ends up with duplicates
Three routes produce most of them, and they are worth telling apart because only one of them leaves the file in a state where automatic removal is obviously safe.
Merged exports
You combine two exports of the same service, and the channels both lists contain
appear twice. The two copies usually carry the same metadata and the same
address, sometimes differing only in the order of the tvg-*
attributes, which is why a comparison on the address alone is the right test.
Provider re-adds
A provider migrates infrastructure and appends the channels again rather than replacing them, so the list grows a second copy alongside the old one. These are exact duplicates at the moment they are added — and they stop being exact the next time the provider rotates the stream addresses, which is why a list checked once can look clean a month later.
Manual edits
A channel pasted in again because the first copy was named badly, with the old one left in place. This is the case where the two entries are genuinely the same stream but the better metadata sits on the copy you would keep, so removing the first occurrence rather than the second is usually the wrong choice.
When it is not a duplicate
Two entries can describe one channel and still not be duplicates. Most of what looks like duplication is one of these cases:
| Case | What it looks like | Remove automatically? |
|---|---|---|
| Two hosts, one feed | Same channel name, different domains | No — one host may work when the other does not |
| Quality tiers | Same name, HD / SD / FHD in the name or URL | No — two streams of one channel, by design |
| Session or token URLs | Same path, different query string | No — a different query may be a different session or region |
| Regional variants | Identical name, different tvg-id | No — the ids point at different guide entries |
| Byte-identical entries | Same address, same metadata | Yes — the only case where the judgement is trivial |
Removing the middle rows automatically is a guess, and a costly one: you may delete the only copy that works from your network. A tool that deduplicates on the address alone is making a defensible, narrow judgement, and anything wider needs a person who knows what the list is for.
One list, one removal
Take a list where the same stream appears twice with different metadata. The first copy has no logo; the second was added later with one:
#EXTINF:-1 tvg-id="news.one" group-title="News",News One
https://example.com/news-one.m3u8
#EXTINF:-1 tvg-id="news.two" group-title="News",News Two
https://example.com/news-two.m3u8
#EXTINF:-1 tvg-id="news.one" tvg-logo="https://example.com/logos/one.png" group-title="News",News One
https://example.com/news-one.m3u8Exact matching finds one duplicate and keeps the first entry, so the export is:
#EXTINF:-1 tvg-id="news.one" group-title="News",News One
https://example.com/news-one.m3u8
#EXTINF:-1 tvg-id="news.two" group-title="News",News Two
https://example.com/news-two.m3u8The logo is gone with the second copy, because the rule kept the first entry whole rather than merging the two. That is the trade-off of a deduplicator that matches on the address alone: it is predictable, it never invents metadata, and it cannot tell which copy had the better name. When the later copy is the better one, delete the earlier entry by hand in the editor's table before running the duplicate action, or rename the surviving entry afterwards — two steps, but the result is the one you wanted rather than the one the rule produced.
The same narrowness is what makes the result safe. A tool that tried to merge the two entries would have to decide which name, group and logo to keep, and any rule for that decision is a guess about a file it cannot see the rest of.
What exact matching actually compares
The validator's duplicate count, the converter's remove-duplicates option and the editor's button all use one rule: two entries are duplicates when their URL strings are identical. Not similar, not equal after cleaning up — identical. Each line is trimmed before it reaches the comparison, so a trailing space or a Windows line ending cannot manufacture a difference; after that the comparison is character for character, and the first entry with a given address is the one kept.
URLs are more forgiving than that rule. Two addresses can reach the same stream and still differ as text, which is why the table below matters: “not a duplicate” is correct about the strings and sometimes wrong about the world.
| Difference | Same stream in practice? | Counted as a duplicate? |
|---|---|---|
Host in a different case, EXAMPLE.COM against example.com | Yes — host names are case-insensitive | No |
The default port spelled out, :443 | Yes | No |
%7E where the other copy has ~ | Yes — percent-encoded unreserved characters are equivalent | No |
| A trailing slash | Usually yes, depending on the server | No |
| Query parameters reordered | Depends entirely on the server | No |
| A different token in the query string | Often no — a different session or account | No, and correctly |
RFC 3986 defines these normalisations for comparing URLs, and this tool deliberately implements none of them. Applying them would mean deciding which differences are irrelevant, and for the last two rows there is no safe answer: two tokens can be two sessions, or one session listed twice. Exact matching keeps both and can be trusted to have removed only what nobody could argue about.
Checking before you delete
Convert the playlist to CSV and sort by name. Real duplicates sit next to each other, and the columns show you whether the addresses actually match or merely look similar. Sorting by URL host is the other useful view: if nearly every channel comes from two hosts, the list is one provider plus a backup, and “duplicates” across hosts are by design.
The converter produces that CSV without uploading anything, and the validator counts exact duplicates before you touch the file, so you know whether the clean-up is worth starting.
Removing them
Load the file into the editor and use the remove-duplicates action: it compares stream addresses exactly and keeps the first entry for each. The exported file is rebuilt from the parsed entries, so attributes survive and the entry-URL pairing stays intact — the failure mode of deleting lines by hand in a text editor.
That pairing is the reason exact matching is trustworthy. An
#EXTINF line and the URL below it are one record, so two records
with the same URL address the same stream; nothing about the name can make them
different streams.
Verifying the result by numbers
Three numbers make a clean-up checkable, and they come from the tools rather than from reading the list by eye. The validator reports entries, unique addresses and duplicates; the editor reports how many URLs its action removed; the export can be validated again to count what is left.
| Number | Where it comes from | What it must equal |
|---|---|---|
| Duplicates | The validator's report | Entries minus unique addresses |
| Removed | The editor's status message | The duplicates figure from the validator |
| Entries afterwards | Re-running the validator on the export | Unique addresses, with duplicates at zero |
When the editor reports a different number from the validator, the usual cause is that the file changed between the two runs: an edit, a re-sort, or a paste from a different export. Both tools parse with the same code, so a matching count is evidence that the same file went through both, not a coincidence.
Keep the original until the cleaned file has played
If a channel you expected is gone, the near-duplicate case is the likely reason, and the original still has it. Duplicate removal is reversible only while you still have the source file. The editor never writes to the file you loaded — it produces a new download — so keeping both versions costs nothing.
What deduplication cannot fix
Removing duplicates changes one thing: how many times a stream is listed. It does not make a playlist more reliable, and the problems people hope it solves are usually elsewhere.
- Mirrors. Two hosts carry the same channel and one of them is blocked on your network. Exact matching keeps both, which is the sensible outcome: deleting the wrong one is how a channel vanishes from a list that was working.
- Rotated addresses. A provider moves a stream and re-adds the channel instead of replacing it. Old address, new address: nothing is detected, and if both answer you now have two entries that work.
-
Regional variants. One name with different
tvg-idvalues is two guide entries, not a duplicate — which is why the validator reports repeated names as information rather than as a fault. - Buffering and stalls. A duplicated entry causes neither. When a channel stalls, the duplicate is the copy that still works.
The one case exact matching cannot reach is worth stating plainly: two entries with different addresses that play the same stream. Deciding that means watching both, which no file-only tool can do — the same limit that stops these tools from telling you whether a stream is live at all.
Sources
-
RFC 8216, HTTP Live Streaming
— §4.3.2.1 describes how an
#EXTINFtag pairs with the single URI that follows it, the rule that makes address matching an exact test. - RFC 3986 §6.2.2, Syntax-Based Normalization — the case, percent-encoding and default-port rules for comparing two URLs, none of which this tool applies.