Is it possible to remove duplicated rows in Notepad++, leaving only a single occurrence of a line?
Asked
Active
Viewed 7.1e+01k times
14 Answers
779
Notepad++ with the TextFX plugin can do this, provided you wanted to sort by line, and remove the duplicate lines at the same time.
To install the TextFX in the latest release of Notepad++ you need to download it from here: https://sourceforge.net/projects/npp-plugins/files/TextFX
The TextFX plugin used to be included in older versions of Notepad++, or be possible to add from the menu by going to Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX -> Install. In some cases it may also be called TextFX Characters, but this is the same thing.
The check boxes and buttons required will now appear in the menu under: TextFX -> TextFX Tools.
Make sure "sort outputs only unique..." is checked. Next, select a block of text (Ctrl+A to select the entire document). Finally, click "sort lines case sensitive" or "sort lines case insensitive"
Colin Pickard
- 43,301
- 13
- 91
- 143
-
35Incredibly powerful plugin, despite its "age". Hope they will NEVER remove that one from the standard NPP plugin offer. The guy who thought about all the features in this plug-in, was kind of a "visionary". – GeertVc Sep 01 '14 at 09:32
-
Note that this method does not give any kind of warning if the file is read-only. My file was sorted anyway, so it seemed that the tool had worked, until I spotted a duplicate. Quite frustrating until I tried @stema's search & replace method, which did warn me. – JV. Dec 04 '14 at 12:44
-
7More powerful than excel. – Vasu Apr 22 '15 at 21:58
-
2Textpad does it with one key - F9 hoping NP++ can also allow hotkey for this operation. – prash Apr 12 '16 at 09:11
-
@GeertVc: was that sarcasm, zynism or something? There's no TextFX plugin in my installation – Thomas Weller Jun 01 '16 at 10:35
-
NO I don't want to sort anything – ACV Dec 20 '16 at 10:01
-
8What about Notepad++ x64 version? Plugin TextFX x64-version not exists – Geograph Jan 14 '18 at 15:22
-
i am not able to check the `Sort outputs only` option. What to do? – th3pirat3 Mar 09 '18 at 18:18
-
20TextFx is not in the 64 bit version. – Rhyous Mar 23 '18 at 00:27
-
5@Geograph And there will be no 64 bit plugin of TextFx see this [note](http://docs.notepad-plus-plus.org/index.php/TextFX%27s_Future). Therefore it would be good to know if there is an alternative plugin providing sort and duplicate removal. – Robert Oct 23 '18 at 13:32
-
The only downside is that it sorts the lines.. I don't wanna change the order of my lines. – Shayan Sep 21 '19 at 14:31
-
TextFX has been phased out - from *[TextFX's Future](http://docs.notepad-plus-plus.org/index.php/TextFX%27s_Future)*: *"... bid farewell to an aging workhorse that has served the community well."* (though the link is broken now) – Peter Mortensen Oct 02 '19 at 20:38
-
And for safety, you might want to remember to UNcheck "sort outputs only unique..." when you're done, so that a month from now, after you've forgotten all this, it does not happen unexpectedly, silently – john v kumpf Oct 15 '19 at 16:32
-
"You will need the TextFX plugin" ... then really it's TextFX doing it, rather than Notepad++ doing it. – Stewart Oct 17 '19 at 14:56
-
@GeertVc I don't see is in the Available tabs for Plugins Manager so I believe it was removed from the standard NPP plugin. – John Odom Aug 10 '20 at 20:11
-
@JohnOdom: That is correct. And I anticipated on that by saving it locally on my system. But you still can find it here: https://sourceforge.net/projects/npp-plugins/files/TextFX/. I would suggest: if you want to use it, take your copy and also save it locally. It's more than worth it. And I know, it isn't working on the 64 bit version of NPP but hey... I don't care... I'm very satisfied with the 32 bit version. As long as I have the capability to install TextFX, that's almost the only thing that matters to me... – GeertVc Aug 12 '20 at 05:25
-
@Stewart: That's all about NPP. It's also not NPP who does the real text editing and manipulation. It's Scintilla which is the real workhorse and doing the heavy work in the background. NPP is merely, and I say this with the utmost respect for the guy who made NPP, a UI on top of lots of other "helper" stuff, like Scintilla, like the many plugins,... I'm using NPP now for already 15+ years or so and since CodeWright (R.I.P.) it's the best one I found out there **for my purposes** (beware: I'm not saying it's **the** best one out there...). – GeertVc Aug 12 '20 at 05:30
-
@GeertVc I did tried installing the latest version from there but I got an error saying that it is not supported by the latest version of Notepad++, so I went with stema's answer instead that works for me. – John Odom Aug 12 '20 at 13:15
-
Thanks for the update @GeertVc, I have updated the answer with the link you posted. – Colin Pickard Aug 12 '20 at 13:22
-
@JohnOdom: what latest version of NPP? 64 bit? Then the answer is: no it's not supported. I'm currently using NPP 7.8.9 - 32 bit (the latest version of NPP AFAICS, related to Hong Kong even...) and TextFX is working like a charm... – GeertVc Aug 13 '20 at 15:07
-
@GeertVc. It's 64-bit, so that would explain it. – John Odom Aug 13 '20 at 19:13
-
Saved a lot of time. Thanks for the pictorial demo. – Chandra Shekhar Feb 15 '21 at 08:20
-
Although this is the accepted answer, it's not applicable to more recent versions of Notepad++. See this answer instead: https://stackoverflow.com/a/58549356/1681788 – Mark Barnes Mar 04 '21 at 11:00
704
Since Notepad++ Version 6 you can use this regex in the search and replace dialogue:
^(.*?)$\s+?^(?=.*^\1$)
and replace with nothing. This leaves from all duplicate rows the last occurrence in the file.
No sorting is needed for that and the duplicate rows can be anywhere in the file!
You need to check the options "Regular expression" and ". matches newline":
^matches the start of the line.(.*?)matches any characters 0 or more times, but as few as possible (It matches exactly on row, this is needed because of the ". matches newline" option). The matched row is stored, because of the brackets around and accessible using\1$matches the end of the line.\s+?^this part matches all whitespace characters (newlines!) till the start of the next row ==> This removes the newlines after the matched row, so that no empty row is there after the replacement.(?=.*^\1$)this is a positive lookahead assertion. This is the important part in this regex, a row is only matched (and removed), when there is exactly the same row following somewhere else in the file.
-
This one is better indeed than the other regex. No need for multiple passes to eliminate all duplicates. – Benny Jun 20 '13 at 03:55
-
11oh, this one is brilliant, it even deletes empty rows, i'm macroing it this very moment :) – Aprillion Jun 28 '13 at 16:14
-
Great to learn. Precise explanation too! Thanks to both the raiser and reply-er! – SarjanWebDev Oct 29 '13 at 01:50
-
72
-
3Is there any way to remove the LAST occurrence? This matches all but the last one... – Cullub Sep 23 '14 at 11:18
-
30In my case where this solution removed all lines, unchecking the `. matches newline` did the trick. – Kuitsi Dec 01 '15 at 07:34
-
1**Perfect!** I was using Notepad++ on a locked-down system with no internet access. No way to download plugins, so this was better for me. – ADTC Jan 06 '16 at 05:47
-
3@SerG In some cases it didn't work for me also, but when I removed "matches newline" it did :) – Davidenko Dec 19 '16 at 08:50
-
In my case, ALL lines were removed, as happened with SerG. But, when I left unchecked "matches newline", it worked perfectly, as happened with Davidenko. – aldemarcalazans Jun 21 '18 at 23:09
-
2If you adjust the capture group a little bit you can fix the side effect of deleting the file: ^([^\r\n]*)$\s+?^(?=.*^\1$) – Nick Kuznia Sep 30 '18 at 05:16
-
1
-
1For me it worked correctly only after sorting out lines with the native ordering function (Menu Edit → Line Operations → Sort Lines Lexicographically Ascending / Descending) – Mariano Paniga Oct 29 '18 at 11:03
-
1Perhaps add some statement about the actual performance? It sounds like it must at least have quadratic performance (both memory and execution). What is the actual number of lines for which it takes more than 1 second to execute? – Peter Mortensen Nov 19 '18 at 17:22
-
1For those who want to keep the first occurrence and delete the rest, reverse the lines first https://superuser.com/questions/331098/flip-or-reverse-line-order-in-notepad and then use the regex above and then reverse again. – Shayan Sep 21 '19 at 14:45
-
-
for being more cautionary, I use Find Next, and remove the duplicate manually. – Noor Hossain Jan 18 '21 at 19:40
99
If the rows are immediately after each other then you can use a regex replace:
Search Pattern: ^(.*\r?\n)(\1)+
Replace with: \1
Peter Mortensen
- 28,342
- 21
- 95
- 123
Grant Peters
- 7,193
- 3
- 41
- 56
-
1Maybe others have had luck with this, but for me ^(.*\n)\1 results in "Cant find the text" – b1nary.atr0phy Apr 28 '12 at 18:18
-
4@b1naryatr0phy make sure you have "Search Mode" set to "Regular expression", I also updated the pattern so that it can handle windows style line endings – Grant Peters May 01 '12 at 13:25
-
3notepad++ has a light regex engine, it dosen't permit advanced functios, not even the "? or \r\n" as it only works on a single line and you use $ for the \r\n characters – Stefan Rogin May 25 '12 at 16:39
-
3this eliminates one by one. You must repeat it many times. I wonder why \n+ -> \n does not work (thought it reports many replacements) – Val Sep 06 '12 at 09:48
-
2@Val, if you make the back-reference part of the match a group with 1-or-more matches required, the pattern will match N contiguous duplicate lines at a time: `^(.*\r?\n)(\1)+` – Kenigmatic Apr 29 '16 at 23:05
-
If you want to find unique characters in a file first use this: https://superuser.com/questions/1088622/notepad-split-line-after-given-number-of-characters then sort the rows and then use this answer. A bit long winded but it works. – wobblycogs Jun 01 '20 at 11:58
38
Notepad++
-> Replace window
Ensure that in Search mode you have selected the Regular expression radio button
Find what:
^(.*)(\r?\n\1)+$
Replace with:
$1
Before:
and we think there
and we think there
single line
Is it possible to
Is it possible to
After:
and we think there
single line
Is it possible to
Peter Mortensen
- 28,342
- 21
- 95
- 123
blueberry0xff
- 3,367
- 28
- 18
35
If you don't care about row order (which I don't think you do), then you can use a Linux/FreeBSD/Mac OS X/Cygwin box and do:
$ cat yourfile | sort | uniq > yourfile_nodups
Then open the file again in Notepad++.
Peter Mortensen
- 28,342
- 21
- 95
- 123
Pablo Santa Cruz
- 162,219
- 30
- 224
- 277
-
3Doesn't work on Windows 7. `'cat' is not recognized as an internal or external command, operable program or batch file.` – Iain Samuel McLean Elder Dec 11 '14 at 16:50
-
2@Iain Elder: cat is a standard Unix utility, which is why this answer specifies that it works on linux, FreeBSD, and MacOSX. The answer also suggests Cygwyn: This is a windows program that gives you a unix style shell, and with it, cat. Long story short (too late!): Win 7 needs Cygwin to do this. – Travis Clark Jan 14 '15 at 16:14
-
11
-
10These are good examples of "the gratuitous use of cat". Forget about the cat utility and just use file redirection thusly: sort **< yourfile** | uniq > yourfile_nodups – scott8035 May 16 '16 at 18:57
-
1@scott8035, I agree that cat is of no use for running that command, but I find it often helpful to start with cat when figuring out a long sequence of non-obvious commands, like cat file | sed ... | sed ... | sed ... and so on. So I'd say that there might be reasons for using cat. Of course cat can be removed at the end, but some are too lazy for that. – FORTRAN Sep 14 '17 at 06:56
-
You can install bash now on Windows 10, just search "Ubuntu" in Microsoft Store and follow the instructions in the Description. – Patronaut Feb 23 '18 at 14:46
25
In version 7.8, you can accomplish this without any plugins - Edit -> Line Operations -> Remove Consecutive Duplicate Lines. You will have to sort the file to place duplicate lines in consecutive order before this works, but it does work like a charm.
Sorting options are available under Edit -> Line Operations -> Sort By ...
-
1There's also now an option for Edit -> Line Operations -> Remove Duplicate Lines which eliminates the need to sort. – Mark Barnes Mar 04 '21 at 10:50
-
Whats the difference between "Remove Duplicate Lines" and "Remove Consecutive Duplicate Lines" – user3304007 Apr 24 '21 at 19:05
-
First option should remove all but one of each matching line in a document (so a, a, b, a, c would become a, b, c). Second option should only remove lines that are repeated immediately after a matching line (a, a, b, a, c would become a, b, a, c). – dr.nixon Apr 26 '21 at 15:48
18
The latter versions of Notepad++ do not apparently include the TextFX plugin at all. In order to use the plugin for sorting/eliminating duplicates, the plugin must be either downloaded and installed (more involved) or added using the plugin manager.
A) Easy way (as described here).
Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX Characters -> Install
B) More involved way, if another version is needed or the easy way does not work.
Download the plugin from SourceForge:
Open the zip file and extract NppTextFX.dll
Place NppTextFX.dll in the Notepad++ plugins directory, such as:
C:\Program Files\Notepad++\pluginsStart Notepad++, and TextFX will be one of the file menu items (as seen in Answer #1 above by Colin Pickard)
After installing the TextFX plugin, follow the instructions in Answer #1 to sort and remove duplicates.
Also, consider setting up a keyboard shortcut using Settings > Shorcut mapper if you use this command frequently or want to replicate a keyboard shortcut, such as F9 in TextPad for sorting.
-
In notepad++ 7.6, the plug-in should be added to `C:\Users\
\AppData\Local\Notepad++\plugins\NppTextFX` . Other than that this still works fine. – P_W999 Dec 18 '18 at 07:34
14
As of now, it's possible to remove all consecutive duplicate lines with Notepad in-built functionality. Sort the lines first:
Edit > Line Operations > "Sort lines lexicographically",
then
Edit > Line Operations > "Remove Consecutive Duplicate Lines".
The regex solution suggested above didn't remove all duplicate lines for me, but just the consecutive ones as well.
SaPropper
- 342
- 2
- 10
8
You may need a plugin to do this. You can try the command line cc.ddl(delete duplicate lines) of ConyEdit. It is a cross-editor plugin for the text editors, including Notepad++.
With ConyEdit running in background, follow the steps below:
- enter the command line
cc.ddlat the end of the text. - copy the text and the command line.
- paste, then you will see what you want.
Example
Donald
- 111
- 1
- 4
6
Search for the regular expression: \b(\w+)\b([\w\W]*)\b\1\b
Replace it with: $1$2
Hit the Replace button until there are no more matches for the regular expression in your file.
Peter Mortensen
- 28,342
- 21
- 95
- 123
Hesham Eraqi
- 2,253
- 3
- 20
- 40
-
Created a test file to try this, but the regular expression did not work reliably to get the job done. – RockPaperLz- Mask it or Casket Mar 20 '16 at 05:29
-
Would you please provide an example that fails so I can improve my answer? – Hesham Eraqi Aug 03 '19 at 00:07
-
1Remove duplicates leaving also the original row number position of other text, I like this solution – Jack Rock Jan 16 '21 at 18:18
4
None worked for me.
A solution is:
Replace
^(.*)\s+(\r?\n\1\s+)+$
with
\1
Manohar Reddy Poreddy
- 16,412
- 7
- 111
- 98
-
Created a test file to try this, but the regular expression did not work reliably to get the job done. – RockPaperLz- Mask it or Casket Mar 20 '16 at 05:28
-
For all my data, it worked fine.I forgot what my solution was. Add more details where it failed so that other people might improve this regex. – Manohar Reddy Poreddy Mar 20 '16 at 05:46
-
I created a file so each line had a integer between 0-999 on it, in random order, sometimes with duplicates. It didn't remove most of the duplicates, and didn't remove any duplicates there were not sequential. – RockPaperLz- Mask it or Casket Mar 20 '16 at 05:58
-
1Please do provide 2 examples for working and for not-working ones. It will help someone. – Manohar Reddy Poreddy Mar 20 '16 at 09:01
-
-
1
-
hey, I lost the context of this regex I wrote long back, but the difference you point is either 1 or more characters vs 0 or more characters, but if I wrote it + instead of * that must mean that I tried * then came to a + solution, so the answer must be correct to the question asked. – Manohar Reddy Poreddy Sep 11 '18 at 13:44
2
The plugin manager is currently unavailable (does not come with the distribution) for Notepad++. You must install it manually (https://github.com/bruderstein/nppPluginManager/releases) and even if you do, a lot of the plugins are not available anymore (no TextFX) plugin.
Maybe there is another plugin which contains the required functionality. Other than that, the only way to do it in Notepad++ is to use some special regex for matching and then replacing (Ctrl + F → Replace tab).
Although there are many functionalities available via Edit menu item (trimming, removing empty lines, sorting, converting EOL) there is no "unique" operation available.
If you have Windows 10 then you can enable Bash (just type Ubuntu in Microsoft Store and follow the instructions in the description to install it) and use cat your_file.txt | sort | uniq > your_file_edited.txt. Of course you must be in the same working directory as "your_file.txt" or refer to it via its path.
Peter Mortensen
- 28,342
- 21
- 95
- 123
Patronaut
- 609
- 8
- 10
1
Whether the file is sorted or not, you can use below regex to remove duplicates in anywhere occurred in your file.
Find what: ^([^\r]*[^\n])(.*?)\r?\n\1$
Replace with: \1\2
Search Mode:
- "Regular expression"
- Check the ". matches newline" option
do "Replace All" as many time as possible until you see "0 occurrences were replaced"
αғsнιη
- 2,026
- 2
- 20
- 34
-
I like this because you're not forced to sort the contents of the file first. It also can be used on any text editor that supports Perl regex. – foundationer May 26 '21 at 19:22
0
Difficult to do this in NPP. Better way is following:
Download cygwin utility, it is simple Linux terminal under windows. It allow to execute any Linux command in Windows. And you have sort -u there.
haykp
- 374
- 6
- 24
-
Why is it difficult? Have you seen other answers? What's wrong with them? – Toto Jan 21 '21 at 15:11