How can I remove duplicate lines in Visual Studio Code?

Question

Say you have the following text:

abc
123
abc
456
789
abc
abc

I want to remove all "abc" lines and just keep one. I don't mind sorting. The result should be like this:

Younes, you better come back and accept @Marc.2377 answer. It is genius. — DrFloyd5, Mar 01 '21 at 14:38

score 280 · Answer 1 · edited Jun 20 '20 at 09:12

280

If the order of lines is not important

Sort lines alphabetically, if they aren't already, and perform these steps:
_{(based on this related question: How do I find and remove duplicate lines from a file using Regular Expressions?)}

Control+F
Toggle "Replace mode"
Toggle "Use Regular Expression" (the icon with the .* symbol)
In the search field, type ^(.*)(\n\1)+$
In the "replace with" field, type $1
Click ("Replace All").

If the order of lines is important so you can't sort

In this case, either resort to a solution outside VS Code (see here), or - if your document is not very large and you don't mind spamming the Replace All button - follow the previous steps, but in steps 4 and 5, enter these:
_{(based on Remove specific duplicate lines without sorting)}

Caution: Blocks for files with too many lines (1000+); may cause VS Code to crash; may introduce blank lines in some cases.

search: ((^[^\S$]*?(?=\S)(?:.*)+$)[\S\s]*?)^\2$(?:\n)?
replace with: $1

and then click the "Replace All" button as many times as there are duplicate occurrences.

You'll know it's enough when the line count stops decreasing when you click the button. Navigate to the last line of the document to keep an eye on that.

edited Jun 20 '20 at 09:12

Community

1
1

answered Aug 23 '17 at 02:16

Marc.2377

5,840
5
43
75

8

`((^[^\S\r\n]*?(?=\S)(?:.*)+$)[\S\s]*?)^\2$(?:\r?\n)?` made my vscode crash.... I did a Find in one file 229 lines. :( – Hickory420 Jul 20 '18 at 06:23
@Hickory420 I tested in my machine with 1000 lines (20-char long, random) and got no crash, but indeed a thread blocks with 100% cpu load for a few seconds at each pass. Yeah, this is hardly pratical for large files. – Marc.2377 Jul 20 '18 at 09:00
Thanks for this. Can you please explain the regex `^(.*)(\n\1)+$`. After removing duplicate rows I want to look at all rows with duplicate first column in the csv and want to modify the regex. – Urvah Shabbir Mar 24 '20 at 07:19
1

Wow I feel like I'm pretty good at regex and this still blew my mind, great answer!! – electrovir Apr 14 '20 at 21:52
@UrvahShabbir, an explanation for that piece of regex is given in the [linked Q&A](https://stackoverflow.com/a/1573425/3258851). Mine is only different in that the `\r?` bit from the other answer is not really necessary. – Marc.2377 Jun 15 '20 at 22:07
It's crazy that we have to do something that complicated when it's a basic "edit > remove duplicates" in Sublime... – dhokas Nov 16 '20 at 14:27
Wow using regex capture groups for replacement is so useful! Thanks – melMass Feb 04 '21 at 15:04

score 95 · Answer 2 · edited Feb 22 '19 at 09:30

Here is a very interesting extension: Transformer

Features:

Unique Lines
Unique Lines As New Document
Filter Lines
Filter Lines As New Document
Sort Lines
Sort Lines By Length
Align To Cursor
Align CSV
Compact CSV
Copy To New Document
Select Lines
Lines As JSON
Trim Lines
Count Duplicate Lines As New Document
Macros

For removing duplicate lines:

Removes duplicate lines from the document
Operates on selection or current block if no selection

I haven't played with it much besides the "Unique Lines" command but it seems quite nicely done (including attempting a macro recorder!).

Skeeve · Answer 3 · 2020-04-21T05:44:59.193

38

To add to @Marc.2377 's reply.

If the order is important and you don't care that you just keep the last of the duplicate lines, simply search for the following regexp if you want to only remove duplicte non-empty lines

^(.+\n)(?=(?:.*\n)*?\1)

If you also want to remove duplicate empty lines, use * instead of +

^(.*\n)(?=(?:.*\n)*?\1)

and replace with nothing.

This will take a line and try to find ahead some more (maybe 0) lines followed by the exact same line taken. It will remove the taken line.

This is just a one-shot regex. No need to spam the replace button.

edited Apr 21 '20 at 05:44

answered Feb 22 '19 at 09:26

Skeeve

3,233
2
12
19

Nicely succinct – angus l Mar 07 '19 at 11:05
4

Nice. I recommend `^(.+\n)(?=(?:.*\n)*?\1)` instead because your regex removed an empty line where it wasn't expected to. Upvoted anyway. – Marc.2377 Mar 10 '19 at 19:46
Good catch… OTOH: duplicate empty lines are also duplicates ;) – Skeeve Mar 11 '19 at 11:31
Thanks to [zaman](https://stackoverflow.com/users/4870357) there is now a screenshot of the search-and-replace box. He also changed the regexp to ignore empty lines. – Skeeve Apr 21 '20 at 05:58
1

@Skeeve Come on, this just a little thanks for ur helpful answer & All for better community :) – Zaman Jun 23 '20 at 15:49
Can someone please explain the `(?=(?:.*\n)*?` part in the regex. – TrigonaMinima Jul 22 '20 at 18:22
1

`xxx(?=…)` is a lookahead-match. So it makes sure that, whatever follows "xxx" matches "…", but does not advance the search. `(?:…)` is just a bracket which does not count in the bracket count. `.*\n` is a pattern for a (possibly empty) line. `*` means that there may be as several lines, even none. The `?` after the asterisk (`*`) means that we want as few lines as possible. As `\1` follows this expression the effect is that we look ahead for all the lines which do not match `\1` until we find a line matching `\1`. I hope this makes it clear. – Skeeve Jul 23 '20 at 06:44

score 29 · Answer 4 · edited Jun 09 '20 at 12:30

29

I just had the same issue and found the Visual Studio Code package "Sort lines". See the Visual Studio Code market place for details (e.g. Sort lines).

This package has the option "Sorting lines (unique)", which did it for me. Take care of any white spaces at the beginning/end of lines. They influence whether lines are considered unique or not.

edited Jun 09 '20 at 12:30

Peter Mortensen

28,342
21
95
123

answered Jul 13 '16 at 09:55

SimonAx

976
1
7
24

1

https://marketplace.visualstudio.com/items?itemName=bibhasdn.unique-lines should also work. – kcpr May 20 '17 at 19:26
1

It seems like the extension no longer has the ability to remove duplicate entries. Combining it with [the answer](https://stackoverflow.com/a/45829605/31532) by @Marc-2377 seems to do the trick for me. – Dan Atkinson Oct 02 '17 at 20:29

score 18 · Answer 5 · edited Jun 09 '20 at 12:33

18

Try find and replace with a regular expression.

Find: ^(.+)((?:\r?\n.*)*)(?:\r?\n\1)$
Replace: $1$2

It is possible to introduce some variance in the first group.

edited Jun 09 '20 at 12:33

Peter Mortensen

28,342
21
95
123

answered Jan 26 '18 at 16:13

Lavock

201
3
7

score 17 · Answer 6 · edited Jun 09 '20 at 12:34

17

Install the DupChecker extension, hit F1, and type "Check Duplicates".

It will check for duplicates and ask if you want to remove them.

edited Jun 09 '20 at 12:34

Peter Mortensen

28,342
21
95
123

answered Jun 15 '18 at 14:45

perfecto25

488
4
11

score 1 · Answer 7 · answered Oct 19 '20 at 17:16

Sublime Text 3

It has blisteringly fast native permutation functions.

Edit > Permute Lines > Unique or ⇧⌘U, and
Edit > Permute Selections > Unique

Visual Studio Code is my daily driver. But, I keep Sublime Text on standby for these situations.

falwick falwick · Answer 8 · 2021-02-15T07:59:02.850

0

If you don't mind some Vim in your VS Code. You can install Vim emulation plugin.

Then you can use vim commands

:sort u

It will sort lines and it will remove duplicates

edited Feb 15 '21 at 07:59

answered Feb 12 '21 at 08:16

falwick falwick

1
2

score -4 · Answer 9 · edited Jun 09 '20 at 12:38

-4

Not actually in Visual Studio Code, but if it works, it works.

Open a new Excel spreadsheet
Paste the data into a column
Go to the Data tab
Select the column of data (if you haven't already)
Click Remove Duplicates (somewhat in the middle of the bar)
Click OK to remove duplicates.

It is not the best answer, as you specified Visual Studio Code, but as I said: If it works, it works :)

edited Jun 09 '20 at 12:38

Peter Mortensen

28,342
21
95
123

answered Jan 03 '20 at 13:24

NostraDavid

49
4

You could make it more relevant by [providing a script](https://superuser.com/questions/110991/can-you-zip-a-file-from-the-command-prompt-using-only-windows-built-in-capabili/111266#111266) that can be called directly from Visual Studio Code. In order words, automates this process. I don't know if it is possible, but a script that would invoke Excel through its exposed COM interfaces. This would make this answer much more valuable as it would be an example of leveraging other applications to do neat stuff. – Peter Mortensen Jun 09 '20 at 12:42

How can I remove duplicate lines in Visual Studio Code?

9 Answers9

If the order of lines is not important

If the order of lines is important so you can't sort

Sublime Text 3

Linked