Regex to delete lines that match the first string?

Question

I have long list of lines with a lot of situations line this, lines that have identical first word (first string before space), but the rest is different. I need to keep only one line with unique first string.

john jane
john 123
john jim jane
jane john
jane 123
jane 456
jim
jim 1

To have this result:

john jane
jane john
jim

So, if first word in line is the match, delete all but one line.

I can delete all duplicate lines, but leave with lines like in example above,

^(.*)(\r?\n\1)+$

This regex delete identical lines, not like in the example. If there is regex or notepad macro to solve this?

Not the best solution for Notepad++: `^((\w+\b).*)\r?\n\2.*` -> `$1` and hit *Replace All* multiple times. — Wiktor Stribiżew, Jul 15 '16 at 10:31
Are the lines with the same first "word" always consecutive? And please answer to anubhava question if you want relevant answers. — Casimir et Hippolyte, Jul 15 '16 at 11:01

score 2 · Answer 1 · edited May 23 '17 at 10:28

2

if you have awk

awk '!seen[$1]++' infile.txt

adapted from this thread: Unix: removing duplicate lines without sorting

edited May 23 '17 at 10:28

Community

1
1

answered Jul 15 '16 at 10:29

Sundeep

19,273
2
19
42

Casimir et Hippolyte · Accepted Answer · 2016-07-15T11:22:48.427

2

With Notepad++ (assuming lines with the same first word are consecutive):

search: ^(\S++).*\K(?:\R\1(?:\h.*|$))+
replacement: nothing

demo

pattern details:

^             # start of the line
(\S++)        # the first "word" (all that isn't a whitespace) captured in group 1
.*            # all characters until the end of the line
\K            # remove characters matched before from the match result
(?:
    \R        # a newline
    \1        # reference to the capture group 1 (same first word)
    (?:
        \h.*  # a horizontal whitespace 
      |       # OR
        $     # the end of the line
    )
)+            # repeat one or more times

edited Jul 15 '16 at 11:22

answered Jul 15 '16 at 11:15

Casimir et Hippolyte

83,228
5
85
113

Confirmed, it is working for my file. Works in ultraedit as well, use it because Notepad++ cant handle very large files. – Jim8645 Jul 15 '16 at 12:26
1

@Jim8645: note that if you use unix/linux, sp asic awk approach is interesting for large files since it doesn't need to load all the file in memory. – Casimir et Hippolyte Jul 15 '16 at 12:31

score 0 · Answer 3 · answered Jul 15 '16 at 10:34

In Perl:

s/^((\w+).*)\n(?:(?:\2.*\n)*)/$1/gm

You can give it a try with this:

#!/bin/usr/perl

use warnings;
use strict;

my $file = "john jane
john 123
john jim jane
jane john
jane 123
jane 456
jim
jim 1
";

$file =~ s/^((\w+).*)\n(?:(?:\2.*\n)*)/$1\n/gm;

print $file;

Regex to delete lines that match the first string?

3 Answers3