How to merge two csv files not including duplicates

Question

If I have a csv files like this

lion@mammal@scary animal
human@mammal@human
hummingbird@bird@can fly
dog@mammal@man's best friend
cat@mammal@purrs a lot
shark@fish@very scary
fish@fish@blub blub

and I have another csv file like this

cat@mammal@purrs a lot
shark@fish@very scary
fish@fish@blub blub
rockets@pewpew@fire
banana@fruit@yellow

I want the output to be like this:

lion@mammal@scary animal
human@mammal@human
hummingbird@bird@can fly
dog@mammal@man's best friend
cat@mammal@purrs a lot
shark@fish@very scary
fish@fish@blub blub
rockets@pewpew@fire
banana@fruit@yellow

some of the things in the first csv file are present in the second csv file; they overlap pretty much. How can I combine these csv files with the correct order? It is guaranteed that the new entries will always be the first few lines in the beginning of the first csv file.

Are you fine if the output is sorted on alphabetical order? – Dale Dec 17 '13 at 02:01 — Dale, Dec 17 '13 at 02:01

BMW · Accepted Answer · 2013-12-17T23:37:26.053

14

Soultion 1:

awk '!a[$0]++' file1.cvs file2.cvs

Solution 2 (if don't care of the original order)

sort -u file1 file2

edited Dec 17 '13 at 23:37

answered Dec 17 '13 at 02:06

BMW

34,279
9
81
95

Digital Trauma · Answer 2 · 2013-12-17T16:30:07.337

0

Here's one way:

Use cat -n to concatenate input files and prepend line numbers
Use sort -u remove duplicate data
Use sort -n to sort again by prepended number
Use cut to remove the line numbering

$ cat -n file1 file2 | sort -uk2 | sort -nk1 | cut -f2-
lion@mammal@scary animal
human@mammal@human
hummingbird@bird@can fly
dog@mammal@man's best friend
cat@mammal@purrs a lot
shark@fish@very scary
fish@fish@blub blub
rockets@pewpew@fire
banana@fruit@yellow
$

edited Dec 17 '13 at 16:30

answered Dec 17 '13 at 02:14

Digital Trauma

13,834
2
40
73

1

Useless Use of Cat Award (http://partmaps.org/era/unix/award.html) – BMW Dec 17 '13 at 02:18
@BMW - why? How would you concatenate the files *and* add line numbers? – Digital Trauma Dec 17 '13 at 02:28
@BMW - Perhaps you are referring to the first two cat invocations here. Actually these are simply for demonstration purposes to show what the contents of the input files are, before processing. I guess this extra output detracts from the answer and is not really necessary, so I'll remove them – Digital Trauma Dec 17 '13 at 02:58

How to merge two csv files not including duplicates

2 Answers2