BASH/sed to remove duplicates from a separated-by-line word list in a text file

Question

I have a long vocabulary list, one word per line. Sometimes, there is a duplicate word, appearing more than once OR TWICE. I need a simple code that will leave the first occurrence of a word, but remove all duplicates (with its line) after it.

1. I don't want to remove any special characters or rearrange anything, only remove the words (one per line). Keeping the same word order is important.

2. It doesn't matter if it overwrites the original file or saves to a new one, whichever is "more efficient".

3. This is a list separated by line, not an array, not separated by space or comma.

4. I have not code to start with, hoping to solve with BASH...

   • sed would be first choice

   • grep would be second choice

   • Third choice would be something like a for loop

Eg: file.txt

apple
banana
car
bicycle
apple
tree
banana
apple
motorcycle

...should become:

apple
banana
car
bicycle
tree
motorcycle

I see some solutions for ARRAYS, but not simple lists, and answers via python, js, and C languages, but not BASH. If this has already been answered, show me where and I will gladly delete this question. The suggested dupl. article uses awk, which is outside of the scope of this question, though it is related and useful.

Answer 1

This might work for you (GNU sed):

sed -nr 'G;/^([^\n]+\n)([^\n]+\n)*\1/!{P;h}' file

Keep a list of unique keys in the hold space and if the current key is not in the list print it and add it to the list.

Answer 2

If you weren't overly concerned about maintaining the order, you could just use the very simple:

sort -u inputFileName >outputFileName

This would get rid of all duplicates, sorting in the process.

For maintaining the order based on first occurrence, it becomes more complex (and memory hungry). Using associative arrays in awk is one way, as per the following example:

pax> cat infile
zanzibar
apple
banana
apple
carrot
banana
sausage
apple

awk '{if(x[$1]==0){x[$1]=1;print}}' infile
zanzibar
apple
banana
carrot
sausage

The way this works is that awk will, the first time it sees a word, store the fact that it's seen it and output the word. Later instances of that word will do nothing because the word has already been marked as seen.

Answer 3

Pure bash:

#!/bin/bash
declare -g -A lines
while IFS='' read -r line
do
  if [[ "${lines["$line"]}" -ne 1 ]]
  then
    echo "$line"
    lines["$line"]=1
  fi
done

EDIT: If you make it into a standalone executable script, you could do it with dedupe.sh < file.txt. If you want to hard-code the file name in there, you can do so like this:

while ....
  ...
done < file.txt

Answer 4

Once you sort the file using sort you can then remove adjacent duplicate lines using uniq.

Man pages uniq

sort unsorted.txt | uniq >> sorted_deduped.txt