0

What are the ways by which duplicate word in a String can be detected?

e.g. "this is a test message for duplicate test" contains one duplicate word test.

Here, the objective is to detect all duplicate words which occur in a String.

Use of regular expression is preferable to achieve the goal.

Debadyuti Maiti
  • 929
  • 4
  • 16
  • 29

2 Answers2

8

The best you can do with regexes is O(N^2) search complexity. You can easily achieve O(N) time and space search complexity by splitting the input into words and using a HashSet to detect duplicates.

Stephen C
  • 632,615
  • 86
  • 730
  • 1,096
  • 1
    Then the tradeoff again is time vs space since you need a backing data structure for detection – gtgaxiola Sep 19 '12 at 15:05
  • 1
    Yes, but as I said the space overhead is `O(N)`; i.e. proportional to the size of the input. – Stephen C Sep 19 '12 at 15:06
  • @StephenC But can you provide any link which shows O(N^2) time complexity? Because this link claims it as O(N). http://stackoverflow.com/questions/5892115/whats-the-time-complexity-of-average-regex-algorithms – Debadyuti Maiti Sep 19 '12 at 15:09
  • That Answer is referring to **real** regular expressions (in the theoretical sense). A real regex does not allow back-references. And if you don't believe me, I suggest that you do some experiments to see how the performance of your regex scales for larger and larger input strings. – Stephen C Sep 19 '12 at 15:15
  • @StephenC Can you provide code example [i.e. dealing with HashSet]? Because I think for "splitting the input into words", I have to use regular expression. Again each word has to be changed to lowerCase or upperCase, otherwise I don't think HashSet will be able to distinguish between duplicate Strings with mixed cases. So, for large input,the String objects [just for comparing] created will be very high, & for changing lower case,splitting the input to words altogether should have some performance overhead. – Debadyuti Maiti Sep 20 '12 at 07:56
  • @StephenC If you have got different implementation on your mind, then please provide a code snippet. – Debadyuti Maiti Sep 20 '12 at 07:58
3

The following Java code resolves the problem of detecting duplicates from a String. There should not be any problem if the duplicate word is separated by newline or punctuation symbols.

    String duplicatePattern = "(?i)\\b(\\w+)\\b[\\w\\W]*\\b\\1\\b";
    Pattern p = Pattern.compile(duplicatePattern);
    String phrase = "this is#$;%@;<>?|\\` p is a is Test\n of duplicate test";
    Matcher m = p.matcher(phrase);
    String val = null;
    while (m.find()) {
        val = m.group();
        System.out.println("Matching segment is \"" + val + "\"");
        System.out.println("Duplicate word: " + m.group(1)+ "\n");
    }

The output of the code will be:

Matching segment is "is#$;%@;<>?|\` p is a is"
Duplicate word: is

Matching segment is "Test
 of duplicate test"
Duplicate word: Test

Here, m.group(1) statement represents the String matched against 1st group of Pattern [here, it's (\\w+)].

Debadyuti Maiti
  • 929
  • 4
  • 16
  • 29