21

Update: In Java 11 bug described below seems to be fixed

(possibly it was fixed even earlier, but I don't know in which version exactly. Bug report about similar problem linked in nhahtdh's answer suggests Java 9).


TL;DR (before fix):
Why [^\\D2], [^[^0-9]2], [^2[^0-9]] get different results in Java?


Code used for tests. You can skip it for now.

String[] regexes = { "[[^0-9]2]", "[\\D2]", "[013-9]", "[^\\D2]", "[^[^0-9]2]", "[^2[^0-9]]" };
String[] tests = { "x", "1", "2", "3", "^", "[", "]" };

System.out.printf("match | %9s , %6s | %6s , %6s , %6s , %10s%n", (Object[]) regexes);
System.out.println("-----------------------------------------------------------------------");
for (String test : tests)
    System.out.printf("%5s | %9b , %6b | %7b , %6b , %10b , %10b %n", test,
            test.matches(regexes[0]), test.matches(regexes[1]),
            test.matches(regexes[2]), test.matches(regexes[3]),
            test.matches(regexes[4]), test.matches(regexes[5]));

Lets say I need regex which will accept characters that are

  • not digits,
  • with exception of 2.

So such regex should represent every character except 0, 1, 3,4, ... , 9. I can write it at least in two ways which will be sum of everything which is not digit with 2:

  • [[^0-9]2]
  • [\\D2]

Both of these regexes works as expected

match , [[^0-9]2] ,  [\D2]
--------------------------
    x ,      true ,   true
    1 ,     false ,  false
    2 ,      true ,   true
    3 ,     false ,  false
    ^ ,      true ,   true
    [ ,      true ,   true
    ] ,      true ,   true

Now lets say I want to reverse accepted characters. (so I want to accept all digits except 2) I could create regex which explicitly contains all accepted characters like

  • [013-9]

or try to negate two previously described regexes by wrapping it in another [^...] like

  • [^\\D2]
  • [^[^0-9]2]
    or even
  • [^2[^0-9]]

but to my surprise only first two versions work as expected

match | [[^0-9]2] ,  [\D2] | [013-9] , [^\D2] , [^[^0-9]2] , [^2[^0-9]] 
------+--------------------+------------------------------------------- 
    x |      true ,   true |   false ,  false ,       true ,       true 
    1 |     false ,  false |    true ,   true ,      false ,       true 
    2 |      true ,   true |   false ,  false ,      false ,      false 
    3 |     false ,  false |    true ,   true ,      false ,       true 
    ^ |      true ,   true |   false ,  false ,       true ,       true 
    [ |      true ,   true |   false ,  false ,       true ,       true 
    ] |      true ,   true |   false ,  false ,       true ,       true 

So my question is why [^[^0-9]2] or [^2[^0-9]] doesn't behave as [^\D2]? Can I somehow correct these regexes so I would be able to use [^0-9] inside them?

Community
  • 1
  • 1
Pshemo
  • 113,402
  • 22
  • 170
  • 242
  • 1
    Just to let you know `[^[^0-9]]` is not same as `[0-9]` – anubhava Feb 21 '14 at 12:42
  • @anubhava Thanks. That is very interesting. From what I see it behaves same as `[^0-9]`. Can you explain it a little? From what I read in posted answer it should be more like union of `not nothing` (which could probably represent all characters with `[^0-9]`. But in this case `Everything UNION Anything` is still `Everything`. In this case it seems like instead of union intersection was used which is kind of confusing. – Pshemo Feb 21 '14 at 12:58
  • Sorry had to goto a meeting. IMO there is no negation in nested `[ and ]`. So `[^[^0-9]]` is interpreted same as UNION of `^` and `[^0-9]` and which is effectively `[^0-9]` itself. – anubhava Feb 21 '14 at 14:09
  • @anubhava If `[^[^0-9]]` would be union of `^` and `[^0-9]` then `^` should be matched by `[^[^0-9^]]` but `"^".matches("[^[^0-9^]]")` returns false :/ (unless I misunderstood you). – Pshemo Feb 21 '14 at 14:14
  • Umm not sure as `[^3[^2]]` matches everything, i.e. any digit or any non-digit. – anubhava Feb 21 '14 at 14:47
  • FYI, `[^3[^2]]` is parsed as `[^3]` union with `[^2]`. – nhahtdh Feb 23 '14 at 14:27
  • As pointed in [response](http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-February/025326.html) to nhahtdh\`s [mail](http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-February/025314.html) this issue is known to Java creators but couldn't be resolved because of backward compatibility. I think we can't do more for now. Thank you all for your time and input. While Keppils answer describe why and how it should work I decided to accept nhahtdh\`s answer because it provides most details about this problem and even possible solution (check earlier mentioned mail to find it). – Pshemo Mar 01 '14 at 17:11
  • This question has been added to the [Stack Overflow Regular Expression FAQ](http://stackoverflow.com/a/22944075/2736496), under "Character Classes". – aliteralmind Apr 10 '14 at 00:18

2 Answers2

16

According to the JavaDoc page nesting classes produces the union of the two classes, which makes it impossible to create an intersection using that notation:

To create a union, simply nest one class inside the other, such as [0-4[6-8]]. This particular union creates a single character class that matches the numbers 0, 1, 2, 3, 4, 6, 7, and 8.

To create an intersection you will have to use &&:

To create a single character class matching only the characters common to all of its nested classes, use &&, as in [0-9&&[345]]. This particular intersection creates a single character class matching only the numbers common to both character classes: 3, 4, and 5.

The last part of your problem is still a mystery to me too. The union of [^2] and [^0-9] should indeed be [^2], so [^2[^0-9]] behaves as expected. [^[^0-9]2] behaving like [^0-9] is indeed strange though.

Keppil
  • 43,696
  • 7
  • 86
  • 111
  • Thanks for your answer. That was also what I thought at start. It looks like in case of `[^2[^0-9]]` `[^2]` is created first and later regex engine uses union to combine it with `[^0-9]`, so it doesn't change anything because sum of this two classes is `[^2]` (`[^0-9]` is subset of `[^2]`). What bothers me is why `[^[^0-9]2]` behaves same as `[^0-9]` but not as `[^2]`? – Pshemo Feb 21 '14 at 12:39
  • @Pshemo: Updated the answer a bit. I was thinking that all the examples in the javadoc has the nested class as the last element. Could the behavior be a bit undefined if that convention isn't followed? – Keppil Feb 21 '14 at 13:04
  • That is quite puzzling. Maybe if `[]` is first element of outer negated character class instead of union intersection is used. DeMorgan can be here do blame but I am not sure how to connect him to this situation. – Pshemo Feb 21 '14 at 13:46
  • Even more puzzling to me is that `[^[^0-9]2]`, `[^[^[^0-9]]2]` and `[^[^[^[^0-9]]]2]` all produce the same result. I tried looking through the code, but it wasn't very easy to follow. – Keppil Feb 21 '14 at 13:59
  • @Keppil are multiple nesting levels even supported? – Martin Ender Feb 26 '14 at 16:00
  • @m.buettner: At least 2 is expected, since the documentation has such example. However, the documentation does not specify anything about nested negated character class. – nhahtdh Feb 26 '14 at 18:26
15

There are some strange voodoo going on in the character class parsing code of Oracle's implementation of Pattern class, which comes with your JRE/JDK if you downloaded it from Oracle's website or if you are using OpenJDK. I have not checked how other JVM (notably GNU Classpath) implementations parse the regex in the question.

From this point, any reference to Pattern class and its internal working is strictly restricted to Oracle's implementation (the reference implementation).

It would take some time to read and understand how Pattern class parses the nested negation as shown in the question. However, I have written a program1 to extract information from a Pattern object (with Reflection API) to look at the result of compilation. The output below is from running my program on Java HotSpot Client VM version 1.7.0_51.

1: Currently, the program is an embarrassing mess. I will update this post with a link when I finished it and refactored it.

[^0-9]
Start. Start unanchored match (minLength=1)
CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
  Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

Nothing surprising here.

[^[^0-9]]
Start. Start unanchored match (minLength=1)
CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
  Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match
[^[^[^0-9]]]
Start. Start unanchored match (minLength=1)
CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
  Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

The next 2 cases above are compiled to the same program as [^0-9], which is counter-intuitive.

[[^0-9]2]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match
[\D2]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Ctype. Match POSIX character class DIGIT (US-ASCII)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match

Nothing strange in the 2 cases above, as stated in the question.

[013-9]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 2 character(s):
    [U+0030][U+0031]
    01
  Pattern.rangeFor (character range). Match any character within the range from code point U+0033 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match
[^\D2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
      Ctype. Match POSIX character class DIGIT (US-ASCII)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match

These 2 cases work as expected, as stated in the question. However, take note of how the engine takes complement of the first character class (\D) and apply set difference to the character class consisting of the leftover.

[^[^0-9]2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match
[^[^[^0-9]]2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match
[^[^[^[^0-9]]]2]
Start. Start unanchored match (minLength=1)
Pattern.setDifference (character class subtraction). Match any character matched by the 1st character class, but NOT the 2nd character class:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
  BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
    [U+0032]
    2
LastNode
Node. Accept match

As confirmed via testing by Keppil in the comment, the output above shows that all 3 regex above are compiled to the same program!

[^2[^0-9]]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
      [U+0032]
      2
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

Instead of NOT(UNION(2, NOT(0-9)), which is 0-13-9, we get UNION(NOT(2), NOT(0-9)), which is equivalent to NOT(2).

[^2[^[^0-9]]]
Start. Start unanchored match (minLength=1)
Pattern.union (character class union). Match any character matched by either character classes below:
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    BitClass. Optimized character class with boolean[] to match characters in Latin-1 (code point <= 255). Match the following 1 character(s):
      [U+0032]
      2
  CharProperty.complement (character class negation). Match any character NOT matched by the following character class:
    Pattern.rangeFor (character range). Match any character within the range from code point U+0030 to code point U+0039 (both ends inclusive)
LastNode
Node. Accept match

The regex [^2[^[^0-9]]] compiles to the same program as [^2[^0-9]] due to the same bug.

There is an unresolved bug that seems to be of the same nature: JDK-6609854.


Explanation

Preliminary

Below are implementation details of Pattern class that one should know before reading further:

  • Pattern class compiles a String into a chain of nodes, each node is in charge of a small and well-defined responsibility, and delegates the work to the next node in the chain. Node class is the base class of all the nodes.
  • CharProperty class is the base class of all character-class related Nodes.
  • BitClass class is a subclass of CharProperty class that uses a boolean[] array to speed up matching for Latin-1 characters (code point <= 255). It has an add method, which allows characters to be added during compilation.
  • CharProperty.complement, Pattern.union, Pattern.intersection are methods corresponding to set operations. What they do is self-explanatory.
  • Pattern.setDifference is asymmetric set difference.

Parsing character class at first glance

Before looking at the full code of CharProperty clazz(boolean consume) method, which is the method responsible for parsing a character class, let us look at an extremely simplified version of the code to understand the flow of the code:

private CharProperty clazz(boolean consume) {
    // [Declaration and initialization of local variables - OMITTED]
    BitClass bits = new BitClass();
    int ch = next();
    for (;;) {
        switch (ch) {
            case '^':
                // Negates if first char in a class, otherwise literal
                if (firstInClass) {
                    // [CODE OMITTED]
                    ch = next();
                    continue;
                } else {
                    // ^ not first in class, treat as literal
                    break;
                }
            case '[':
                // [CODE OMITTED]
                ch = peek();
                continue;
            case '&':
                // [CODE OMITTED]
                continue;
            case 0:
                // [CODE OMITTED]
                // Unclosed character class is checked here
                break;
            case ']':
                // [CODE OMITTED]
                // The only return statement in this method
                // is in this case
                break;
            default:
                // [CODE OMITTED]
                break;
        }
        node = range(bits);

        // [CODE OMITTED]
        ch = peek();
    }
}

The code basically reads the input (the input String converted to null-terminated int[] of code points) until it hits ] or the end of the String (unclosed character class).

The code is a bit confusing with continue and break mixing together inside the switch block. However, as long as you realize that continue belongs to the outer for loop and break belongs to the switch block, the code is easy to understand:

  • Cases ending in continue will never execute the code after the switch statement.
  • Cases ending in break may execute the code after the switch statement (if it doesn't return already).

With the observation above, we can see that whenever a character is found to be non-special and should be included in the character class, we will execute the code after the switch statement, in which node = range(bits); is the first statement.

If you check the source code, the method CharProperty range(BitClass bits) parses "a single character or a character range in a character class". The method either returns the same BitClass object passed in (with new character added) or return a new instance of CharProperty class.

The gory details

Next, let us look at the full version of the code (with the part parsing character class intersection && omitted):

private CharProperty clazz(boolean consume) {
    CharProperty prev = null;
    CharProperty node = null;
    BitClass bits = new BitClass();
    boolean include = true;
    boolean firstInClass = true;
    int ch = next();
    for (;;) {
        switch (ch) {
            case '^':
                // Negates if first char in a class, otherwise literal
                if (firstInClass) {
                    if (temp[cursor-1] != '[')
                        break;
                    ch = next();
                    include = !include;
                    continue;
                } else {
                    // ^ not first in class, treat as literal
                    break;
                }
            case '[':
                firstInClass = false;
                node = clazz(true);
                if (prev == null)
                    prev = node;
                else
                    prev = union(prev, node);
                ch = peek();
                continue;
            case '&':
                // [CODE OMITTED]
                // There are interesting things (bugs) here,
                // but it is not relevant to the discussion.
                continue;
            case 0:
                firstInClass = false;
                if (cursor >= patternLength)
                    throw error("Unclosed character class");
                break;
            case ']':
                firstInClass = false;

                if (prev != null) {
                    if (consume)
                        next();

                    return prev;
                }
                break;
            default:
                firstInClass = false;
                break;
        }
        node = range(bits);

        if (include) {
            if (prev == null) {
                prev = node;
            } else {
                if (prev != node)
                    prev = union(prev, node);
            }
        } else {
            if (prev == null) {
                prev = node.complement();
            } else {
                if (prev != node)
                    prev = setDifference(prev, node);
            }
        }
        ch = peek();
    }
}

Looking at the code in case '[': of the switch statement and the code after the switch statement:

  • The node variable stores the result of parsing a unit (a standalone character, a character range, a shorthand character class, a POSIX/Unicode character class or a nested character class)
  • The prev variable stores the compilation result so far, and is always updated right after we compiles a unit in node.

Since the local variable boolean include, which records whether the character class is negated, is never passed to any method call, it can only be acted upon in this method alone. And the only place include is read and processed is after the switch statement.

Post under construction

nhahtdh
  • 52,949
  • 15
  • 113
  • 149
  • That is quite interesting. Thank you for your input. – Pshemo Feb 23 '14 at 06:45
  • 2
    @Pshemo: I wrote to core-lib-dev, and it turns out the problem has been discussed and a patch has been suggested [since 2011](http://mail.openjdk.java.net/pipermail/core-libs-dev/2011-June/006957.html). I did explain the current behavior in more details in [the email I wrote to them](http://mail.openjdk.java.net/pipermail/core-libs-dev/2014-February/025314.html). Should I update this post with those info? – nhahtdh Feb 26 '14 at 13:48
  • That is very impressive. My knowledge about how regex is implemented in Java is very limited. I think it would be good idea to include this info to your answer (especially part about `case:']'` in `CharProperty clazz(boolean consume)` method) to show how/why regex is handling our special cases in way it does, and how can this be corrected. – Pshemo Feb 26 '14 at 14:21
  • I was going to accept your answer earlier, but unfortunately I seeing how these double negation cases are treated without explaining "why is it handled this way" feel like this answer was uncompleted. Whit this part I will probably be able to understand it better. Thank you for your time. – Pshemo Feb 26 '14 at 14:27
  • 1
    @Pshemo: You can just take your time (and I will also take mine, since I'm a bit tired right now). – nhahtdh Feb 26 '14 at 15:17
  • Is this still "*under construction*" and does "*Currently, the program is an embarrassing mess. I will update this post with a link when I finished it and refactored it*." still apply? :) – user1803551 May 14 '15 at 17:57
  • 2
    @user1803551: The program here https://github.com/nhahtdh/pattern-dissector (it's way behind my current code, but should output something similar to the above). However, I don't have the time to finish the rest of the analysis. It will take 6-8 hours of focused work (it's half-joke, but half-true) – nhahtdh May 14 '15 at 18:48