How to delimit both "=" and "==" in Java when reading

Question

I want to be able to output both "==" and "=" as tokens.

For example, the input text file is:

biscuit==cookie apple=fruit+-()

The output:

biscuit
=
=
cookie
apple
=
fruit
+
-
(
)

What I want the output to be:

biscuit
==
cookie
apple
=
fruit
+
-
(
)

Here is my code:

    Scanner s = null;
    try {
        s = new Scanner(new BufferedReader(new FileReader("input.txt")));
        s.useDelimiter("\\s|(?<=\\p{Punct})|(?=\\p{Punct})");

        while (s.hasNext()) {

            String next = s.next();
            System.out.println(next);
       }
    } finally {
        if (s != null) {
            s.close();
        }
    }

Thank you.

Edit: I want to be able to keep the current regex.

Answer 1

Just split the input string according to the below regex .

String s = "biscuit==cookie apple=fruit"; 
String[] tok = s.split("\\s+|\\b(?==+)|(?<==)(?!=)");
System.out.println(Arrays.toString(tok));

Output:

[biscuit, ==, cookie, apple, =, fruit]

Explanation:

• \\s+ Matches one or more space characters.
• | OR
• \\b(?==+) Matches a word boundary only if it's followed by a = symbol.
• | OR
• (?<==) Lookafter to = symbol.
• (?!=) And match the boundary only if it's not followed by a = symbol.

Update:

String s = "biscuit==cookie apple=fruit+-()"; 
String[] tok = s.split("\\s+|(?<!=)(?==+)|(?<==)(?!=)|(?=[+()-])");
System.out.println(Arrays.toString(tok));

Output:

[biscuit, ==, cookie, apple, =, fruit, +, -, (, )]

Answer 2

In other words you want to split on

1. one or more whitespaces
2. place which has = after it and non-= before it (like foo|= where | represents this place)
3. place which has = before it it and non-= after it (like =|foo where | represents this place)

In other words

s.useDelimiter("\\s+|(?<!=)(?==)|(?<==)(?!=)");
//             ^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^
//cases:         1)        2)        3)

Since it looks like you are building parser I would suggest using tool which will let you build correct grammar like http://www.antlr.org/. But if you must stick with regex then other improvement which will let you build regex easier would be using Matcher#find instead of delimiter from Scanner. This way your regex and code could look like

    String data = "biscuit==cookie apple=fruit+-()";

    String regex = "<=|==|>=|[\\Q<>+-=()\\E]|[^\\Q<>+-=()\\E]+";
    Pattern p = Pattern.compile(regex);
    Matcher m = p.matcher(data);

    while (m.find())
        System.out.println(m.group());

Output:

biscuit
==
cookie apple
=
fruit
+
-
(
)

You can make this regex more general by using

String regex = "<=|==|>=|\\p{Punct}|\\P{Punct}+";
//                       ^^^^^^^^^^ ^^^^^^^^^^^-- standard cases
//              ^^ ^^ ^^------------------------- special cases

Also this approach would require reading data from file first, and storing it in single String which you would parse. You can find many ways of how to read text from file for instance in this question: Reading a plain text file in Java

so you can use something like

String data = new String(Files.readAllBytes(Paths.get("input.txt")));

You can specify encoding which String should use while reading bytes from file by using constructor String(bytes, encoding). So you can write it as new String(butes,"UTF-8") or to avoid typos while selecting encoding use one of stored in StandardCharsets class like new String(bytes, StandardCharsets.UTF_8).

Answer 3

You might be able to qualify those punctuations with some additional assertions.

 # "\\s|(?<===)|(?<=\\p{Punct})(?!(?<==)(?==))|(?=\\p{Punct})(?!(?<==)(?==))"

   \s 
|  (?<= == )
|  (?<= \p{Punct} )
   (?!
        (?<= = )
        (?= = )
   )
|  (?= \p{Punct} )
   (?!
        (?<= = )
        (?= = )
   )

Info update

If some characters aren't covered in \p{Punct} just add them as a separate class within
the punctuation subexpressions.

For engines that don't do certain properties well inside classes, use this ->

 #  Raw:   \s|(?<===)|(?<=\p{Punct}|[=+])(?!(?<==)(?==))|(?=\p{Punct}|[=+])(?!(?<==)(?==))

    \s 
 |  (?<= == )
 |  (?<= \p{Punct} | [=+] )
    (?!
         (?<= = )
         (?= = )
    )
 |  (?= \p{Punct} | [=+] )
    (?!
         (?<= = )
         (?= = )
    )

For engines that handle properties well inside classes, this is a better one ->

 #  Raw:   \s|(?<===)|(?<=[\p{Punct}=+])(?!(?<==)(?==))|(?=[\p{Punct}=+])(?!(?<==)(?==))

    \s 
 |  (?<= == )
 |  (?<= [\p{Punct}=+] )
    (?!
         (?<= = )
         (?= = )
    )
 |  (?= [\p{Punct}=+] )
    (?!
         (?<= = )
         (?= = )
    )

Answer 4

(?===)|(?<===)|\s|(?<!=)(?==)|(?<==)(?!=)|(?=\p{P})|(?<=\p{P})|(?=\+)

You can try this.Se demo.

http://regex101.com/r/wQ1oW3/18