44

It is a well known fact that modern regular expression implementations (most notably PCRE) have little in common with the original notion of regular grammars. For example you can parse the classical example of a context-free grammar {anbn; n>0} (e.g. aaabbb) using this regex (demo):

~^(a(?1)?b)$~

My question is: How far can you go? Is it also possible to parse the context-sensitive grammar {anbncn;n>0} (e.g. aaabbbccc) using PCRE?

NikiC
  • 95,987
  • 31
  • 182
  • 219
  • 3
    What the heck do `(?R)` and `~` do? What is wrong with `preg_match('a*b*c*',...)`? – Chriszuma Sep 15 '11 at 16:37
  • 7
    First of all, PHP regexes need delimiters. That's what the `~` is for. Second of all, a*b*c* matches acccccccccc – NullUserException Sep 15 '11 at 16:39
  • 6
    @Chriszuma: The `~` are just delimiters (you could use `/` and many other characters as well). The `(?R)` signifies recursion. It means "put the whole regular expression in here again". – NikiC Sep 15 '11 at 16:39
  • Ah, I was still thinking about Python regexes, my mistake. – Chriszuma Sep 15 '11 at 16:43
  • @NikiC: With the recursion in place `(?R)`, you are mainly looking for nested matches. With that example, looks more like a case for backreferencing. But backreferencing content is one thing, backreferencing content dimensions is another; not sure PCRE can handle that directly, unless the dimensions are known up front. – Orbling Sep 15 '11 at 16:45
  • 1
    That's correct about `a*b*c*` matching acccccccccccc, but I don't see what's wrong with `a+b+c+`. – Justin Morgan Sep 15 '11 at 16:46
  • 1
    @Chriszuma: Python does not support recursive regexp in the `re` module, it was a contender to be a feature [but was dropped from consideration](http://bugs.python.org/msg83993). The only alternative in Python is [pyparsing](http://stackoverflow.com/questions/1656859/how-can-a-recursive-regexp-be-implemented-in-python). – Orbling Sep 15 '11 at 16:48
  • Justin Morgan, same problem, it will match `abcccc`. – Qtax Sep 15 '11 at 16:48
  • 1
    As an aside: the purpose of a pattern matching grammar such as Regular Expressions is to *simplify* programming. If trying to create a fancy pattern is harder to do than with some simple iterative code, then you shouldn't bother with the RegEx. For this particular case, I'd search the string for instances of `b+` and check before and after the matches to see if an identical `a` and `c` cluster exist. – zzzzBov Sep 15 '11 at 16:49
  • 4
    Wait, I understand now. The tricky part is that `n` has to be the same for each group, correct? So if there are 5 `a`s, then there have to be exactly 5 each of `b` and `c`. No, I know that's not possible without recursion, and if you can do it even with recursion, then I certainly don't know how. – Justin Morgan Sep 15 '11 at 16:49
  • 1
    @Orbling: I don't know either, that's why I'm asking ;) PCRE surprises me again and again, so I could well imagine that there is some dirty hack to make it possible ;) – NikiC Sep 15 '11 at 16:50
  • @zzzzBov: Yes, you are right. This question is more of the theoretical kind, to understand how powerful PCRE really is. Not something you would use practically ;) – NikiC Sep 15 '11 at 16:58
  • 1
    @NikiC: Aye, people massively underestimate the power available. Though there is most likely a limit. ;-) Could be done if the maximum length of a run was relatively short I imagine. Have you seen the new(-ish) [PCRE code facility `(?{ code })`](http://perldoc.perl.org/perlre.html#Extended-Patterns)? Modern versions of Perl support it. That could be used to keep a count per group. – Orbling Sep 15 '11 at 17:05
  • @Orbling: But that would be cheating ;) That wouldn't be a pure regex solution anymore ;) – NikiC Sep 15 '11 at 17:07
  • 1
    @NikiC: Many would argue that `(?R)` isn't either. ;-) – Orbling Sep 15 '11 at 17:22
  • Matching an even numbers of char-groups is easier: `~(a(?1)?b)(c(?2)?d)~` – Bart Kiers Sep 15 '11 at 17:27
  • Bart Kiers, that wouldn't work, as `abcccddd` would match. You still need that recursive lookahead to make sure the number of b's and c's is the same. – Qtax Sep 15 '11 at 21:22
  • it may be possible to do it with regular expressions but only for limited cases. In general, you need context free grammars, try flex or antlr – Dan Bizdadea Sep 15 '11 at 16:45
  • You are going to love this one: [How can we match a^n b^n with Java regex?](http://stackoverflow.com/questions/3644266/how-can-we-match-an-bn-with-java-regex/) - the appendix has an answer for your question. – Kobi Sep 19 '11 at 06:38
  • @Kobi: Yep, the alternative solution is interesting. Though I prefer the recursive one, as it's easier to grasp. – NikiC Sep 19 '11 at 16:07
  • Real regex have only the operations concatenation (a concat b = ab), alternation (a alternate b = a|b), and Kleene star (a repeated 0 or more times). So they are pretty useless compared to modern regexen and we almost never mean the formal definition when we talk about regex. So just forget about formal theory when discussing regex because it doesn't apply – Brandon Nov 24 '11 at 11:59

4 Answers4

35

Inspired by NullUserExceptions answer (which he already deleted as it failed for one case) I think I have found a solution myself:

$regex = '~^
    (?=(a(?-1)?b)c)
     a+(b(?-1)?c)
$~x';

var_dump(preg_match($regex, 'aabbcc'));    // 1
var_dump(preg_match($regex, 'aaabbbccc')); // 1
var_dump(preg_match($regex, 'aaabbbcc'));  // 0
var_dump(preg_match($regex, 'aaaccc'));    // 0
var_dump(preg_match($regex, 'aabcc'));     // 0
var_dump(preg_match($regex, 'abbcc'));     // 0

Try it yourself: http://codepad.viper-7.com/1erq9v


Explanation

If you consider the regex without the positive lookahead assertion (the (?=...) part), you have this:

~^a+(b(?-1)?c)$~

This does nothing more than check that there's an arbitrary number of as, followed by an equal number of bs and cs.

This doesn't yet satisfy our grammar, because the number of as must be the same, too. We can ensure that by checking that the number of as equals the number of bs. And this is what the expression in the lookahead assertion does: (a(?-1)?b)c. The c is necessary so we don't only match a part of the bs.


Conclusion

I think this impressively shows that modern regex is not only capable of parsing non-regular grammars, but can even parse non-context-free grammars. Hopefully this will lay to rest the endless parroting of "you can't do X with regex because X isn't regular"

NikiC
  • 95,987
  • 31
  • 182
  • 219
  • Close. Fails on `abbcc`. You've got it checking for a `b` to match each `a`, and a `c` to match each `b`, but if you are short on `a`'s, nobody will know. I bet with a lookbehind you could make it work. – Chriszuma Sep 15 '11 at 17:17
  • @JBernardo: Fixed (hopefully) – NikiC Sep 15 '11 at 17:19
  • @Chriszuma: I think it's fixed now (with an additional assertion) – NikiC Sep 15 '11 at 17:20
  • 2
    I think it is. You can simplify your first lookahead though, with `c` instead of `(?!b)` since it is already inside a lookahead group. – Chriszuma Sep 15 '11 at 17:25
  • @Chriszuma Thanks! Was thinking to complicated ;) – NikiC Sep 15 '11 at 17:46
  • 7
    +1 for showing that modern regex implementations *can* understand non-regular, and even non-context-free grammars. Hopefully this will lay to rest the endless parroting of "you can't do `X` with regex because `X` isn't regular". But that's wishful thinking, huh? – NullUserException Sep 15 '11 at 17:51
  • @NullUserException: I added that to the end of the answer. Hope you don't mind :) – NikiC Sep 15 '11 at 17:55
  • This is a whole lot easier in *real* Perl patterns. You may able to effect the same result by using code callbacks in PCRE, but I’ve never tried there. – tchrist Sep 15 '11 at 18:18
  • 2
    I would assert that regex is the wrong tool for the job in a case such as this one. Just because you *can* use a scalpel to cut a steak doesn't mean you *should*. – zzzzBov Oct 14 '11 at 17:25
  • 1
    @zzzzBov The question (and answer) are of purely theoretic nature ;) I assert that I will not use that code in production, if it makes you happy :) – NikiC Oct 14 '11 at 17:36
  • The codepad link is broken. – Don Hatch Jan 24 '21 at 01:11
  • Very cool. I think it can alternatively be done using a lookbehind instead of lookahead: `~^(a(?-1)?b)c+(?<=a(b(?-1)?c))$~x` Unfortunately in php that's unimplemented; it provokes: `preg_match(): Compilation failed: lookbehind assertion is not fixed length at offset 13` It does seem to work in python3 with `regex` module, although relative references apparently aren't supported so they have to be changed to absolute: `import regex; regex.search(r"^(a(?1)?b)c+(?<=a(b(?2)?c))$", "aabbcc") is not None` etc. – Don Hatch Jan 24 '21 at 01:44
11

Here is an alternative solution using balancing groups with .NET regex:

^(?'a'a)+(?'b-a'b)+(?(a)(?!))(?'c-b'c)+(?(b)(?!))$

Not PCRE, but may be of interest.

Example at: http://ideone.com/szhuE

Edit: Added the missing balancing check for the group a, and an online example.

Qtax
  • 31,392
  • 7
  • 73
  • 111
  • 2
    @NikiC: Actually they're quite simple. In .NET, every time you use the named capture syntax `(?'a' ... )`, it is actually pushing the capture onto a *stack* of captures. Then, the syntax `(?'-a' ... )` pops an item off the `a` stack (and fails if the stack is empty). You can also use a capture stack in a conditional regex, which is what the syntax `(?(a) ... )` is. A stack evaluates as `true` if it contains items. – porges Sep 23 '11 at 05:34
  • 2
    So the code above does: push all 'a' captures onto a stack, push all 'b' captures onto a stack (while popping off an 'a' for each 'b'), then assert that the 'a' stack is empty (`(?!)` being an unconditional failure), then do the same thing with the 'c's, using the 'b' stack. – porges Sep 23 '11 at 05:35
  • 1
    @Porges, just a note, it doesn't have to be a named capture group. Unnamed work just as well (except that `(?'-x'...)` does not create a new/add to a stack). A shorter version of my regex would be: `^(a)+(?'b-1'b)+(?(1)(?!))(?'-b'c)+(?(b)(?!))$` – Qtax Sep 23 '11 at 14:12
11

My question is: How far can you go?

In the interests of not creating code that is an unreadable morass of punctuation, I'm going to risk the downvotes and answer a different, though very much related, question: how far should you go?

Regular expression parsers are a brilliant thing to have in your toolkit but they are not the be all and end all of programming. The ability to write parsers in a readable manner is also a brilliant thing to have in your toolkit.

Regular expressions should be used right up to the point where they start making your code hard to understand. Beyond that, their value is dubious at best, damaging at worst. For this specific case, rather than using something like the hideous:

~^(?=(a(?-1)?b)c)a+(b(?-1)?c)$~x

(with apologies to NikiC), which the vast majority of people trying to maintain it are either going to have to replace totally or spend substantial time reading up on and understanding, you may want to consider something like a non-RE, "proper-parser" solution (pseudo-code):

# Match "aa...abb...bcc...c" where:
# - same character count for each letter; and
# - character count is one or more.

def matchABC (string str):
    # Init string index and character counts.
    index = 0
    dim count['a'..'c'] = 0

    # Process each character in turn.
    for ch in 'a'..'c':
        # Count each character in the subsequence.
        while index < len(str) and str[index] == ch:
            count[ch]++
            index++

    # Failure conditions.
    if index != len(str):        return false # did not finish string.
    if count['a'] < 1:           return false # too few a characters.
    if count['a'] != count['b']: return false # inequality a and b count.
    if count['a'] != count['c']: return false # inequality a and c count.

    # Otherwise, it was okay.
    return true

This will be far easier to maintain in the future. I always like to suggest to people that they should assume those coming after them (who have to maintain the code they write) are psychopaths who know where you live - in my case, that may be half right, I have no idea where you live :-)

Unless you have a real need for regular expressions of this kind (and sometimes there are good reasons, such as performance in interpreted languages), you should optimise for readability first.

paxdiablo
  • 772,407
  • 210
  • 1,477
  • 1,841
  • 4
    **1.** These patterns are not meant for production code, they're purely recreational. **2.** Your code allows other characters - `abc!` would return true. **3.** You don't mind the order - `caabbc` would also return true. **4.** Like most code: with more lines of code come more errors. **5.** You can count the characters in a single pass: `while index < len(str): count[str[index]]++, index++`. **6.** You can test and [comment](http://stackoverflow.com/q/4034386) patterns. it looks great. – Kobi Sep 20 '11 at 04:17
  • 1
    @Kobi, you need to re-examine my code and the specifications in the question. It does not allow more characters past the end, that's handled by the first failure check. It does not allow out-of-order abc sequences - the `for` loop takes care of that. In fact your point 5 is the incorrect one since it will allow abcabc to pass, which is clearly incorrect. – paxdiablo Sep 20 '11 at 04:40
  • I should also mention that the correlation between lines of code and errors is not always so - it's generally between code _complexity_ and errors. Sometimes complexity and lines of code are related but that's not always the case. By way of example, the above code is not that complex despite its line count. – paxdiablo Sep 20 '11 at 04:50
  • @Kobi, no probs, I thought it might have been a simple misunderstanding of the `'a'..'c'` bit, with you thinking it might mean those characters in any order rather than sequential as I intended. That's the danger inherent in using airy-fairy pseudo-code rather than a specific language :-) – paxdiablo Sep 20 '11 at 04:57
  • 1
    @paxdiablo 1. As Kobi already said: The question is more of the theoretical type, not the practical one. I don't know any place where a^n b^n c^n has any practical use. It's just theory. – NikiC Sep 20 '11 at 13:32
  • @paxdiablo 2. As Kobi already showed your code isn't entirely easy to grasp either. I personally would have chosen a mix of regex and code in production. A match against `(a+)(b+)(c+)` and subsequent check that the `strlen` of all captured groups is the same would be the most understandable variant for most people (imho). – NikiC Sep 20 '11 at 13:37
  • @paxdiablo 3. Hm, forgot that 3. was. So just +1 as your point is basically right, though doesn't quite apply in this situation (as it's theory). – NikiC Sep 20 '11 at 13:43
  • _How_ you achieve the readability is open to discussion, I have no problem with that. It may be that a simpler regex with checking captured group lengths _is_ easier to understand. I wrote a "proper" parser (albeit more complex than absolutely necessary) since it's a fairly easy task for me, given my background. My contention is not that what I wrote is the ideal answer, just that you should in most cases favour readability over "cleverness" :-) – paxdiablo Sep 20 '11 at 13:46
  • @paxdiablo Just on a sidenote, my personal favorite would be: `return preg_match('/^a+b+c+$/', $str) && 1 === count(array_count_values(count_chars($str, 1)));` – NikiC Sep 20 '11 at 14:03
  • Don't forget that you can cause whitespace to be insignificant in almost every regex flavor, allowing you to insert newlines and comments into the regex anywhere to improve readability – Brandon Nov 24 '11 at 12:01
  • this is why your write unit tests. first you assert that your sollution works, then you tidy it up without having to worry about introducing new bugs! – sara Feb 19 '16 at 07:20
  • This is a very interesting (in my opinion) theory question about PCRE; this "answer" is a gratuitous derailing lecture about something else. Posts like this make this site increasingly unpleasant for me. What I don't understand is, why does this clutter appear higher in the page than either of the real answers below? (@NikiC 's and Qtax's) – Don Hatch Jan 24 '21 at 01:08
3

Qtax Trick

A solution that wasn't mentioned:

^(?:a(?=a*(\1?+b)b*(\2?+c)))+\1\2$

See what matches and fails in the regex demo.

This uses self-referencing groups (an idea @Qtax used on his vertical regex).

Community
  • 1
  • 1
zx81
  • 38,175
  • 8
  • 76
  • 97
  • 1
    @Unihedron Yes, saw that, but his solution here uses balancing groups (.NET), whereas this one uses self-referencing groups (Perl, PCRE). Completely different beasts—but of course you'd know that from the [line number question](http://stackoverflow.com/q/23727476/1078583) :) – zx81 Jul 29 '14 at 07:03
  • Hehe, *"Qtax trick"*, thanks, but I have to give credit to [polygenelubricants](http://stackoverflow.com/users/276101/polygenelubricants) for that kind of trick. I learned of it from his great regex questions and answers, like [How can we match a^n b^n with Java regex?](http://stackoverflow.com/questions/3644266/how-can-we-match-an-bn-with-java-regex/) – Qtax Jan 20 '15 at 18:18