17
<table((?!</table>).)*</table>

matches all my table tags, however, 

<table(.(?!</table>))*</table>

does not. The second one seems to make sense if I try to write out the expression in words, but I can't make sense of the first.

Can someone explain the difference to me?

For reference, I got the term `Tempered Greedy Token' from here: http://www.rexegg.com/regex-quantifiers.html#tempered_greed

jmrah
  • 3,633
  • 2
  • 20
  • 30
  • As an aside, note that this "tempered" way is particularly inefficient. – Casimir et Hippolyte Jun 17 '15 at 19:42
  • 1
    Then .. _he_ made it up. In fact, this is not standard jargon in regex land. And if I were to take a poll, am betting %99 of regex guru's would laugh at it. –  Mar 16 '17 at 16:08
  • Well, the author of that website seems pretty guru to me. Plus, I think it's helpful to have names for patterns - regex or otherwise. I would just let the gurus laugh. Btw, is there a more standard name for this pattern? – jmrah Mar 16 '17 at 17:02
  • Yeah. I would call it the _Record Seperator Construct_ because that's it's only use. It's the last resort, most _ineficient_ way to match anything. Same goes for using an assertion at the beginning of a regex (this must never be done!!). –  Mar 16 '17 at 17:11
  • 1
    I'm sorry, but I'll ask you the same question you asked me: Did you make `Record Seperator Construct` up? I doubt that is any more standard than `Tempered Greedy Token`. In fact, google gives me zero hits for that phrase. – jmrah Mar 17 '17 at 00:30
  • Well buddy, you didn't ask me what `Record $eperator Construct` is: `Regex1: (?s)(?=(.+?)(helloWorld|$))\1\2 Completed iterations: 50 / 50 ( x 1000 ) Matches found per iteration: 4 Elapsed Time: 0.91 s, 906.47 ms, 906468 µs` Now compare that with `Temered Greedy Token` : `Regex2: (?s)(?:(?!helloWorld).)+(?:helloWorld|$) Completed iterations: 50 / 50 ( x 1000 ) Matches found per iteration: 4 Elapsed Time: 1.74 s, 1744.49 ms, 1744490 µs` Which one shows up on google ? –  Mar 18 '17 at 04:12
  • Here's another one `Record $eperator Construct: (?s)))\1\2 Completed iterations: 50 / 50 ( x 1000 ) Matches found per iteration: 3 Elapsed Time: 0.58 s, 581.96 ms, 581957 µs` versus `Temered Greedy Token: (?s)
    ).)*
    Completed iterations: 50 / 50 ( x 1000 ) Matches found per iteration: 3 Elapsed Time: 1.63 s, 1627.50 ms, 1627500 µs` Do you really think that guy is a guru when he espouses _crud_ and actually makes up a _faux_ name for it ?...     –  Mar 18 '17 at 04:29
  • It seems like you are taking issue with two separate things. One, that the expression is inefficient, and two, that the author of rexegg made up a term for it. For the first issue, I feel it necessary to point out that the efficiency of the pattern was never a part of the original question, though I do thank you for taking the time to post those benchmarks! For you second issue, you're going to have to ask the author why he chose that term. I can't answer that one! It looks like he has a comments section at the bottom of this page though. You could try that. – jmrah Mar 18 '17 at 14:55
  • @jrahhali - The question is labeled `Temper Greedy Token` of which you post a link to a guy's website who made this phrase up and is promoting a Regex Consulting Service. This phrase cannot be branded to a sequence of regex constructs, period! Any by you _pushing_ it as if it is something relavent to something significant, undermines the fact that regex is a flowing language that can only be defined by individual constructs in _techniques_ that can't be defined. Even _unrolled loop_, a common and more meningfull phrase, is irrelavent. –  Mar 18 '17 at 18:21
  • So, I wouldn't take this form as some _gold standard_ since it comes in many shapes `))*
    ` _after_. What's real here is assertions, not some branded technique. And, should you ever be actually looking for something specific, it should never be checked with a _leading_ assertion. This put's a 2x performance hit right out of the gate.     –  Mar 18 '17 at 18:52
  • Well, you're definitely entitled to your opinion, and thanks for sharing it! I personally see value in naming constructs, so disagree with you on that point. Further, I didn't intend to *push* anything as some *gold standard*. It was just an straight-forward, innocent question. But even if I intended to, I don't see a problem with that. I'm sorry if that ruffles some feathers. I suggest you create an answer with all the points you've listed. That would probably be the most helpful thing to do here, while at the same time, reign in this dialog from becoming more off topic. What do you think? – jmrah Mar 18 '17 at 19:54
  • @jrahhali - Ah, yeah.. I see it time to time. New to regex people, grasping at coined phrases as if trying to memorize some way they have seen something done. It's human nature to do that. I try to subtly enlighten, but sometimes it doesn't work. Here is a _fact_ for you - regex is an _applied_ science, it's not theoretical. It changes shape and form _every_ time. There is _no_ technique. You are _alone_ with a variety of singular constructs and an idea of what you want to create. You can only get better by practice, and a lot of it. Memorizing phrases won't help at all, good luck to you! –  Mar 18 '17 at 21:47
  • Good luck to you too! – jmrah Mar 18 '17 at 22:27

3 Answers3

51

Since Google returns this SO question on top of the results for the tempered greedy token, I feel obliged to provide a more comprehensive answer.

What is a Tempered Greedy Token?

The rexegg.com tempered greedy token reference is quite concise:

In (?:(?!{END}).)*, the * quantifier applies to a dot, but it is now a tempered dot. The negative lookahead (?!{END}) asserts that what follows the current position is not the string {END}. Therefore, the dot can never match the opening brace of {END}, guaranteeing that we won't jump over the {END} delimiter.

That is it: a tempered greedy token is a kind of a negated character class for a character sequence (cf. negated character class for a single character).

NOTE: The difference between a tempered greedy token and a negated character class is that the former does not really match the text other than the sequence itself, but a single character that does not start that sequence. I.e. (?:(?!abc|xyz).)+ won't match def in defabc, but will match def and bc, because a starts the forbidden abc sequence, and bc does not.

It consists of:

  • (?:...)* - a quantified non-capturing group (it may be a capturing group, but it makes no sense to capture each individual character) (a * can be +, it depends on whether an empty string match is expected)
  • (?!...) - a negative lookahead that actually imposes a restriction on the value to the right of the current location
  • . - (or any (usually single) character) a consuming pattern.

However, we can always further temper the token by using alternations in the negative lookahead (e.g. (?!{(?:END|START|MID)})) or by replacing the all-matching dot with a negated character class (e.g. (?:(?!START|END|MID)[^<>]) when trying to match text only inside tags).

Consuming part placement

Note there is no mentioning of a construction where a consuming part (the dot in the original tempered greedy token) is placed before the lookahead. Avinash's answer is explaining that part clearly: (.(?!</table>))* first matches any character (but a newline without a DOTALL modifier) and then checks if it is not followed with </table> resulting in a failure to match e in <table>table</table>. The consuming part (the .) MUST be placed after the tempering lookahead.

When to use tempered greedy token?

Rexegg.com gives an idea:

  • When we want to match a block of text between Delimiter 1 and Delimiter 2 with no Substring 3 in-between (e.g. {START}(?:(?!{(?:MID|RESTART)}).)*?{END}
  • When we want to match a block of text containing a specific pattern inside without overflowing subsequent blocks (e.g. instead of lazy dot matching as in <table>.*?chair.*?</table>, we'd use something like <table>(?:(?!chair|</?table>).)*chair(?:(?!<table>).)*</table>).
  • When we want to match the shortest window possible between 2 strings. Lazy matching won't help when you need to get abc 2 xyz from abc 1 abc 2 xyz (see abc.*?xyz and abc(?:(?!abc).)*?xyz).

Performance Issue

Tempered greedy token is resource-consuming as a lookahead check is performed after each character matched with the consuming pattern. Unrolling the loop technique can significantly increase tempered greedy token performance.

Say, we want to match abc 2 xyz in abc 1 abc 2 xyz 3 xyz. Instead of checking each character between abc and xyz with abc(?:(?!abc|xyz).)*xyz, we can skip all characters that are not a or x with [^ax]*, and then match all a that are not followed with bc (with a(?!bc)) and all x that are not followed with yz (with x(?!yz)): abc[^ax]*(?:a(?!bc)[^ax]*|x(?!yz)[^ax]*)*xyz.

Community
  • 1
  • 1
Wiktor Stribiżew
  • 484,719
  • 26
  • 302
  • 397
  • 3
    After this post being resureccted in the comments section, I've decided to accept your very fleshed-out answer instead. Thank you for taking the time to put this together. – jmrah Mar 18 '17 at 15:00
9

((?!</table>).)* would checks for that particular character going to be matched must not be a starting character in the string </table>. If yes, then only it matches that particular character. * repeats the same zero or more times.

(.(?!</table>))* matches any character only if it's not followed by </table>, zero or more times . So this would match all the chars inside the table tag excpet the last character, since the last char is followed by </table>. And the following pattern </table> asserts that there must be a closing table tag at the end of the match. This makes the match to fail.

See here

Avinash Raj
  • 160,498
  • 22
  • 182
  • 229
  • I'm still struggling with understanding the first paragraph. Good news is that I understand your explanation about why (.(?!))* fails. EDIT: Ohhh, ok, i think i understand now! – jmrah Jun 17 '15 at 19:53
3

A tempered greedy token really just means:

"match, but only up to a point"

how you do it:

you put the token you don't want to match as a negative lookahead (?!notAllowedToMatch) in front of a dot . (match any one thing), then you repeat that whole thing with a star *:

((?!notAllowedToMatch).)*

how it works:

"look, and eat one" over and over, moving one character at time from left to right through the input string, until the disallowed sequence (or end of string) is seen, at which point the match stops.

Wiktor's more detailed answer is nice, I just thought a simpler explanation was in order.

Scott Weaver
  • 6,328
  • 2
  • 23
  • 37