hlovdal is right. You can not solve this with a regex.
What you need (and perhaps intend) is a parser.
For such a simple syntax a recursive descending parser should do the job. Usually a parser consists of a lexer (or tokenizer) (which divides the input into tokens / terms, see StringTokenizer in Java API) and a grammar. Usually the single terms can be expressed via regex. In your case the tokens/terms would be numbers (\d+), brackets "()" and the keywords (AND, OR). After transforming your input into a sequence of tokens, you process one by one and track the state of the parser, usually by deciding what to do while having a look at the next token.
WhoAmI suggestion is really similar. By suggesting to count the brackets he suggests a really simple kind of parser. At least it's lexer has to differentiate between Bracket-Tokens and the rest. The parsers state would be the count of open brackets.
There are also several frameworks out there to help generating parsers, like ANTLR, yacc etc. But perhaps they're too complex for your purposes.