15

How do you search Google App Engine logs in the new Cloud Console using regular expressions?

This blog post suggests you just need to type regex:my.*query to search, but that does not seem to work in the logging console. When I do that, it auto-corrects to the following query text:regex:my.*query.

speedplane
  • 14,130
  • 14
  • 78
  • 128

3 Answers3

10

The Stackdriver Logging product does not currently support regular expressions. It was originally supported a while back (as you saw in the blog post), but we found that it was rarely used and that many of those uses were for simple patterns that had simpler solutions without the performance and other penalties of regexes.

In basic filter mode (the default), text searches automatically are case-insensitive and match substrings of field values, and you can use ".." to represent numerical ranges. In advanced filter mode, the "has" operator accomplishes the same thing through using a : instead of an = in your filter expression, e.g. path.to.field: "value". (See also: Write effective advanced filters)

If these operators don't accomplish your goal, consider filing feedback through the speech bubble button in the top right of the Cloud Console providing details of your use case and what you're ultimately trying to accomplish, and we'll incorporate that feedback as we plan the future direction of the product.

Ben Rister
  • 306
  • 1
  • 4
  • Hm. I used regex all the time. Is there any way to easily "AND" two searches together? For example, I want to be able to search for `user1@example.com` and `timeout`, even though the terms would not be right next to eachother. – speedplane May 16 '16 at 20:26
  • If your logs are structured, simply searching for both of them does exactly that. If you're just doing free-text searches, you'd need to use advanced mode, since basic mode ORs like-typed searches together, while in advanced mode everything is ANDed by default (and you can really do anything you want there anyway). – Ben Rister May 17 '16 at 13:12
  • 1
    Okay thanks.. The default ORing in the simple mode really through me off. I've never seen a search interface default to ORing terms together. Would be nice if you could add search terms without going into advanced mode. – speedplane May 17 '16 at 15:00
  • FWIW, it's a little more nuanced than that in basic mode. _Different_ kinds of constraints are ANDed, _like_ kinds of constraints are ORed: if you wrote `path:/foo path:/bar`, you probably don't want that to return nothing because no single entry has both of those paths simultaneously. – Ben Rister May 19 '16 at 14:17
  • 1
    That's an incredibly confusing search interface right out of the box. Has the team considered moving to just using AND? Pretty much all search engines (e.g., google) work that way. – speedplane May 19 '16 at 16:23
  • Google web search actually doesn't work that way. =) In any case, we inherited this from the old GAE viewer and customers have dependences on this behavior, and it generally works better for basic use cases, even if it is a gotcha for advanced use cases. The new advanced filter interface does indeed have default AND across all clauses, and we expect more people to use it by default over time as we make some changes we have in the queue for it. – Ben Rister May 23 '16 at 14:55
  • 1
    Yes, to be pedantic, I'm sure Google's search algorithm is much more complex than ANDing terms together. That said, it is generally the case that more search terms leads to narrower results. I'm not sure I've seen a search bar that does the opposite (besides Cloud Console). – speedplane May 24 '16 at 01:09
  • Yeah, makes no sense to use OR IMHO. Same feedback here, switch to AND. – Jonas K Sep 25 '17 at 20:58
10

Although, I am late here but you can do it in stackdriver.

=           # equal
!=          # not equal
> < >= <=   # numeric ordering
:           # "has" matches any substring in the log entry field

In case, you want to find all GET response with 500 in textPayload , you need to add in filters:

textPayload:"500"

To search all zone with central1 in it:

resource.labels.zone:"-central1-"

That's it. You can refer this link for more advance filters

Shashank Vivek
  • 13,776
  • 7
  • 48
  • 88
2

Nowadays Google Cloud Operations Logging has support for regular expressions. This feature was published on 2020-09-17, see https://cloud.google.com/blog/products/management-tools/cloud-logging-gets-regular-expression-support. You regex works in this format:

textPayload=~"my.*query"

You can query regular expression matches with operator =~ and non-matches with operator !~:

=~          # regular expression search for a pattern
!~          # regular expression search not for a pattern

More info on syntax and examples can be found in the Google Cloud Operations Suite Logging query language reference: https://cloud.google.com/logging/docs/view/logging-query-language#regular-expressions

Veikko
  • 2,446
  • 2
  • 18
  • 25