27

I need to monitor very different log files for errors, success status etc. And I need to grab corresponding metrics using Prometheus and show in Grafana + set some alerting on it. Prometheus + Grafana are OK I already use them a lot with different exporters like node_exporter or mysql_exporter etc. Also alerting in new Grafana 4.x works very well.

But I have quite a problem to find suitable exporter/ program which could analyze log files "on fly" and extract metrics from them.

So far I tried:

  • mtail (https://github.com/google/mtail) - works but existing version cannot easily monitor more files - in general it cannot bind specific mtail program (receipt for analysis) to some specific log file + I cannot easily add log file name into tag
  • grok_exporter (https://github.com/fstab/grok_exporter) - works but I can extract only limited information + one instance can monitor only one log file which mean I would have to start more instances exporting on more ports and configure all off them in prometheus - which makes too many new points of failure
  • fluentd prometheus exporter (https://github.com/kazegusuri/fluent-plugin-prometheus) - works but looks like I can extract only very simple metrics and I cannot make any advanced regexp analysis of a line(s) from log file

Does any one here has a really running solution for monitoring advanced metrics from log files using "some exporter" + Prometheus + Grafana? Or instead of exporter some program from which I could grab results using Prometheus push gateway. Thanks.

JosMac
  • 1,824
  • 1
  • 12
  • 20
  • Are the logs shipped/processes somewhere? Might be easier to hook into that process somehow. – Carl Bergquist Dec 15 '16 at 10:07
  • Logs are grabbed by fluentd so I tried this but metrics I can get using fluentd prometheus exporter seems to be very simple and limited. I tried to add external processing in my bash scripts in fluentd but I do not know why inside fluentd it was incredibly slow with long lags. Outside fluentd scripts were OK – JosMac Dec 15 '16 at 10:17

2 Answers2

15

Take a look at Telegraf. It does support tailing logs using input plugins logparser and tail. To export metrics as prometheus endpoint use prometheus_client output plugin. You also may apply on the fly aggregations. I've found it simpler to configure for multiple log files than grok_exporter or mtail

LazyS
  • 166
  • 1
  • 6
1

Those are the 3 answers currently for getting log data into Prometheus.

You could also look into getting whatever is producing the logs to expose Prometheus metrics directly.

brian-brazil
  • 24,975
  • 4
  • 64
  • 67
  • 1
    Problem is processes are quite different (bash scripts, go programs) but main problem is a lot of them are legacy things no one wants to fiddle with. So the safest way is to process log files. – JosMac Dec 15 '16 at 10:14
  • Problem also is I have at least 5+ very different log files for every instance/ server and 15+ instances – JosMac Dec 15 '16 at 10:18
  • 1
    @JosMac: Then you want centralized logging (e.g. with Graylog) and export metrics from there. – Martin Schröder Dec 15 '16 at 21:29
  • Thanks, @MartinSchröder - Graylog looks interesting but it is a complex solution and I just need some "tailing extractor" which will be able to calculate some metrics "on fly" and either expose them as web service or push them into Prometheus Push Gateway. Because I need to put metrics from log files into our overall monitoring and alerting in Grafana. – JosMac Dec 16 '16 at 12:47