Stack Overflow
Stack Overflow

Questions tagged [logfile-analysis]

The act of analyzing `logfiles` using methods such as, but not limited to: `parsing` and `deciphering`, etc.

The act of analyzing logfiles using methods such as, but not limited to: parsing and deciphering, etc.

77 questions
131
votes
11 answers

Parsing huge logfiles in Node.js - read in line-by-line

I need to do some parsing of large (5-10 Gb)logfiles in Javascript/Node.js (I'm using Cube). The logline looks something like: 10:00:43.343423 I'm a friendly log message. There are 5 cats, and 7 dogs. We are in state "SUCCESS". We need to read each…
victorhooi
  • 14,225
  • 20
  • 79
  • 112
38
votes
1 answer

How can I tell when my dataset in R is going to be too large?

I am going to be undertaking some logfile analyses in R (unless I can't do it in R), and I understand that my data needs to fit in RAM (unless I use some kind of fix like an interface to a keyval store, maybe?). So I am wondering how to tell ahead…
Heather Stark
  • 607
  • 7
  • 18
25
votes
10 answers

Software for Webserver Log Analysis?

Can I get some recommendations (preferably with some reasons) for good log analysis software for Apache 2.2 access log files? I have heard of Webalizer and AWStats, but have never really used any of them, and would like to know: What they can…
kaybenleroll
  • 15,996
  • 16
  • 51
  • 66
21
votes
5 answers

Logfile analysis in R?

I know there are other tools around like awstats or splunk, but I wonder whether there is some serious (web)server logfile analysis going on in R. I might not be the first thought to do it in R, but still R has nice visualization capabilities and…
Matt Bannert
  • 25,237
  • 34
  • 134
  • 195
10
votes
3 answers

Exe created with py2exe doesn't work and returns logfile with errors

It's my first post so go easy on me. My problem is similar as in here, but I couldn't make it work, though I really tried: " py2exe with enthought and pandas ". Having installed separately both versions of python(x84-64, x32) with libraries I…
lottee
  • 113
  • 1
  • 7
8
votes
4 answers

Searching huge log files

Troubleshooting, analyzing & filtering log files is by far one of the most laborious daily jobs. My issue is searching through a log file, which could be well over 4 gigs in size. Simply loading the file takes up to 15 mins. I'm running a fairly…
Will
  • 81
  • 1
  • 2
8
votes
3 answers

Plotting many lines as a heatmap

I have a large number (~1000) of files from a data logger that I am trying to process. If I wanted to plot the trend from a single one of these log files I could do it using plot(timevalues,datavalues) I would like to be able to view all of these…
Hugoagogo
  • 1,465
  • 15
  • 30
8
votes
3 answers

extracting error information from rails log files

i am developing on 5 different rails projects, plus also refactoring some (moving from older rails versions to 2.3) - what is the best way to extract the error information from the logfiles, so i can see all the depreciation warnings, runtime errors…
z3cko
  • 3,034
  • 8
  • 38
  • 59
7
votes
2 answers

How is MapReduce a good method to analyse http server logs?

I've been looking at MapReduce for a while, and it seems to be a very good way to implement fault-tolerant distributed computing. I read a lot of papers and articles on that topic, installed Hadoop on an array of virtual machines, and did some very…
Nicolas
  • 2,088
  • 1
  • 17
  • 25
7
votes
4 answers

Perfmon File Analysis Tools

I have a bunch of perfmon files that have captured information over a period of time. Whats the best tool to crunch this information? Idealy I'd like to be able to see avg stats per hour for the object counters that have been monitored.
Daniel Pollard
  • 1,232
  • 1
  • 12
  • 11
6
votes
3 answers

How do I create a graph in Zabbix with a calucated field based on the count of log entries?

I have an item setup to monitor fatal errors, and I want to add another item so that it calculates the count of the fatal errors and graphs them. Using this key: log["/d2/httpd/logs/myDomain-error_log","PHP Fatal","UTF-8",100] It properly gets the…
Kevin Korb
  • 136
  • 1
  • 9
6
votes
4 answers

How do I read information from text files?

I have hundreds of text files with the following information in each file: *****Auto-Corelation Results****** 1 .09 -.19 .18 non-Significant *****STATISTICS FOR MANN-KENDELL TEST****** S= 609 VAR(S)= 162409.70 Z= …
Geekuna Matata
  • 1,229
  • 5
  • 19
  • 34
4
votes
3 answers

Is there any "generic log file analyzer/editor"?

I am asking because I am sick of watching tab separated log files everyday like this: 164.56.95.119:18260 24986 2011-03-13 16:10:02.532 -5 JobQueueReceiver JSM 6019 4 Audit.RTUsage.JSM removed…
Michael Mao
  • 8,868
  • 21
  • 71
  • 91
4
votes
1 answer

What do the square brackets in LaTeX logs mean?

I'm currently working on a parser that reads complete LaTeX logs. Most of the log format is, though weird, easy to figure out, but these square brackets are puzzling me. Here's an example from near the end of one of my logs: Overfull \hbox…
Stefan Majewsky
  • 5,177
  • 2
  • 23
  • 49
4
votes
1 answer

Building indexes for files in Perl

I'm currently new to Perl, and I've stumbled upon a problem : My task is to create a simple way to access a line of a big file in Perl, the fastest way possible. I created a file consisting of 5 million lines with, on each line, the number of the…
Jonathan Taws
  • 1,108
  • 10
  • 24
1
2 3 4 5 6