0

I've got a CentOS installation with a busy webserver. I need to aquire stats from the log files and keep the old ones, ordered by date.

Every 30th second, the current logfile should be closed and processed (analysing entries and storing them into a database). Since this generates a lot of logfiles, I want to group them into directories, named by date.

At the moment, I have two files; rotation.conf and rotatenow.sh. The shell-file creates directories, based on YmdHMS. After that, I run the command "logrotate ./rotation.conf -v --force" in order to invoke the proces, but how do I make the config-file put the log into the newly generated directory? Can the whole thing be done inside the config-file?

now="$(date)"
now="$(date +'%Y-%m-%d-%H:%M:%S')"
foldernavn="/var/www/html/stats/logs/nmdstats/closed/$now"
mkdir $foldernavn
logrotate ./nmdhosting.conf -v --force

At the moment, the config-file looks like this:

/var/www/html/stats/logs/nmdhosting/access_log {
    ifempty
    missingok
    (I am stuck)
    (do some post-processing - run a Perl-script)
}

Any ideas would be deeply appreciated.

Update: I tried a different approach, adding this to the httpd.conf:

TransferLog "|usr/sbin/rotatelogs /var/www/html/stats/logs/nmdstats/closed/activity_log.%Y%m%d%H%M%S 30".

It works, but apparently, it can't run a pre/post processing script when using this method. This is essential in order to update the database. I could perhaps run a shell/Perl-script using a cronjob, but I don't trust that method. The search goes on...

update 2: I've also tested cronolog but the - for my project - required functionalities haven't been implemented yet, but are on the to-do. Since the latest version is from 2002, I'm not going to wait around for it to happen :)

However, I was unaware of the inotify-tools, so I managed to set up a listener:

srcdir="/var/www/html/stats/logs/nmdstats/history/"
inotifywait -m -e create $srcdir |
while read filename eventlist eventfile
do
    echo "This logfile has just been closed: $eventfile"
done

I think, I can handle it from here. Thank you, John

Anders
  • 197
  • 2
  • 11

1 Answers1

1

No need for cron: if you use the TranserLog httpd.conf option to create a new log file every 30 seconds, you can run a post-processing daemon which watches the output directory with inotifywait (or Python's pyinotify, etc.). See here: inotify and bash - this will let you get notified by the OS very soon after a new file is created etc.

Community
  • 1
  • 1
John Zwinck
  • 207,363
  • 31
  • 261
  • 371