152

I'm trying to set up a cron job as a sort of watchdog for a daemon that I've created. If the daemon errors out and fails, I want the cron job to periodically restart it... I'm not sure how possible this is, but I read through a couple of cron tutorials and couldn't find anything that would do what I'm looking for...

My daemon gets started from a shell script, so I'm really just looking for a way to run a cron job ONLY if the previous run of that job isn't still running.

I found this post, which did provide a solution for what I'm trying to do using lock files, not I'm not sure if there is a better way to do it...

BenMorel
  • 30,280
  • 40
  • 163
  • 285
LorenVS
  • 11,817
  • 9
  • 43
  • 54

16 Answers16

150

Use flock. It's new. It's better.

Now you don't have to write the code yourself. Check out more reasons here: https://serverfault.com/a/82863

/usr/bin/flock -n /tmp/my.lockfile /usr/local/bin/my_script
Community
  • 1
  • 1
Jess
  • 20,424
  • 18
  • 108
  • 130
123

I do this for a print spooler program that I wrote, it's just a shell script:

#!/bin/sh
if ps -ef | grep -v grep | grep doctype.php ; then
        exit 0
else
        /home/user/bin/doctype.php >> /home/user/bin/spooler.log &
        #mailing program
        /home/user/bin/simplemail.php "Print spooler was not running...  Restarted." 
        exit 0
fi

It runs every two minutes and is quite effective. I have it email me with special information if for some reason the process is not running.

jjclarkson
  • 5,658
  • 5
  • 35
  • 61
63

As others have stated, writing and checking a PID file is a good solution. Here's my bash implementation:

#!/bin/bash

mkdir -p "$HOME/tmp"
PIDFILE="$HOME/tmp/myprogram.pid"

if [ -e "${PIDFILE}" ] && (ps -u $(whoami) -opid= |
                           grep -P "^\s*$(cat ${PIDFILE})$" &> /dev/null); then
  echo "Already running."
  exit 99
fi

/path/to/myprogram > $HOME/tmp/myprogram.log &

echo $! > "${PIDFILE}"
chmod 644 "${PIDFILE}"
rsanden
  • 1,130
  • 13
  • 18
  • 3
    +1 Using a pidfile it's probably much safer than grepping for a running program with the same name. – Elias Dorneles Oct 06 '12 at 03:22
  • /path/to/myprogram & > $HOME/tmp/myprogram.log & ?????? did you perhaps mean /path/to/myprogram >> $HOME/tmp/myprogram.log – matteo Nov 13 '12 at 19:31
  • 1
    Shouldn't the file be removed when the script finishes? Or am I missing something very obvious? – Hamzahfrq Jul 19 '14 at 17:05
  • @matteo: I prefer not appending, because I run these for years. Appending is fine if that's what you want. – rsanden Jul 20 '14 at 23:52
  • @Hamzahfrq: Do you mean the PID file? I don't see any benefit to removing it, because you can't rely on that outcome (you still have to account for the corner case when it wouldn't be removed anyway). Predictable behavior is better. – rsanden Jul 20 '14 at 23:56
  • Could you please explain how this script is working? I have modified it to generate the PID file every time it runs and remove it before exiting. So, it checks in the beginning and exits if the file exists. The PID file is removed on each reboot in case it is present there. But I think you r script has some other way to handle it – Hamzahfrq Jul 21 '14 at 14:46
  • @Hamzahfrq there was an extra & before the > that's what was confusing; I notice you removed it. Makes sense both to append or not of course. – matteo Jul 21 '14 at 14:53
  • 1
    @matteo: Yes, you're right. I had fixed that in my notes years ago but forgot to update it here. Worse, I missed it in your comment too, noticing only "`>`" versus "`>>`". Sorry about that. – rsanden Jul 23 '14 at 00:05
  • 5
    @Hamzahfrq: Here's how it works: The script first checks to see if the PID file exists ("`[ -e "${PIDFILE}" ]`". If it does not, then it will start the program in the background, write its PID to a file ("`echo $! > "${PIDFILE}"`"), and exit. If the PID file instead does exist, then the script will check your own processes ("`ps -u $(whoami) -opid=`") and see if you're running one with the same PID ("`grep -P "^\s*$(cat ${PIDFILE})$"`"). If you're not, then it will start the program as before, overwrite the PID file with the new PID, and exit. I see no reason to modify the script; do you? – rsanden Jul 23 '14 at 00:07
  • The PID approach was chosen, I think in case the program name appeared elsewhere in PS. Isn't there a risk that a PID will be reused by the OS and result in a false match? Deletion of the calling PID file would fix it, but it's a little tricky to do in the scope of the this wrapper script. You may need an internal wrapper script to call the program and delete the PID file when done. If you do go for the original approach though and lock by program name but also by username, then at least the user has control over the locking credential. – aghsmith Jun 28 '20 at 11:29
40

It's suprising that no one mentioned about run-one. I've solved my problem with this.

 apt-get install run-one

then add run-one before your crontab script

*/20 * * * * * run-one python /script/to/run/awesome.py

Check out this askubuntu SE answer. You can find link to a detailed information there as well.

Bedi Egilmez
  • 1,190
  • 13
  • 25
  • 2
    What's important to mention is that *this tool is available out of the box in Ubuntu* 20 (and maybe the versions prior as well) – Adam Sibik Dec 26 '20 at 10:22
22

Don't try to do it via cron. Have cron run a script no matter what, and then have the script decide if the program is running and start it if necessary (note you can use Ruby or Python or your favorite scripting language to do this)

Earlz
  • 57,517
  • 89
  • 275
  • 484
  • 5
    The classic way is to read a PID file that the service creates when it starts, check if the process with that PID is still running, and restart if not. – tvanfosson Mar 02 '10 at 21:02
9

You can also do it as a one-liner directly in your crontab:

* * * * * [ `ps -ef|grep -v grep|grep <command>` -eq 0 ] && <command>
quezacoatl
  • 115
  • 1
  • 2
  • 5
    not very safe, what if there are other commands that matches the search for grep? – Elias Dorneles Oct 06 '12 at 03:22
  • 1
    This could also be written as * * * * * [ `ps -ef|grep [c]ommand` -eq 0 ] && where wrapping the first letter of your command in brackets excludes it from the grep results. – Jim Clouse Feb 05 '13 at 19:01
  • I had to use the following syntax: `[ "$(ps -ef|grep [c]ommand|wc -l)" -eq 0 ] && ` – thameera Mar 24 '16 at 19:36
  • 2
    This is hideous. `[ $(grep something | wc -l) -eq 0 ]` is a really roundabout way to write `! grep -q something`. So you want simply `ps -ef | grep '[c]ommand' || command` – tripleee Sep 19 '16 at 11:16
  • (Also, as an aside, if you really wanted to actually count the number of matching lines, that's `grep -c`.) – tripleee Sep 19 '16 at 11:17
7

The way I am doing it when I am running php scripts is:

The crontab:

* * * * * php /path/to/php/script.php &

The php code:

<?php
if (shell_exec('ps aux | grep ' . __FILE__ . ' | wc  -l') > 1) {
    exit('already running...');
}
// do stuff

This command is searching in the system process list for the current php filename if it exists the line counter (wc -l) will be greater then one because the search command itself containing the filename

so if you running php crons add the above code to the start of your php code and it will run only once.

talsibony
  • 7,378
  • 5
  • 42
  • 41
  • This is what I needed as all the other solutions required installing something on the client server, which I don't have access to do. – Jeff Davis Jan 19 '17 at 22:26
5

As a follow up to Earlz answer, you need a wrapper script that creates a $PID.running file when it starts, and delete when it ends. The wrapper script calls the script you wish to run. The wrapper is necessary in case the target script fails or errors out, the pid file gets deleted..

Byron Whitlock
  • 49,611
  • 27
  • 114
  • 164
  • Oh cool... I never thought about using a wrapper... I couldn't figure out a way to do it using lock files because I couldn't guarantee that the file would get deleted if the daemon errored out... A wrapper would work perfectly, I'm going to give jjclarkson's solution a shot, but I'll do this if that doesn't work... – LorenVS Mar 02 '10 at 21:06
4

With lockrun you don't need to write a wrapper script for your cron job. http://www.unixwiz.net/tools/lockrun.html

Aaron C. de Bruyn
  • 2,310
  • 1
  • 28
  • 37
3

I would recommend to use an existing tool such as monit, it will monitor and auto restart processes. There is more information available here. It should be easily available in most distributions.

Zitrax
  • 16,107
  • 15
  • 79
  • 98
  • Every answer except this one answers the surface question "How can my cron job make sure it only runs one instance?" when the real question is "How can I keep my process running in the face of restarts?", and the correct answer is indeed to not use cron, but instead a process supervisor like monit. Other options include [runit](http://smarden.org/runit/), [s6](https://www.skarnet.org/software/s6/) or, if your distribution already uses systemd, just creating a systemd service for the process that needs to be kept alive. – clacke Dec 31 '18 at 04:01
3

This one never failed me:

one.sh:

LFILE=/tmp/one-`echo "$@" | md5sum | cut -d\  -f1`.pid
if [ -e ${LFILE} ] && kill -0 `cat ${LFILE}`; then
   exit
fi

trap "rm -f ${LFILE}; exit" INT TERM EXIT
echo $$ > ${LFILE}

$@

rm -f ${LFILE}

cron job:

* * * * * /path/to/one.sh <command>
DJV
  • 1,140
  • 9
  • 20
3
# one instance only (works unless your cmd has 'grep' in it)
ALREADY_RUNNING_EXIT_STATUS=0
bn=`basename $0`
proc=`ps -ef | grep -v grep | grep "$bn" | grep -v " $$ "`
[ $? -eq 0 ] && {
    pid=`echo $proc | awk '{print $2}'`
    echo "$bn already running with pid $pid"
    exit $ALREADY_RUNNING_EXIT_STATUS
}

UPDATE .. better way using flock:

/usr/bin/flock -n /tmp/your-app.lock /path/your-app args 
ekerner
  • 4,995
  • 1
  • 32
  • 28
1

I'd suggest the following as an improvement to rsanden's answer (I'd post as a comment, but don't have enough reputation...):

#!/usr/bin/env bash

PIDFILE="$HOME/tmp/myprogram.pid"

if [ -e "${PIDFILE}" ] && (ps -p $(cat ${PIDFILE}) > /dev/null); then
  echo "Already running."
  exit 99
fi

/path/to/myprogram

This avoids possible false matches (and the overhead of grepping), and it suppresses output and relies only on exit status of ps.

Community
  • 1
  • 1
dbenton
  • 179
  • 1
  • 8
  • 1
    Your `ps` command would match PIDs for other users on the system, not just your own. Adding a "`-u`" to the `ps` command changes the way the exit status works. – rsanden Jul 21 '14 at 00:15
1

Simple custom php is enough to achieve. No need to confuse with shell script.

lets assume you want to run php /home/mypath/example.php if not running

Then use following custom php script to do the same job.

create following /home/mypath/forever.php

<?php
    $cmd = $argv[1];
    $grep = "ps -ef | grep '".$cmd."'";
    exec($grep,$out);
    if(count($out)<5){
        $cmd .= ' > /dev/null 2>/dev/null &';
        exec($cmd,$out);
        print_r($out);
    }
?>

Then in your cron add following

* * * * * php /home/mypath/forever.php 'php /home/mypath/example.php'
lingeshram
  • 542
  • 5
  • 11
1

Docs: https://www.timkay.com/solo/

solo is a very simple script (10 lines) that prevents a program from running more than one copy at a time. It is useful with cron to make sure that a job doesn't run before a previous one has finished.

Example

* * * * * solo -port=3801 ./job.pl blah blah
Erlang Parasu
  • 129
  • 1
  • 2
  • 5
0

Consider using pgrep (if available) rather than ps piped through grep if you're going to go that route. Though, personally, I've got a lot of mileage out of scripts of the form

while(1){
  call script_that_must_run
  sleep 5
}

Though this can fail and cron jobs are often the best way for essential stuff. Just another alternative.

  • 2
    This would just start the daemon over and over again and does not solve the problem mentioned above. – cwoebker May 05 '13 at 02:48