5

How should I script daemontools superviser?

I've started to use D. J. Bernsteins' daemontools to keep some processes running and is working fine. But now I need to monitor a couple of additional conditions on those processes and I've failed to find good info on how to do that.

My scenario is that I have some processes running for a web app (pharo smalltalk virtual machines) and they respond http, each one in their port (that's for the for loadbalance). I would like to somehow ping those to verify that they are not only running but responding to http requests. If they don't respond in a certain way to a request for more than 30s they should be treated as crashed and simply be restarted.

Is this even possible with daemontools? if so, how should I write this script and where should I place it? or where's the documentation on this?

Martijn Pieters
  • 889,049
  • 245
  • 3,507
  • 2,997
pauel
  • 876
  • 10
  • 26

2 Answers2

6

The simplest solution is to create another daemontool task with a script that sleeps for 30 seconds and then tests for the presence of the service (using wget or curl for example). If the service doesn't respond timely you can restart the service (svc -t yourapp) and/or send a notification. The run-script of the new service could look as simple as this:

#!/bin/sh
sleep 30
if ! wget --quiet --timeout=5 --delete-after "http://yourapp.com/" ; then
  svc -t /etc/service/yourapp
fi

I've also made good experience with tools like Munin. Again you need to provide a script that provides information about the state of your image. If you setup your images with a REST service you can even provide really interesting metrics such as active sessions, inactive session, gc parameters, memory consumption, database statistics, ... The tool then draws nice graphs over time and lets you specify boundaries to get notified when things behave badly.

Lukas Renggli
  • 8,210
  • 21
  • 44
  • Thanx. but is there no way to do this in the supervisor itself? – pauel May 21 '12 at 17:16
  • Yes, see the first paragraph of my answer. – Lukas Renggli May 21 '12 at 19:13
  • yeah I got that. it's just that I thought the supervisor that runs the daemon could do that in any way. Anyways your solution helps me a lot. Many thanks – pauel May 21 '12 at 22:02
  • @Lukas - I think what pauel has in mind is a situation where the webserver runs in the foreground under supervise and sends its log lines to STDOUT/STDERR to be handler by the daemontools logging utility instead of writing directly to its own application log. Which of course is no problem if you're writing your own webserver, but will generally require some hacking if you're not. And it doesn't address the "pingability" requirement - which is of course the point of your suggestion to "wrap" the webserver with a REST API. – Peter Aug 13 '12 at 11:15
3

To detect if the service is responsive, you can use curl (a perfect fit for HTTP and command-line/shell scripts)

curl --connect-timeout 10 http://8.8.8.8 
curl: (28) connect() timed out!

and you can write more sophisticated things, like checking that the response is 200 (OK) etc.

Sean DeNigris
  • 6,024
  • 27
  • 36
Igor Stasenko
  • 1,039
  • 5
  • 7