I find daemontools to be incredibly useful for quickly turning scripts into daemons. I also like to run the latest version of Ubuntu, which means there is no (well-maintained) package available. There’s a package in ‘universe’ but it puts things in strange places and generally inspires distrust (explained below). As installing daemontools from source on Ubuntu hasn’t been straight forward (I had trouble both on 8.10 and 9.04), I’m gonna post a quick howto which pulls together a few pieces of information scattered throughout the internets.
Crash Course
daemontools comes with a bunch of tools. The full docs are here, but I’ll cover the basics.
- supervise is a daemon that takes a directory as an argument and tries to execute the file named ‘run’ in that directory, while keeping state in the ’supervise’ subdirectory. Naturally, you want to give it a folder with an executable file named ‘run’ and a wirtable subdirectory called ’supervise’ in it.
- svstat, svok, svc, etc are used to manipulate individual supervise jobs and check their status. These aren’t covered here
- svscan is a daemon that also takes a directory as an argument and proceeds to look for supervise-able directories within. Any time it finds one that doesn’t already have a corresponding supervise process, it fires one up. Thus if the supervise instance you count on to do some background processing crashes for a whatever reason, svscan will just start it back up
- svscanboot is the script that should be run at boot-time to start svscan. It will direct svscan to look for jobs in the /service directory.
In an ideal world, you throw a bunch of shit into /service, svscan picks it up and fires up a supervise process for all the subdirectories. It runs as a daemon and restarts supervise whenever needed (svscan itself is basically bulletproof in my experience). When your machine is rebooted for whatever reason, svscanboot restarts svscan and everything resumes as before. The reason I choose not to use the Ubuntu package is because it does not create this ideal world for me.
Compiling
First of all, daemontools won’t compile as is. Follow along the instructions here until you get to the exciting part where you type ‘package/install’ and then get an error message that looks like this:
/usr/bin/ld: errno: TLS definition in /lib/libc.so.6 section .tbss
mismatches non-TLS reference in envdir.o
/lib/libc.so.6: could not read symbols: Bad value
collect2: ld returned 1 exit status
make: *** [envdir] Error 1
Never fret. I have found a couple of possible fixes, but the patch below appears to be the cleanest one. [1]
diff -ur daemontools-0.76.old/src/error.h daemontools-0.76/src/error.h
--- daemontools-0.76.old/src/error.h 2001-07-12 11:49:49.000000000 -0500
+++ daemontools-0.76/src/error.h 2003-01-09 21:52:01.000000000 -0600
@@ -3,7 +3,7 @@
#ifndef ERROR_H
#define ERROR_H
-extern int errno;
+#include <errno.h>
extern int error_intr;
extern int error_nomem;
Assuming you are still at the point in the installation instructions right before the ‘package/install’ command, copy the patch into /tmp/daemontools-errno.patch, then
$ cd src
$ patch < /tmp/daemontools-errno.patch
$ cd ..
Then resume the original daemontools instructions as prescribed.
Testing
The installation creates two folders: /command and /service. The former contains all the different daemontools commands, while the latter is the default folder which the svscan daemon scans for jobs. To test that our installation worked, let’s create a proper test job (you’ll probably want to be superuser for this, since /service is owned by root by default)
$ cd /service
$ mkdir tester
$ mkdir tester/supervise
$ touch tester/run
$ chmod +x tester/run
in tester/run, put any script. It can be empty, but if you really want to verify things are working, make it output something to a tmp file. Sample:
#!/bin/bash
echo "Foo!" >> /tmp/foo.log
sleep 5
Once everything is in place, as superuser run the following commands. If your output looks different, something’s broke.
$ /command/svscan /service &
[1] 9641
$ ps -ef | grep svscan
root 9641 17108 0 18:12 pts/1 00:00:00 /command/svscan /service
mikhailp 9646 17108 0 18:12 pts/1 00:00:00 grep svscan
$ ps -ef | grep supervise
root 9642 9641 0 18:12 pts/1 00:00:00 supervise tester
mikhailp 9660 17108 0 18:12 pts/1 00:00:00 grep supervise
If you made your tester script log something, check that the output is showing up where expected. If all is well, shut ‘er down and move on.
$ killall svscan
$ killall supervise
Starting at Bootup
Assuming you want to user supervise for some critical offline part of your application, you want it to start automatically whenever your server boots up. You don’t want to have to manually start up svscan every time.
Ubuntu uses upstart for task management instead of sysvinit. In order to let upstart start and stop the svscan daemon correctly, the following script needs to be copied into /etc/event.d/svscan [2]
# svscan - daemontools
#
# This service starts daemontools from the point the system is
# started until it is shut down again.
start on startup
start on runlevel 1
start on runlevel 2
start on runlevel 3
start on runlevel 4
start on runlevel 5
start on runlevel 6
stop on shutdown
exec /command/svscanboot
Once that’s in place, you can use start svscan and stop svscan. Try starting it and checking ps output as described in the Testing section. Finally, reboot the box (if possible) and repeat the same checks. At this point, svscan should start up with your system and spawn supervise processes on any job it finds in /service.
That’s all there is to it. Enjoy.
1: Patch originally found here, linked from here.
2: Corrected from this entry posted in 2006 which has some obsolete info
