• notifying system administrators when something goes wrong,
• stopping services when system performance is dropping too low and starting them when it's going up again,
• periodically restarting memory-leaking processes,
• dumping system statistics in critical situations,

• load average,
• used and free memory amount,
• used and free swap space,

To get the latest release, please visit sysfence home page at SourceForge.

You may also get sysfence.spec file from PLD GNU/Linux CVS repository to build RPM package quickly.

Sysfence reads it's configuration from file(s) specified in argument list. Config files may contain one or more rules describing conditions and actions to be performed.

Rule has syntax like this:

if {
   resource1 > limit1
   or
   { resource2 < limit2 and resource3 < limit3 }
}
run once 'command-to-be-run'

The block enclosed within {} brackets describes condition. When it's result is TRUE, following command is invoked.

The once keyword is optional. If present, the command is executed only once after condition becomes TRUE. Next execution will take place only if condition becomes FALSE and then TRUE again. Without once keyword, command is invoked periodically, after every resource check that gives TRUE, no matter what was the condition result before.

Command specified right after run keyword is passed to /bin/sh, so it may contain more than one instruction or even whole script. But be careful - rule checking is suspended unless command execution has been completed! (Other rules are unaffected.)

As resources, following ones can be given:

• la1 - load average during last minute.
• la5 - load average during last 5 minutes.
• la15 - load average during last 15 minutes.
• memfree - lower limit for free memory amount.
• memused - upper limit for memory used by processes.
• swapfree - lower limit for free swap space.
• swapused - upper limit for swap space in use.

Do you have problems with MySQL server choking and freezing whole system? I do. To find queries that cause problems, you may use:

if {
    la1 >= 8.00
} run once 'echo "SHOW FULL PROCESSLIST" | mysql | mail my@email.com'

Of course, that wouldn't prevent your system from being blocked, but following rule could. MySQL will be restarted if LA for last minute is over 10.0 or LA for last five minutes is over 6.0.

if { la1 >= 10.00 or la5 >= 6.0 }
run '/etc/rc.d/init.d/mysql stop; sleep 120; /etc/rc.d/init.d/mysql start'

We may also restart some services that probably have memory leaks and use lots of swap space if not restarted periodically. Let's assume that 256MB of used swap is enough to give our Zope server a break.

if {
    swapused >= 256M
} run '/etc/rc.d/init.d/zope restart'

We may also alert admins... Notice that you don't have to be r00t:

if {
    la15 > 4.0
    and
    {
        swapfree < 64M
        or
        memfree < 128M
    }
} run 'echo "i wish you were here..." | sendsms +48ADMINCELLPHONE'

Using sysfence version 0.7 or later you may give rule a name that will be used in logs:

rule "high load" { la1 > 3.0 and la15 > 2.0 } log

rule keyword has the same meaning as if. There are also synonymes for other keywords. Detailed list is included within sysfence package.

Sysfence has undergone generel redesign. New sources are available in CVS repository, but weren't released yet (some functionality is still missing).
All data structures are described in DIA file accessible via CVS. You may also see PNG version of data structures graph.

Sysfence is being developed by:

• Michał Sałaban
  e-mail: emes /at/ pld-linux org,
  jabber: emes@jabber.org
• Mirek Kopertowski
  e-mail: m.kopertowski /at/ post pl

This software is released under GNU Public License version 2.0