In a previous article, I stressed a common problem with all PHP op-code caches/accelerators: they die with segmentation faults every once in a while.

To get around this problem, here is a script that would restart Apache when a segmentation fault is detected.

This script was written by Firebright Inc., with a few modifications, such as sending an email notice.

Here is the logwatcher.php script:

<?php
// path to apache log file
//
define("DEFAULT_APACHE_LOG_PATH", "/var/log/apache2/error.log");

// command to use to restart apache
//
define("DEFAULT_APACHE_RESTART_COMMAND", "/etc/init.d/apache2 restart");

// defines the polling interval (in seconds)
//
define("DEFAULT_POLLING_INTERVAL", 45);

// defines the format for date outputted in log entries (RFC 2228 format date)
//
define("DATE_FORMAT", "[r]");

// where to log watcher status
//
define("LOG_OUTPUT_FILENAME", "/var/log/logwatcher.log");

// conditions to test for (action is top level array element key)
//
$array_action_checks = Array();
$array_action_checks['restart'] = Array('exit signal Segmentation fault');

// list of commands mapped to actions
//
$array_action_commands = Array('restart' => DEFAULT_APACHE_RESTART_COMMAND);

/************************************************************
* END CONFIGURATION, BEGIN IMPLEMENTATION *
************************************************************/

$last_position = 0;
// main loop
//

if ($argc != 2) {
log_message("Called with incorrect number of arguments");
echo "Usage: php logwatcher.php youremail@example.com\n";
exit(1);
}
else {
$email = $argv[1];
}


log_message("logwatcher started");

while (true) {
$last_position = check_file($last_position);
sleep(DEFAULT_POLLING_INTERVAL);
}

function check_file($last_position) {
$file_name = DEFAULT_APACHE_LOG_PATH;
$fp = @fopen($file_name, "r");
if ($fp == null) {
die("unable to open file at $file_name\n");
}
if ($last_position == 0) {
// first time through the file for this instance.. Skip to EOF
//
fseek($fp, 0, SEEK_END);
} else {
// seek to last known position to skip past already handled log entries
//
fseek($fp, $last_position, SEEK_SET);
}

// check for patterns on current line
//
$action_taken = false;
while (($line = fgets($fp, 4096)) != null) {
$action = check_line($line);
if ($action != "") {
// TODO: log that action is taken
//
// take action only once for a given seek, otherwise seek silently to EOF
//
if (!$action_taken) {
log_message("Apache APC/eAccelerator caused a segmentation fault.");
log_message("Executing: " . get_action_command($action));
system(get_action_command($action));
log_message("Executed: " . get_action_command($action));
email_notify();
log_message("Email notification sent");
$action_taken = true;
}
}
}
// record end of file position for next pass through
//
$last_position = ftell($fp);

// close the file pointer
//
fclose($fp);
return $last_position;
}

function log_message($message) {
error_log(date(DATE_FORMAT) . " " . $message . "\n", 3, LOG_OUTPUT_FILENAME);
}

function check_line($line) {
global $array_action_checks;
// walk through each action
//
foreach ($array_action_checks as $action => $array_checks) {
foreach ($array_checks as $check) {
// walk through each check and see if it matches the current line
//
if (preg_match("/" . $check . "/", $line)) {
return $action;
}
}
}
return "";
}
function get_action_command($action) {
global $array_action_commands;
$command = @$array_action_commands[$action];
if ($command == null) {
log_message("Could not retrieve command for action: $action");
return "";
}
return $command;
}


function email_notify() {
$body = "The server has encountered an APC/eAccelerator segmentation fault error.
Apache has been automatically restarted.
The log file " . LOG_OUTPUT_FILENAME . " should have the exact time and number
that this happened.";

mail($email, 'Apache has been restarted', $body);
}

And here is the logwatcher.sh shell script that is used to start it. Change the email addresses to fit your needs.

#!/bin/sh

BASE_DIR=/root/bin
SCRIPT=$BASE_DIR/logwatcher.php
PID_FILE=/var/run/logwatcher.pid
EMAIL=someone@example.com,someoneelse@example.com

# If there is an old process, kill it
kill `cat $PID_FILE`
# Make sure the file is clean
rm -f $PID_FILE

cd $BASE_DIR
nohup php $SCRIPT $EMAIL> /dev/null &
PID=$!

echo $PID > $PID_FILE

Now, all you need to do is edit your /etc/rc.local, and add a line to call the logwatcher.sh script upon booting.

Resources and Links

Comments

Thu, 2011/08/25 - 05:04

Great script

Just one question. If I want to check for other strings, is this the correct way to add to the array?

$array_action_checks['restart'] = Array('exit signal Segmentation fault','still did not exit','Out of memory: kill process','php invoked oom-killer');

Wed, 2008/01/23 - 05:46

Is there any way to simulate a seg fault or otherwise test that this script is working properly? I believe I have everything in place but don't know how to be sure.

Thanks for the script. This will be really useful for our site.

Fri, 2008/01/25 - 05:32

One way to simulate it is to call the url: "www.example.com/exit signal Segmentation fault"

That works for us.

Best
/Johs.

Thu, 2013/04/18 - 15:14

I am replying to a old post but this shows how easily to force the Apache server to restart with this script. It can be used as DOS attack by keep sending the crafted URL request, "www.example.com/exit signal Segmentation fault", to the server.

This script should not be used unless there is a way to rectify this security hole.

Thu, 2008/02/21 - 15:33

The best way to simulate this script would be to fetch one of apache's child PIDs and perform:

kill -s SIGSEGV [PID]

Thu, 2008/01/24 - 12:28

Thanks for a very good script! However after installing logrotation, the script stoped working. We figured out that it was because the filepointer was pointing out of the file after the log had been rotated.
Therefore we changed the line:


if ($last_position == 0) {

to:


if ($last_position == 0 || ($last_position > filesize($file_name)))

Best,
/Johs.

Tue, 2008/01/29 - 17:09

I'm a minimalist by nature so here is my version of the script:

- Our error log files can get big (500 Megs) so opening the file and reading each line wasn't efficient.
- By using tail to read the last line in the log file it makes it easy.
- Configure your notifications as needed
- Configure your method for restarting apache

Crontab entry:
* * * * * /usr/bin/php /path/to/the/script/log_check.php > /dev/null

$rst = exec("tail -n 1 /var/log/httpd/error_log");

if (preg_match("/exit signal Segmentation fault/", $rst) == 1)
{
#print "APACHE NEEDED TO BE RESTARTED";
exec("service httpd restart"); # this is what ever you use to restart apache #
#print "APACHE RESTARTED";
mail('your@email.com', 'SERVERNAME - Seg Fault Restart','FYI');
} else {
#print "ALL COOL";
}

?>

Fri, 2008/12/19 - 06:29

Why start a heavy PHP interpreter, if you can do it in bash with a few lines :-)

You could add several logs to be followed, or use a complex regex (I'm using egrep here).

---CUT---
#!/bin/bash

errLog="/var/log/apache2/error.log"
apacheRestart="/etc/init.d/apache2 restart"
failRegex="exit signal Segmentation fault"
scriptLog="/data/logs/http/`basename $0`.log"

while true; do
echo "`date` (re)starting control loop"
tail --follow=name --retry -n 0 "$errLog" 2>/dev/null | while read logLine; do
if [[ `echo "$logLine" | egrep "$failRegex"` ]]; then
echo "`date` Segfault detected, restarting apache"
$apacheRestart
break
fi
done
sleep 5
done >> "$scriptLog" 2>&1 &
---CUT---

I installed it to send me an e-mail on failure, works like a charm! For Centos use httpd, otherwise identical:

#!/bin/bash

errLog="/var/log/httpd/error_log"
apacheRestart="/etc/init.d/httpd restart"
failRegex="exit signal Segmentation fault"
scriptLog="/data/logs/http/`basename $0`.log"

while true; do
echo "`date` (re)starting control loop"
tail --follow=name --retry -n 0 "$errLog" 2>/dev/null | while read logLine; do
if [[ `echo "$logLine" | egrep "$failRegex"` ]]; then
echo "`date` Segfault detected, restarting apache"| mail -s "Apache Restarted Notification." somemail@someone.com
$apacheRestart
break
fi
done
sleep 5
done

Thu, 2010/09/30 - 12:11

Great article! Was reading up about the performance comparison between APC, eAccelerator, and XCache.

How about mon? I know that it can't beat a bash/shell script with a tail command, but it can help monitor not just segfaults.

http://www.kernel.org/software/mon/faq.html

More info about it
http://www.debianhelp.co.uk/mon.htm

Pages

Is your Drupal or Backdrop CMS site slow?
Is it suffering from server resources shortages?
Is it experiencing outages?
Contact us for Drupal or Backdrop CMS Performance Optimization and Tuning Consulting