logwatcher: restart Apache after a segmentation fault

In a previous article, I stressed a common problem with all PHP op-code caches/accelerators: they die with segmentation faults every once in a while.

To get around this problem, here is a script that would restart Apache when a segmentation fault is detected.

This script was written by Firebright Inc., with a few modifications, such as sending an email notice.

Here is the logwatcher.php script:

<?php
// path to apache log file
//
define("DEFAULT_APACHE_LOG_PATH", "/var/log/apache2/error.log");

// command to use to restart apache
//
define("DEFAULT_APACHE_RESTART_COMMAND", "/etc/init.d/apache2 restart");

// defines the polling interval (in seconds)
//
define("DEFAULT_POLLING_INTERVAL", 45);

// defines the format for date outputted in log entries (RFC 2228 format date)
//
define("DATE_FORMAT", "[r]");

// where to log watcher status
//
define("LOG_OUTPUT_FILENAME", "/var/log/logwatcher.log");

// conditions to test for (action is top level array element key)
//
$array_action_checks = Array();
$array_action_checks['restart'] = Array('exit signal Segmentation fault');

// list of commands mapped to actions
//
$array_action_commands = Array('restart' => DEFAULT_APACHE_RESTART_COMMAND);

/************************************************************
 * END CONFIGURATION, BEGIN IMPLEMENTATION                  *
 ************************************************************/

$last_position = 0;
// main loop
//

if ($argc != 2) {
  log_message("Called with incorrect number of arguments");
  echo "Usage: php logwatcher.php youremail@example.com\n";
  exit(1);
}
else {
  $email = $argv[1];
}


log_message("logwatcher started");

while (true) {
  $last_position = check_file($last_position);
  sleep(DEFAULT_POLLING_INTERVAL);
}

function check_file($last_position) {
  $file_name = DEFAULT_APACHE_LOG_PATH;
  $fp = @fopen($file_name, "r");
  if ($fp == null) {
    die("unable to open file at $file_name\n");
  }
  if ($last_position == 0) {
    // first time through the file for this instance.. Skip to EOF
    //
    fseek($fp, 0, SEEK_END);
  } else {
    // seek to last known position to skip past already handled log entries
    //
    fseek($fp, $last_position, SEEK_SET);
  }

  // check for patterns on current line
  //
  $action_taken = false;
  while (($line = fgets($fp, 4096)) != null) {
    $action = check_line($line);
    if ($action != "") {
      // TODO: log that action is taken
      //
      // take action only once for a given seek, otherwise seek silently to EOF
      //
      if (!$action_taken) {
        log_message("Apache APC/eAccelerator caused a segmentation fault.");
        log_message("Executing: " . get_action_command($action));
        system(get_action_command($action));
        log_message("Executed: " . get_action_command($action));
        email_notify();
        log_message("Email notification sent");
        $action_taken = true;
      }
    }
  }
  // record end of file position for next pass through
  //
  $last_position = ftell($fp);

  // close the file pointer
  //
  fclose($fp);
  return $last_position;
}

function log_message($message) {
  error_log(date(DATE_FORMAT) . " " . $message . "\n", 3, LOG_OUTPUT_FILENAME);
}

function check_line($line) {
  global $array_action_checks;
  // walk through each action
  //
  foreach ($array_action_checks as $action => $array_checks) {
    foreach ($array_checks as $check) {
      // walk through each check and see if it matches the current line
      //
      if (preg_match("/" . $check . "/", $line)) {
        return $action;
      }
    }
  }
  return "";
}
function get_action_command($action) {
  global $array_action_commands;
  $command = @$array_action_commands[$action];
  if ($command == null) {
    log_message("Could not retrieve command for action: $action");
    return "";
  }
  return $command;
}


function email_notify() {
  $body = "The server has encountered an APC/eAccelerator segmentation fault error.
Apache has been automatically restarted.
The log file " .  LOG_OUTPUT_FILENAME . " should have the exact time and number 
that this happened.";

  mail($email, 'Apache has been restarted', $body);
} 

And here is the logwatcher.sh shell script that is used to start it. Change the email addresses to fit your needs.

#!/bin/sh

BASE_DIR=/root/bin
SCRIPT=$BASE_DIR/logwatcher.php
PID_FILE=/var/run/logwatcher.pid
EMAIL=someone@example.com,someoneelse@example.com

# If there is an old process, kill it
kill `cat $PID_FILE`
# Make sure the file is clean
rm -f $PID_FILE

cd $BASE_DIR
nohup php $SCRIPT $EMAIL> /dev/null &
PID=$!

echo $PID > $PID_FILE

Now, all you need to do is edit your /etc/rc.local, and add a line to call the logwatcher.sh script upon booting.

Resources and Links

• Original logwatcher.php script at Bryght's repository.