survivor: Confusing Error Messages Explained
About Confusing Error Messages

Although error messages are generally written to be transparent and understandable in context, some, by virtue of the underlying code design, may seem a little confusing.

Parsing Configuration Files

Error

When parsing the configuration files, 
   *lex* Unexpected token at line 79 (state 7): "#commented"

Sample

   1024 *lex* Unexpected token at line 79 (state 7): "#commented"
   1024 *lex* Unexpected token at line 79 (state 7): "out"
   1024 *lex* Unexpected token at line 79 (state 7): "block"
   1024 || 3 errors encountered while parsing /home/symon/sample/config/check.cf
   sc: WARNING: Configuration parse failed

Explanation

The configuration parser, due to the complexities of the underlying parsing mechanism, requires all comments to be terminated with a newline. A comment at the end of the file might not have a newline after it, causing the parser to fail.

Simply add a newline after the comment to fix this problem.

Command Interface (sc)

Error

   State::lock_type_state failed to open .../lock

Sample

   % sc -i instance clstate oncall
   [oncall]
   sc: WARNING: State::lock_calllist_state failed to open
   /var/instance/state/calllist/oncall/lock
   sc: WARNING: State::lock_calllist_state failed to open
   /var/instance/state/calllist/oncall/lock
   sc: WARNING: State::lock_calllist_state failed to open
   /var/instance/state/calllist/oncall/lock
   -> somebody@site.org is now on call

Explanation

When configuration files are updated, it is the responsibility of the scheduler to update the state directories in accordance with the new configuration. If other utilities, such as the command interface, are run with the new configuration before the scheduler is told of the update, they may try to access state files or directories that do not yet exist.

In the case of the sample above, the oncall call list was added to calllist.cf. Before the scheduler was told of the update, the command interface was run to get the state of the new call list. Since the scheduler has not made the call list state directories consistent, the directory /var/instance/state/calllist/oncall does not yet exist, and so its lock file cannot be opened.

Scheduler (ss)

Error

Lots and lots of spewage in /var/log/survivor-instance.log.

Sample

   ss: WARNING: state_consistency failed to create service state directory
   ss: WARNING: Unable to create host history directory
   ss: WARNING: CheckState::lastcheck failed to open
   ss: WARNING: CheckState::write_results failed to reset permissions on

Explanation

Most likely, the user running the scheduler is not INSTUSER or is not a member of INSTGROUP, and so cannot properly access and update the files. See the building instructions for more details.

Error

On certain Linux platforms, SIGHUP sometimes causes the scheduler to exit, complaining the scheduler is restarting too frequently.

Sample

   
   ss: WARNING: Keepalive process is restarting the scheduler too frequently.
   ss: WARNING: There may be a configuration file error or some other problem.
   ss: WARNING: Keepalive process is exiting as a precaution.

Explanation

This is a bug in the system's implementation of sigwait(3). Instead of returning the proper signal sent to the appropriate thread (in this case, SIGHUP, or 1), the non-existant signal 0 is sent. Since the scheduler cannot tell what signal 0 really means, the scheduler exits. When run under keepalive (ss -k), the keepalive daemon will restart the scheduler, effectively simulating a SIGHUP. However, if this is done too often (more than once per minute), the keepalive exits, assuming there is a bigger problem.

Update: This appears to have been fixed. libc version 2.3.2 is known to work properly.

Error

   Failed to queue check 'service' (check may already be scheduled)

Sample

   ss: WARNING: Failed to queue check 'syslogd' (check may already be
   scheduled)

Explanation

Every minute, the check scheduler attempts to schedule any checks that are due to be executed. If a particular check takes more than a minute to execute, the check scheduler will attempt to schedule that check again.

To prevent the same process from being queued multiple times (and thus causing backlogs or concurrency problems), the scheduler will produce the above error message if a previously scheduled check with the same name has not yet completed.

Error

   CheckState::lastcheck failed to open .../lastcheck

Sample

   ss: WARNING: CheckState::lastcheck failed to open
   /var/instance/state/host/hostname/service/lastcheck (No such file
   or directory)

Explanation

When a new check or host is added, there is no way to guarantee that the check scheduler will notice before the alert scheduler. When both hostname and service are known to be valid, this warning may still be generated. This is due to the alert scheduler noticing the new check or host before the check scheduler, and so there is no lastcheck state to be examined.

This warning should not continue after a minute or two, by which time the new check or host will have been queued by the check scheduler. Note that setting an alert shift time in schedule.cf will not eliminate this message, as the alert shift time only adjusts the period during which alerts are generated, and not the relative times compared to the check scheduler within that period.

Update: Due to changes within the scheduler, this error should no longer occur.

Web Interface (sw)

Error

(blank output) or 
   HTTP/1.1 500 Internal Server Error

Sample

   HTTP/1.1 500 Internal Server Error

Explanation

Some dynamic libraries, including the OpenSSL libraries such as libssl or libcrypto, are not in the runtime LD_LIBRARY_PATH of the web interface.

Ordinarily, the paths to these libraries are encoded into the executable when the package is built. If, however, these libraries are moved or removed, when the program is executed the runtime linker will fail to resolve symbol definitions and the program will not run.

It may be possible to replicate this failure by manually running the program: 

 % unsetenv LD_LIBRARY_PATH
 % ./sw
 % ld.so.1: ./sw: fatal: libssl.so.0.9.6: open failed: No such file or directory
 Killed

To fix this problem, replace the libraries or rebuild the package with the new locations included in Makefile.inc.

$Date: 2006/11/20 02:54:21 $
$Revision: 0.13 $
 keywords