Introduction

Nagios has the ability to distinguish between "normal" services and "volatile" services. The is_volatile option in each service definition allows you to specify whether a specific service is volatile or not. For most people, the majority of all monitored services will be non-volatile (i.e. "normal"). However, volatile services can be very useful when used properly...

What Are They Useful For?

Volatile services are useful for monitoring...

• things that automatically reset themselves to an "OK" state each time they are checked
• events such as security alerts which require attention every time there is a problem (and not just the first time)

What's So Special About Volatile Services?

Volatile services differ from "normal" services in three important ways. Each time they are checked when they are in a hard non-OK state, and the check returns a non-OK state (i.e. no state change has occurred)...

• the non-OK service state is logged
• contacts are notified about the problem (if that's what should be done)
• the event handler for the service is run (if one has been defined)

These events normally only occur for services when they are in a non-OK state and a hard state change has just occurred. In other words, they only happen the first time that a service goes into a non-OK state. If future checks of the service result in the same non-OK state, no hard state change occurs and none of the events mentioned take place again.

The Power Of Two

If you combine the features of volatile services and passive service checks, you can do some very useful things. Examples of this include handling SNMP traps, security alerts, etc.

How about an example... Let's say you're running PortSentry to detect port scans on your machine and automatically firewall potential intruders. If you want to let Nagios know about port scans, you could do the following..

In Nagios:

• Configure a service called Port Scans and associate it with the host that PortSentry is running on.
• Set the max_check_attempts option in the service definition to 1. This will tell Nagios to immediate force the service into a hard state when a non-OK state is reported.
• Either set the active_checks_enabled option to 0 or set the check_time option in the service definition to a timeperiod that contains no valid time ranges. Doing either of these will prevent Nagios from ever actively checking the service. Even though the service check will get scheduled, it will never actually be checked.

In PortSentry:

• Edit your PortSentry configuration file (portsentry.conf), define a command for the KILL_RUN_CMD directive as follows:
  
  KILL_RUN_CMD="/usr/local/Nagios/libexec/eventhandlers/submit_check_result <host_name> 'Port Scans' 2 'Port scan from host $TARGET$ on port $PORT$. Host has been firewalled.'"
  
  Make sure to replace <host_name> with the short name of the host that the service is associated with.

Create a shell script in the /usr/local/nagios/libexec/eventhandlers directory named submit_check_result. The contents of the shell script should be something similiar to the following...

	#!/bin/sh
 
	# Write a command to the Nagios command file to cause
	# it to process a service check result
 
	echocmd="/bin/echo"
 
	CommandFile="/usr/local/nagios/var/rw/nagios.cmd"
 
	# get the current date/time in seconds since UNIX epoch
	datetime=`date +%s`
 
	# create the command line to add to the command file
	cmdline="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$1;$2;$3;$4"
 
	# append the command to the end of the command file
	`$echocmd $cmdline >> $CommandFile`

Note that if you are running PortSentry as root, you will have to make additions to the script to reset file ownership and permissions so that Nagios and the CGIs can read/modify the command file. Details on permissions/ownership of the command file can be found here.

So what happens when PortSentry detects a port scan on the machine?

• It blocks the host (this is a function of the PortSentry software)
• It executes the submit_check_result shell script to send the security alert info to Nagios
• Nagios reads the command file, recognized the port scan entry as a passive service check
• Nagios processes the results of the service by logging the CRITICAL state, sending notifications to contacts (if configured to do so), and executes the event handler for the Port Scans service (if one is defined)