Scheduled Downtime

Introduction

Nagios allows you to schedule periods of planned downtime for hosts and service that you're monitoring. This is useful in the event that you actually know you're going to be taking a server down for an upgrade, etc. When a host a service is in a period of scheduled downtime, notifications for that host or service will be suppressed.

Downtime File

Scheduled host and service downtime is stored in the file you specify by the downtime_file directive in your main configuration file.

Downtime Retention

Scheduled host and service downtime is automatically preserved across program restarts. When Nagios starts up, it will scan the downtime file, delete any old or invalid entries, and schedule downtime for all valid host and service entries.

Scheduling Downtime

You can schedule downtime for hosts and service through the extinfo CGI (either when viewing host or service information). Click in the "Schedule downtime for this host/service" link to actually schedule the downtime.

Once you schedule downtime for a host or service, Nagios will add a comment to that host/service indicating that it is scheduled for downtime during the period of time you indicated. When that period of downtime passes, Nagios will automatically delete the comment that it added. Nice, huh?

Types of Scheduled Downtime

There are two types of scheduled downtime - "fixed" and "flexible". When you schedule downtime for a host or service through the web interface you'll be asked if the downtime is fixed or not. Here's an explanation of how "fixed" and "flexible" downtime differs:

"Fixed" downtime starts and stops at the exact start and end times that you specify when you schedule it. Okay, that was easy enough...

"Flexible" downtime is intended for times when you know that a host or service is going to be down for X minutes (or hours), but you don't know exactly when that'll start. When you schedule flexible downtime, Nagios will start the scheduled downtime sometime between the start and end times you specified. The downtime will last for as long as the duration you specified when you scheduled the downtime. This assumes that the host or service for which you scheduled flexible downtime either goes down (or becomes unreachable) or goes into a non-OK state sometime between the start and end times you specified. The time at which a host or service transitions to a problem state determines the time at which Nagios actually starts the downtime. The downtime will then last for the duration you specified, even if the host or service recovers before the downtime expires. This is done for a very good reason. As we all know, you can think you've got a problem fixed (and restart a server) ten times before it actually works right. Smart, eh?

How Scheduled Downtime Affects Notifications

When a host or service is in a period of scheduled downtime, Nagios will not allow notifications to be sent out for the host or service. suppression of notifications is accomplished by adding an additional filter to the notification logic. You will not see an icon in the CGIs indicating that notifications for that host/service are disabled. When the scheduled downtime has passed, Nagios will allow notifications to be sent out for the host or service as it normally would.

Overlapping Scheduled Downtime

I like to refer to this as the "Oh crap, its not working" syndrome. You know what I'm talking about. You take a server down to perform a "routine" hardware upgrade, only to later realize that the OS drivers aren't working, the RAID array blew up, or the drive imaging failed and left your original disks useless to the world. Moral of the story is that any routine work on a server is quite likely to take three or four times as long as you had originally planned...

Let's take the following scenario:

  1. You schedule downtime for host A from 7:30pm-9:30pm on a Monday
  2. You bring the server down about 7:45pm Monday evening to start a hard drive upgrade
  3. After wasting an hour and a half battling with SCSI errors and driver incompatabilities, you finally get the machine to boot up
  4. At 9:15 you realize that one of your partitions is either hosed or doesn't seem to exist anywhere on the drive
  5. Knowing you're in for a long night, you go back and schedule additional downtime for host A from 9:20pm Monday evening to 1:30am Tuesday Morning.

If you schedule overlapping periods of downtime for a host or service (in this case the periods were 7:40pm-9:30pm and 9:20pm-1:30am), Nagios will wait until the last period of scheduled downtime is over before it allows notifications to be sent out for that host or service. In this example notifications would be suppressed for host A until 1:30am Tuesday morning.