Nagios Notifications

Nagios provides monitoring for systems, applications, services, and business processes in your IT infrastructure. Install the notification-based Nagios integration if you've configured custom filters for Nagios notifications that are working well for your needs. BigPanda will correlate problems detected in Nagios into actionable incidents in BigPanda so you can understand and respond faster to critical issues your infrastructure.

Key Features

  • Integrates BigPanda with Nagios by using notifications.
    If you don't use Nagios notifications, use the log-based Nagios Integration. For more information about the differences, see Nagios.

  • Processes only the Nagios alerts that generate notifications and correlates them into high-level incidents.

  • Provides insights into the infrastructure that is most important to you.

  • Allows you to independently configure custom tags to send to BigPanda.

How It Works

Nagios sends notifications to a BigPanda Contact, which is configured during the installation process. BigPanda then processes and correlates the alerts from Nagios to create incidents in BigPanda. Alert data in BigPanda is updated only when another notification is received, which is based on the notification interval defined for the service.

The BigPanda agent also parses these files:

  • The status.dat file—performs a full sync to send cleared events for open notifications in BigPanda. This sync ensures that BigPanda can recover from connectivity lapses. You can customize the interval or disable the full sync.

  • The objects.cache file—allows BigPanda to enrich alerts with host-groups and service-groups as well as custom information. For custom enrichments, configure custom tags or contact BigPanda support.

When an alert closes in Nagios, the alert is closed in BigPanda.

Scheduling Downtime for Maintenance

You can schedule downtime in Nagios for a maintenance window. During a scheduled downtime, Nagios suppresses normal notifications for the host or service and sends a start event for the downtime.

After receiving the start event for a scheduled downtime, BigPanda handles subsequent alerts from objects in maintenance as follows:

  • Does not show new incidents in the incident feed.

  • Correlates alerts to existing incidents, if applicable, and marks the alerts in Maintenance status (indicated in grey).

  • Does not send AutoShare notifications for incidents where all associated alerts are in maintenance.

After receiving the end event for the scheduled downtime, BigPanda resumes normal processing for subsequent alerts from the objects.

📘

Which alerts are in Maintenance

BigPanda handles scheduled downtime according to the options you select when configuring the downtime. For example, if you configure a host to be in maintenance but not the services on that host, then alerts for those services do not change to maintenance status in BigPanda.

System Requirements

The Nagios integration supports 64-bit servers. If your Nagios implementation is running on a 32-bit server, contact BigPanda support.

Installing the Integration

Administrators can install the integration by following the on-screen instructions in BigPanda. For more information, see Installing an Integration.

Nagios Notifications Data Model

BigPanda normalizes alert data from Nagios into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the correlation process

Standard Tags

Tag

Description

Attributes

host

Name of the monitored host.

Primary Property.

check

Name of the service being checked on the host.

Secondary Property.

hostgroups

List of groups where the host is configured as a member. To enable this tag, you must open a request with BigPanda support and provide access to the objects.cache file.

servicegroups

List of groups where the service is configured as a member. To enable this tag, you must open a request with BigPanda support and provide access to the objects.cache file.

acknowledged

Indicator of whether the alert has been acknowledged in Nagios.

flapping

Indicator of whether the monitor is flapping.

maintenance

Indicator of whether the monitor is scheduled for downtime.

Custom Tags

You can independently configure the Nagios Notifications integration to send custom Nagios object variables to BigPanda. These custom tags become available in BigPanda along with the standard tag data to help users resolve problems.

To send custom tags to BigPanda:

  1. In your Nagios configuration files, find the name and object type for the custom variable.

For example, open the Nagios file /usr/local/nagios/etc/objects/localhost.cfg.

In the following example, a custom variable for service object is named _test_service_tag.
Note: Custom variable names must begin with an underscore (_).

define service{
        use                             local-service         
        host_name                       devbox
        service_description             Swap Usage
        check_command                   check_local_swap!20!10
        _test_service_tag               Test-Service-Tag       ; my custom var
        }
  1. Edit the BigPanda contact file bigpanda-contact.cfg.
    The default location for the file is: /etc/bigpanda/bigpanda-contact.cfg.

  2. Add the argument name and custom variable to the BigPanda notification command.

To see the custom tag in BigPanda, argument names must begin with _SERVICE for service-based notifications or _HOST for host-based notifications.

For example:
For a service with a custom variable named _TEST_SERVICE_TAG, add: _SERVICETEST_SERVICE_TAG="$_SERVICETEST_SERVICE_TAG$"

For a host with a custom variable named custom_var, add: _HOSTTEST_SERVICE_TAG="$_HOSTTEST_SERVICE_TAG$"

define command {
       command_name     notify-service-by-bigpanda
       command_line     /usr/bin/bigpanda-notification HOSTOUTPUT="$HOSTOUTPUT$" HOSTSTATE="$HOSTSTATE$" 
HOSTNAME="$HOSTNAME$" HOSTGROUPNAMES="$HOSTGROUPNAMES$" LASTSERVICESTATECHANGE="$LASTSERVICESTATECHANGE$" 
LASTHOSTSTATECHANGE="$LASTHOSTSTATECHANGE$" LONGSERVICEOUTPUT="$LONGSERVICEOUTPUT$" 
LONGHOSTOUTPUT="$LONGHOSTOUTPUT$" NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" SERVICEOUTPUT="$SERVICEOUTPUT$" 
SERVICEDESC="$SERVICEDESC$" SERVICESTATE="$SERVICESTATE$" SERVICEGROUPNAMES="$SERVICEGROUPNAMES$" 
_SERVICETEST_SERVICE_TAG="$_TEST_SERVICE_TAG$"
}

📘

Tag names may differ from custom variable names

BigPanda automatically normalizes the custom variable names to meet the tag naming requirements, as necessary. For example, if you send a custom variable with the name _Host.name, BigPanda normalizes the tag name to host_name.

Customizing Nagios Notifications

The BigPanda agent performs a full sync of all Nagios events at regular intervals to ensure that all cleared events are sent. You can configure the query interval or disable the full sync by editing the config file for the agent. You can also change the directories for event files and the dump folder.

Prerequisites

Obtain access to the server where the BigPanda agent is installed.

Configuring the Query Interval

You may want to configure the full sync to reduce the load on your Nagios server, depending on the average volume of events Nagios generates.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.

  2. Locate the settings for the Nagios full sync, named nagios/statusdat.

  3. Change the value of the refresh_interval to your preferred interval between full syncs, in seconds.
    The default value is 3600 (1 hour).

Disabling the Full Sync

🚧

The full sync ensures that BigPanda can recover from connectivity lapses. Disable this feature only if you need to reduce the load on your Nagios server.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf

  2. Locate the settings for the Nagios full sync, named nagios/statusdat.

  3. Change the value of "enabled" to false.

{
    "endpoint": "/data/events/alerts",
    "name": "nagios/statusdat",
    "enabled": false,
    "app_key": "092ut0jwoiqp3t9iwp09843qt",
    "action": "POST",
    "config": {
        "status_dat": {
        "filename": "/usr/local/nagios/var/status.dat",
        "ok_only": true,
        "refresh_interval": 3600
        }
    }
}

Changing the Directories for Event Files and the Dump Folder

Event files contain the alert payload that is sent to BigPanda. The integration temporarily persists this information to disk to ensure delivery if the communication channel is disrupted. Event files are also copied to the dump folder, which is used for debugging purposes and is not processed by the agent. You may want to change these locations; for example, to use a disk with more storage.

  1. Ensure your agent is upgraded to version 4.1.0-2 +. You can check the version in the agent configuration file: /etc/bigpanda/bigpanda.conf. If your version is older than the required version, upgrade to the latest version of the BigPanda agent.

  2. Open the bigpanda-contact config file, which is located at /etc/bigpanda/bigpanda-contact.cfg.

  3. In the BigPanda notification command, add any of the following flags to the command_line to change the destination directory settings.

Flag

Description

-queue-dir

Changes the destination (write) directory for event files (default: /var/lib/bigpanda/queue). You must also change the source (read) directory in the BigPanda agent config file (see step 4).

-dump-queue-dir

Changes the location of the dump folder (default: /var/lib/bigpanda/queue_dump).

-create-dirs

Specifies whether to create the destination directories if they doesn't exist (default: true).

define command {
    command_name     notify-service-by-bigpanda
    command_line     /usr/bin/bigpanda-notification -queue-dir "/var/lib/MyDirectory/queue" -dump-queue-dir "/var/lib/MyDirectory/queue_dump" -create-dirs "false"
HOSTOUTPUT="$HOSTOUTPUT$" HOSTSTATE="$HOSTSTATE$" HOSTNAME="$HOSTNAME$" HOSTGROUPNAMES="$HOSTGROUPNAMES$" LASTSERVICESTATECHANGE="$LASTSERVICESTATECHANGE$" LASTHOSTSTATECHANGE="$LASTHOSTSTATECHANGE$" LONGSERVICEOUTPUT="$LONGSERVICEOUTPUT$" LONGHOSTOUTPUT="$LONGHOSTOUTPUT$" NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" SERVICEOUTPUT="$SERVICEOUTPUT$" SERVICEDESC="$SERVICEDESC$" SERVICESTATE="$SERVICESTATE$" SERVICEGROUPNAMES="$SERVICEGROUPNAMES$"
}

define command {
    command_name     notify-host-by-bigpanda
    command_line     /usr/bin/bigpanda-notification -queue-dir "/var/lib/MyDirectory/queue" -dump-queue-dir "/var/lib/MyDirectory/queue_dump" -create-dirs "false"
HOSTOUTPUT="$HOSTOUTPUT$" HOSTSTATE="$HOSTSTATE$" HOSTNAME="$HOSTNAME$" HOSTGROUPNAMES="$HOSTGROUPNAMES$" LASTSERVICESTATECHANGE="$LASTSERVICESTATECHANGE$" LASTHOSTSTATECHANGE="$LASTHOSTSTATECHANGE$" LONGSERVICEOUTPUT="$LONGSERVICEOUTPUT$" LONGHOSTOUTPUT="$LONGHOSTOUTPUT$" NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" SERVICEOUTPUT="$SERVICEOUTPUT$" SERVICEDESC="$SERVICEDESC$" SERVICESTATE="$SERVICESTATE$" SERVICEGROUPNAMES="$SERVICEGROUPNAMES$"
}
  1. If you changed the destination (write) directory for event files, change the source (read) directory:
  • Open the BigPanda agent config file, which is located at etc/bigpanda/bigpanda.conf.

  • Locate the settings for Nagios notifications, named nagios/notifications.

  • Change the value of the queue_directory setting to the same directory that you specified in Step 3.

{
    "name": "nagios/notifications",
    "enabled": false,
    "action": "POST",
    "app_key": "092ut0jwoiqp3t9iwp09843qt",
    "endpoint": "/data/integrations/nagios/notifications?app_key=092ut0jwoiqp3t9iwp09843qt",
    "config": {
        "queue_directory": "/var/lib/MyDirectory/queue",
        }
}

Uninstalling Nagios Notifications

To stop sending Nagios events to BigPanda, remove references to BigPanda from the Nagios configuration files. Then, uninstall the agent from the server or remove the Nagios integration from the agent.

Prerequisites

  • Manually resolve any open incidents related to the integration to remove them from your incident feed. Optionally, remove any additional data in BigPanda. For example, modify or remove any Environments or AutoShare rules that reference the integration.

  • Obtain access to the server where the BigPanda agent is installed.

  • Determine if the agent on the server supports only the Nagios integration or supports multiple BigPanda integrations.

  • Determine the OS on the server.

  • Obtain access to the Nagios configuration files.

Removing BigPanda from Nagios Configuration Files

  1. Open the Nagios main configuration file.
    The default location is: /usr/local/nagios/etc/nagios.cfg.

  2. Remove all references to the BigPanda contact and the bigpanda-contact.cfg file.

  3. Restart the Nagios service.

Uninstalling The Agent

If the agent supports only the Nagios integration, you can uninstall the agent from the server.

🚧

Do not uninstall the BigPanda agent if it is supporting other BigPanda integrations. In this case, remove the Nagios integration from the agent.

Removing the Nagios Integration from the Agent

If the agent is supporting multiple integrations, you can remove only the specific integration without disabling the other BigPanda integrations that the agent supports.

  1. Stop the BigPanda agent by running one of the following commands.
  • For RHEL 5 and 7, CentOS 5 and 7, Debian, and Ubuntu:

sudo service bigpanda stop

  • For RHEL 6 and CentOS 6:

sudo initctl stop bigpanda

  1. Remove the integration from the agent configuration file by running the following command.

❗️

Replace the app key with the corresponding value in BigPanda

sudo bigpanda-config -r <INTEGRATION APP KEY>

  1. Start the BigPanda agent by running one of the following commands.
  • For RHEL 5 and 7, CentOS 5 and 7, Debian, and Ubuntu:

sudo service bigpanda start

  • For RHEL 6 and CentOS 6:

sudo initctl start bigpanda

Post-Requisites

Delete the Integration in BigPanda to remove the Nagios integration from your UI.