Nagios Notifications

The notification-based Nagios integration is ideal if you've configured custom filters for Nagios notifications.

Nagios provides monitoring for systems, applications, services, and business processes in your IT infrastructure. Install the notification-based Nagios integration if you've configured custom filters for Nagios notifications that are working well for your needs. BigPanda will correlate problems detected in Nagios into actionable incidents in BigPanda so you can understand and respond faster to critical issues in your infrastructure.

Nagios sends notifications to a BigPanda Contact, which is configured during the installation process. BigPanda then processes and correlates the alerts from Nagios to create incidents in BigPanda. Alert data in BigPanda is updated only when another notification is received, which is based on the notification interval defined for the service.

The BigPanda agent also parses these files:

  • The status.dat file—performs a full sync to send cleared events for open notifications in BigPanda. This sync ensures that BigPanda can recover from connectivity lapses. You can customize the interval or disable the full sync.

  • The objects.cache file—allows BigPanda to enrich alerts with host-groups and service-groups as well as custom information. For custom enrichments, configure custom tags or contact BigPanda support.

When an alert closes in Nagios, the alert is closed in BigPanda.

Scheduling Downtime for Maintenance

You can schedule downtime in Nagios for a maintenance window. During a scheduled downtime, Nagios suppresses normal notifications for the host or service and sends a start event for the downtime.

After receiving the start event for a scheduled downtime, BigPanda handles subsequent alerts from objects in maintenance as follows:

  • Does not show new incidents in the incident feed.

  • Correlates alerts to existing incidents, if applicable, and marks the alerts in Maintenance status (indicated in grey).

  • Does not send AutoShare notifications for incidents where all associated alerts are in maintenance.

After receiving the end event for the scheduled downtime, BigPanda resumes normal processing for subsequent alerts from the objects.

📘

Which alerts are in Maintenance

BigPanda handles scheduled downtime according to the options you select when configuring the downtime. For example, if you configure a host to be in maintenance but not the services on that host, then alerts for those services do not change to maintenance status in BigPanda.

Key Features

  • Integrates BigPanda with Nagios by using notifications.
    If you don't use Nagios notifications, use the log-based Nagios Integration. For more information about the differences, see Nagios.

  • Processes only the Nagios alerts that generate notifications and correlates them into high-level incidents.

  • Provides insights into the infrastructure that is most important to you.

  • Allows you to independently configure custom tags to send to BigPanda.

System Requirements

The Nagios integration supports 64-bit servers. If your Nagios implementation is running on a 32-bit server, contact BigPanda Support and request a product change.

Nagios Notifications Data Model

BigPanda normalizes alert data from Nagios into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the deduplication and correlation processes.

Standard Tags

TagDescriptionAttributes
hostName of the monitored host.Primary Property.
checkName of the service being checked on the host.Secondary Property.
hostgroupsList of groups where the host is configured as a member. To enable this tag, provide access to the objects.cache file, then contact BigPanda support and request a product change.
servicegroupsList of groups where the service is configured as a member. To enable this tag, provide access to the objects.cache file, then contact BigPanda support and request a product change.
acknowledgedIndicator of whether the alert has been acknowledged in Nagios.
flappingIndicator of whether the monitor is flapping.
maintenanceIndicator of whether the monitor is scheduled for downtime.

Custom Tags

You can independently configure the Nagios Notifications integration to send custom Nagios object variables to BigPanda. These custom tags become available in BigPanda along with the standard tag data to help users resolve problems.

To send custom tags to BigPanda:

  1. In your Nagios configuration files, find the name and object type for the custom variable.

For example, open the Nagios file /usr/local/nagios/etc/objects/localhost.cfg.

In the following example, a custom variable for service object is named _test_service_tag.
Note: Custom variable names must begin with an underscore (_).

define service{
        use                             local-service         
        host_name                       devbox
        service_description             Swap Usage
        check_command                   check_local_swap!20!10
        _test_service_tag               Test-Service-Tag       ; my custom var
        }
  1. Edit the BigPanda contact file bigpanda-contact.cfg.
    The default location for the file is: /etc/bigpanda/bigpanda-contact.cfg.

  2. Add the argument name and custom variable to the BigPanda notification command.

To see the custom tag in BigPanda, argument names must begin with _SERVICE for service-based notifications or _HOST for host-based notifications.

For example:
For a service with a custom variable named _TEST_SERVICE_TAG, add: _SERVICETEST_SERVICE_TAG="$_SERVICETEST_SERVICE_TAG$"

For a host with a custom variable named custom_var, add: _HOSTTEST_SERVICE_TAG="$_HOSTTEST_SERVICE_TAG$"

define command {
       command_name     notify-service-by-bigpanda
       command_line     /usr/bin/bigpanda-notification HOSTOUTPUT="$HOSTOUTPUT$" HOSTSTATE="$HOSTSTATE$" 
HOSTNAME="$HOSTNAME$" HOSTGROUPNAMES="$HOSTGROUPNAMES$" LASTSERVICESTATECHANGE="$LASTSERVICECHECK$" LASTHOSTSTATECHANGE="$LASTHOSTCHECK$" LONGSERVICEOUTPUT="$LONGSERVICEOUTPUT$" 
LONGHOSTOUTPUT="$LONGHOSTOUTPUT$" NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" SERVICEOUTPUT="$SERVICEOUTPUT$" 
SERVICEDESC="$SERVICEDESC$" SERVICESTATE="$SERVICESTATE$" SERVICEGROUPNAMES="$SERVICEGROUPNAMES$" 
_SERVICETEST_SERVICE_TAG="$_TEST_SERVICE_TAG$"
}

📘

Tag names may differ from custom variable names

BigPanda automatically normalizes the custom variable names to meet the tag naming requirements, as necessary. For example, if you send a custom variable with the name _Host.name, BigPanda normalizes the tag name to host_name.

Install the Integration

Administrators can install the integration by following the on-screen instructions in BigPanda.

Customize Nagios Notifications

The BigPanda agent performs a full sync of all Nagios events at regular intervals to ensure that all cleared events are sent. You can configure the query interval or disable the full sync by editing the config file for the agent. You can also change the directories for event files and the dump folder.

Configure the Query Interval

You may want to configure the full sync to reduce the load on your Nagios server, depending on the average volume of events Nagios generates.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf.

  2. Locate the settings for the Nagios full sync, named nagios/statusdat.

  3. Change the value of the refresh_interval to your preferred interval between full syncs, in seconds.
    The default value is 3600 (1 hour).

Disable the Full Sync

🚧

The full sync ensures that BigPanda can recover from connectivity lapses. Disable this feature only if you need to reduce the load on your Nagios server.

  1. Open the config file, which is located at /etc/bigpanda/bigpanda.conf

  2. Locate the settings for the Nagios full sync, named nagios/statusdat.

  3. Change the value of "enabled" to false.

{
    "endpoint": "/data/events/alerts",
    "name": "nagios/statusdat",
    "enabled": false,
    "app_key": "092ut0jwoiqp3t9iwp09843qt",
    "action": "POST",
    "config": {
        "status_dat": {
        "filename": "/usr/local/nagios/var/status.dat",
        "ok_only": true,
        "refresh_interval": 3600
        }
    }
}

Change the Directories for Event Files and the Dump Folder

Event files contain the alert payload that is sent to BigPanda. The integration temporarily persists this information to disk to ensure delivery if the communication channel is disrupted. Event files are also copied to the dump folder, which is used for debugging purposes and is not processed by the agent. You may want to change these locations; for example, to use a disk with more storage.

  1. Ensure your agent is upgraded to version 4.1.0-2 +. You can check the version in the agent configuration file: /etc/bigpanda/bigpanda.conf. If your version is older than the required version, upgrade to the latest version of the BigPanda agent.

  2. Open the bigpanda-contact config file, which is located at /etc/bigpanda/bigpanda-contact.cfg.

  3. In the BigPanda notification command, add any of the following flags to the command_line to change the destination directory settings.

FlagDescription
-queue-dirChanges the destination (write) directory for event files (default: /var/lib/bigpanda/queue). You must also change the source (read) directory in the BigPanda agent config file (see step 4).
-dump-queue-dirChanges the location of the dump folder (default: /var/lib/bigpanda/queue_dump).
-create-dirsSpecifies whether to create the destination directories if they doesn't exist (default: true).
define command {
    command_name     notify-service-by-bigpanda
    command_line     /usr/bin/bigpanda-notification -queue-dir "/var/lib/MyDirectory/queue" -dump-queue-dir "/var/lib/MyDirectory/queue_dump" -create-dirs "false"
HOSTOUTPUT="$HOSTOUTPUT$" HOSTSTATE="$HOSTSTATE$" HOSTNAME="$HOSTNAME$" HOSTGROUPNAMES="$HOSTGROUPNAMES$" LASTSERVICESTATECHANGE="$LASTSERVICESTATECHANGE$" LASTHOSTSTATECHANGE="$LASTHOSTSTATECHANGE$" LONGSERVICEOUTPUT="$LONGSERVICEOUTPUT$" LONGHOSTOUTPUT="$LONGHOSTOUTPUT$" NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" SERVICEOUTPUT="$SERVICEOUTPUT$" SERVICEDESC="$SERVICEDESC$" SERVICESTATE="$SERVICESTATE$" SERVICEGROUPNAMES="$SERVICEGROUPNAMES$"
}

define command {
    command_name     notify-host-by-bigpanda
    command_line     /usr/bin/bigpanda-notification -queue-dir "/var/lib/MyDirectory/queue" -dump-queue-dir "/var/lib/MyDirectory/queue_dump" -create-dirs "false"
HOSTOUTPUT="$HOSTOUTPUT$" HOSTSTATE="$HOSTSTATE$" HOSTNAME="$HOSTNAME$" HOSTGROUPNAMES="$HOSTGROUPNAMES$" LASTSERVICESTATECHANGE="$LASTSERVICESTATECHANGE$" LASTHOSTSTATECHANGE="$LASTHOSTSTATECHANGE$" LONGSERVICEOUTPUT="$LONGSERVICEOUTPUT$" LONGHOSTOUTPUT="$LONGHOSTOUTPUT$" NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" SERVICEOUTPUT="$SERVICEOUTPUT$" SERVICEDESC="$SERVICEDESC$" SERVICESTATE="$SERVICESTATE$" SERVICEGROUPNAMES="$SERVICEGROUPNAMES$"
}
  1. If you changed the destination (write) directory for event files, change the source (read) directory:
  • Open the BigPanda agent config file, which is located at etc/bigpanda/bigpanda.conf.

  • Locate the settings for Nagios notifications, named nagios/notifications.

  • Change the value of the queue_directory setting to the same directory that you specified in Step 3.

{
    "name": "nagios/notifications",
    "enabled": false,
    "action": "POST",
    "app_key": "092ut0jwoiqp3t9iwp09843qt",
    "endpoint": "/data/integrations/nagios/notifications?app_key=092ut0jwoiqp3t9iwp09843qt",
    "config": {
        "queue_directory": "/var/lib/MyDirectory/queue",
        }
}

Uninstall the Integration

Deleting an integration requires changes to both the integrated system and BigPanda. You must uninstall the integration on the integrated system and then delete the integration from BigPanda.

🚧

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Stop Sending Data to BigPanda

Manually resolve any open alerts sent from the integration to remove the associated incidents from your incident feed. These incidents will not automatically resolve without an ok status from the original sending integration.

Remove BigPanda from Nagios Configuration Files

  1. Open the Nagios main configuration file.
    The default location is: /usr/local/nagios/etc/nagios.cfg.

  2. Remove all references to the BigPanda contact and the bigpanda-contact.cfg file.

  3. Restart the Nagios service.

Uninstall the Agent

If the agent supports only the Nagios integration, you can uninstall the agent from the server.

🚧

Do not uninstall the BigPanda agent if it is supporting other BigPanda integrations. In this case, remove the Nagios integration from the agent.

Remove the Nagios Integration from the Agent

If the agent is supporting multiple integrations, you can remove only the specific integration without disabling the other BigPanda integrations that the agent supports.

  1. Stop the BigPanda agent by running the following command.
sudo service bigpanda stop
  1. Remove the integration from the agent configuration file by running the following command.

❗️

Replace the app key with the corresponding value in BigPanda

sudo bigpanda-config -r <INTEGRATION APP KEY>
  1. Start the BigPanda agent by running the following command.
sudo service bigpanda start

Delete the Integration from BigPanda

  1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
  2. In the integration details on the right, click Delete Integration. A support message opens, pre-populated with a request to delete the selected integration.
  3. Press Enter to send the request.
  4. The BigPanda support team will remove the integration from the UI.

To stop sending Nagios events to BigPanda, you can uninstall the agent from the server or remove the Nagios integration from the agent.