Datadog

Datadog provides monitoring alerts for custom metrics across systems, applications, and services. Install this integration to correlate your Datadog alerts into actionable incidents in BigPanda so you can find and fix problems in your infrastructure quicker and easier.

Supported Versions

Type

Authentication Type

SaaS Deployments

Webhook

Bearer Token

Key Features

  • Easily send all Datadog alerts to BigPanda with a native integration in Datadog
  • Correlate alerts from Datadog Monitors to help you understand and respond faster to production issues.

How It Works

The integration works by configuring the native BigPanda integration in Datadog. The Datadog alerting pipeline will then send all alerts to BigPanda. BigPanda then processes and correlates the alert data from Datadog to create and maintain up-to-date incidents in BigPanda.

Data Model

BigPanda normalizes alert data from Datadog into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the correlation process.

Data Normalization

All properties from the Datadog alert are mapped 1:1 to tags on the BigPanda alert. The below chart describes some special properties of specific Datadog alert tags.

Tag

Description

Additional Info

host

Host that triggered the alert. This tag is relevant only when a specific host is selected in the Datadog monitor definition.

Primary Property.

monitor_name

Name of the monitor in Datadog.

Alternate Primary Property (if host does not exist)

alert_metric

Metric that triggered the alert. This tag is relevant only when a specific metric is selected in the Datadog monitor definition. For example:

system.mem.total/free/used
system.disk.total/used/free
process.up
http.can_connect

Secondary Property.

event_type

Type of Datadog event. For example:

service_check
metric_alert_monitor
query_alert_monitor

Alternate Secondary Property (if alert_metric does not exist)

alert_query

Query of the triggered monitor in Datadog.

description

Description of the alert. This is the Datadog event message with markdown removed.

link

Link to the Datadog event this alert represents.

last_trigger

Last trigger time for this Datadog alert.

Parsed from the Datadog event message.

logs_link

Link to related logs in Datadog, if they exist.

Parsed from the Datadog event message.

monitor_link

Link to the actual Datadog monitor that this alert is generated by.

Parsed from the Datadog event message.

processes_link

Link to related processes in Datadog, if they exist.

Parsed from the Datadog event message.

resource_link

Link to the related resource in Datadog, if it exists.

Parsed from the Datadog event message.

tags

An array of tags on the Datadog monitor without associated keys (values only).

XXX (Datadog tags)

All tags on the Datadog monitor which are in key:value format are mapped directly corresponding tags on the BigPanda alert.

status and description tags are reserved keys on the BigPanda alert and any Datadog monitor tags with these keys will be renamed to tag.KEY

Statuses

BigPanda will automatically correlate a Datadog status into a BigPanda status that can be used to search and filter incidents.

Datadog Status

BigPanda Status

triggered, retriggered

Critical

warn

Warning

recovered

Ok

no data, re-no data`

Unknown

🚧

Unknown Status

The Unknown status is not a default status within BigPanda, but can be enabled if you wish to see no data and re-no data alerts in BigPanda. To enable the Unknown alert status in BigPanda, please contact [email protected].

Installing the Integration

In BigPanda

Administrators can install the integration by following the on-screen instructions in BigPanda. For more information, see Installing an Integration.

In Datadog

Once you've created the integration in BigPanda you can then finish setup via the BigPanda integration tile in the Datadog console.

If you do not wish to send all of your Datadog monitor alerts to BigPanda you can uncheck the Route All Monitor Events option of the BigPanda integration in Datadog and then manually add @bigpanda to the notifications for each monitor you wish to send.

Uninstalling the Integration

You must delete the BigPanda integration from Datadog to stop sending alerts to BigPanda.

Prerequisites

Manually resolve any open incidents related to the integration to remove them from your incident feed. Optionally, remove any additional data in BigPanda. For example, modify or remove any Environments or AutoShare rules that reference the integration.

Procedure

  1. In Datadog, go to the BigPanda integration tile
  2. Under the Configuration tab click Uninstall Integration

Post-Requisites

Delete the Integration in BigPanda to remove the Datadog integration from your UI.

Upgrading an Existing Datadog Integration

If you are using an existing Datadog integration the best approach is to replace it with a completely new integration using the steps above: Installing the Integration

However if you have many monitors notifications already configured you may wish to keep using your existing Webhook integration. In that case you can follow the below steps:

  1. In Datadog, go to the Integrations Settings from the sidebar
  2. Locate your existing Webhook integration sending to BigPanda and click Configure
  3. Click Edit
  4. Change the URL to the following, replacing BEARER_TOKEN and APP_KEY with your BigPanda account bearer token and Datadog integration's app key respectively:
https://inbound.bigpanda.io/datadog/alerts?access_token=BEARER_TOKEN&app_key=APP_KEY
  1. Change the payload the following:
{
   "host": "$HOSTNAME",
   "title": "$EVENT_TITLE",
   "alert_metric": "$ALERT_METRIC",
   "alert_query": "$ALERT_QUERY",
   "body": "$EVENT_MSG",
   "body_clean": "$TEXT_ONLY_MSG",
   "link": "$LINK",
   "tags": "$TAGS",
   "event_type": "$EVENT_TYPE",
   "priority": "$PRIORITY",
   "alert_status": "$ALERT_STATUS",
   "alert_transition": "$ALERT_TRANSITION",
   "last_updated": "$LAST_UPDATED",
   "date": "$DATE",
   "email": "$EMAIL",
   "datadog_org_name": "$ORG_NAME",
   "datadog_org_id": "$ORG_ID",
   "id":"$ID",
   "aggreg_key": "$AGGREG_KEY"
}
  1. Click Save