Alerts
The Alerts API allows you to easily integrate a monitoring system with BigPanda. Monitoring systems generally send out events when problems are detected and when problems have been resolved (fixed).
The API receives event payloads from a monitoring system. Incoming events sent via the REST API are processed according to the BigPanda alert correlation logic. Depending on the results of the correlation, the system then creates a new incident or updates an existing incident based on Event Aggregation rules.
If the tool you are integrating with does not support HTTP headers, you can use the access_token URL parameter for authentication. For example: <https://api.bigpanda.io/data/v2/alerts?access_token=><YOUR TOKEN>
Sending multiple alerts
BigPanda uses the timestamp to determine the latest status of an incident. If it is not included, BigPanda uses the time when the alert is received. To ensure that BigPanda accurately reflects the current status, when sending multiple alerts, you must include the timestamp for each alert or sort the alerts array by when the events occurred, in ascending order.
Sending duplicate events
When an event sent via the Alerts API is an exact duplicate of a previous event, BigPanda returns an HTTP response code of 204 No Content.
Resolving alerts
To maintain only the most relevant information in the incident feed, we recommend configuring monitoring systems to send resolving events to BigPanda when an alert is no longer active.
How Many Events to Send (Load Limitations)
Payload size
Alert payloads must be 6MB or smaller. Larger payloads will fail to process with BigPanda. We recommend reviewing your configurations to ensure that only actionable, useful information is being sent to BigPanda.
If an integration generates more than a few hundred events per hour, review the guidelines for when to send events and ensure that the integration is properly configured. Even with hundreds of thousands of monitoring metrics, status changes do not happen very often. If an integration exceeds this guideline, it may be sending duplicate or erroneous events to BigPanda.
Under maximum load, the API can support up to a few hundred events per minute.
Generic Alerts
Authentication
All BigPanda APIs require Bearer Token Authorization in the call headers.
This API uses the Org Token type of Authorization token.
BigPanda recommends adding Authentication headers only in the secure tool you use to make API calls.
Integrating Monitoring Systems
You can use the Alerts API to build a custom integration between BigPanda and your monitoring system. If the monitoring system supports custom HTTP callbacks, the easiest way to integrate it is by using a webhook. If it doesn't, you can write a custom script.
Using a Webhook to Integrate a Monitoring System
If the monitoring system supports custom HTTP callbacks, you can follow this general process to integrate it with BigPanda:
In BigPanda, create an App Key. Each integration must have an App Key in BigPanda to identify it as a unique source.
In your monitoring system, configure the webhook to send alerts to BigPanda.
Test the integration by sending a test alert.
Using a Custom Script to Integrate a Monitoring System
If the monitoring system does not support webhooks, you can write a custom script. Follow this general process to integrate it with BigPanda:
In BigPanda, create an App Key. Each integration must have an App Key in BigPanda to identify it as a unique source.
Configure your monitoring system to call your custom script.
Test the integration by sending a test alert.
To learn more about integrating tools using the BigPanda Rest API, see the instructions in BigPanda.
Best Practices
Follow these guidelines when building custom integrations with the BigPanda Alerts API.
When to Send Events
Send an event every time the status of an alert changes. For example, if a new Critical alert is opened or a Critical alert becomes a Warning. The status of an alert typically changes when a metric goes above or below a certain threshold.
Send an updated event when you need to change tag values for an alert. In some rare cases, you may want to send an event for an alert with the same status but a different value for one or more tags. For example, assume you have an alert with status of Warning and description of problem1. You can send a new Warning event with a description of problem2, and the alert status remains the same but the description is updated.
Send an OK resolution event when an alert is no longer active. BigPanda does not resolve alerts automatically. Alerts that have not been resolved remain open, and the corresponding incident also remains open and continues to appear in the incident feed.
Resolution events
Some monitoring tools don't trigger resolution events. In this case, you must manually resolve incidents to remove them from the incident feed.
What Tags to Send
The attributes included in the JSON payload become tags in BigPanda.
You can add any number of custom JSON attributes with a string, integer, or array value to the payload. Common fields include host, host, service, application, device, check, sensor, cluster, node, data center, region, and description.
When configuring the data to send through the Alerts API, send attributes that:
Drive deduplication, correlation, and UI titling. Event attributes enable BigPanda to turn noisy events into high-quality alerts and actionable incidents.
Add context about the event, including where or when the alert triggered. Additional information helps your team understand the full situation while investigating an issue.
Enable automation and categorization. Tags such as
business elementorteamhelp you build environments, autoshare rules, analytics, and workflows.Set next steps for triage and remediation. Including links to runbook wikis or investigation tools helps your team take action quicker. If a tag value starts with
http, BigPanda automatically adds a link button.
Nested objects
Nested objects are not supported when sending events to the Rest API
Incident_identifier
The incident_identifier is a unique ID used in BigPanda to recognize if two events are related to each other. By default BigPanda will use a combination of the primary and secondary properties to create a value for incident_identifier. Adding an incident_identifier into the payload will override the default value, and allows you to configure specific deduplication rules.
Read more about how incident_identifiers work in the Alert Identification documentation.
Sending Multiple Alerts
If you want to send more than one alert in a single API call, you can modify the JSON payload to send an array of alert objects. When sending multiple alerts at the same time, you can choose to specify the app_key one time instead of for each alert.
Sending multiple alerts with the REST API
BigPanda uses the timestamp to determine the latest status of an alert. If it is not included, BigPanda uses the time when the event is received. To ensure that BigPanda accurately reflects the current status, when sending multiple events, you must include the timestamp for each event or sort the alerts array by when the events occurred, in ascending order.
Parameters
The Alert object schema includes the following attributes:
Attribute | Description | Type |
|---|---|---|
app_key | The unique identifier of the integration, created during the first step of integration. | String |
status | Status of the event. One of | String |
timestamp | (Optional) Time that the event occurred. Unix epoch time when X (in seconds). If no time is specified, the value defaults to the time received in BigPanda. This field is often used for systems that send events in batches. | Timestamp (in seconds) |
<event_property> | Event properties to become tags in BigPanda. The alert object can include any number of custom JSON attributes in string, integer, or array format. | String |
<event_property> | Event properties to become tags in BigPanda. The alert object can include any number of custom JSON attributes in string, integer, or array format. When sending an array, the array values are combined into a single tag, pipe delimited in the UI. | Array |
<event_property> | Event properties to become tags in BigPanda. The alert object can include any number of custom JSON attributes in string, integer, or array format. | Integer |
description | (Optional) Brief summary of the alert for certain monitoring tools - max. 2048 characters. | String |
incident_identifier | (Optional) A unique identifier used to recognize if two events are related to each other. Adding an incident_identifier into the payload will override the default value, and allows you to configure specific deduplication rules. | String |
primary_property | (Optional) Define a specific event property to act as primary property for this payload. The defined event property must be included in the payload. If not, the primary property will be defined by any included default fields. If no primary property is included, the payload will be dropped. | String |
secondary_property | (Optional) Define a specific event property to act as secondary property for this payload. The defined event property must be included in the payload. If not, the secondary property will be defined by any included default fields. | String |
Sample Alert Payloads
Single Alert:
{ "app_key": "<APP KEY>",
"status": "critical",
"host": "production-database-1",
"check": "CPU overloaded",
"timestamp": 1402303570,
"application": "Billing",
"regions":["US1","US2","EMEA"],
"count": 47
"description": "CPU is above warning limit (40%)",
"primary_property": "application",
"secondary_property": "host"}
}
Multi-Alerts
{ "app_key": "<APP KEY>",
"alerts": [{
"status": "critical",
"host": "production-database-1",
"check": "CPU overloaded",
"description": "CPU is above warning limit (40%)"
}, {
"status": "critical",
"host": "production-database-2",
"check": "CPU overloaded",
"timestamp": 1402303570,
"application": "Billing"
}]
}Event Properties
The attributes included in the JSON payload become tags in BigPanda.
You can add any number of custom JSON attributes with a string, integer, or array value to the payload. Common fields include host, host, service, application, device, check, sensor, cluster, node, data center, region, and description.
When configuring the data to send through the Alerts API, send attributes that:
Drive deduplication, correlation, and UI titling. Event attributes enable BigPanda to turn noisy events into high-quality alerts and actionable incidents.
Add context about the event, including where or when the alert triggered. Additional information helps your team understand the full situation while investigating an issue.
Enable automation and categorization. Tags such as
business elementorteamhelp you build environments, autoshare rules, analytics, and workflows.Set next steps for triage and remediation. Including links to runbook wikis or investigation tools helps your team take action quicker. If a tag value starts with
http, BigPanda automatically adds a link button.
Primary and Secondary Properties
Primary and secondary properties are key fields used for event deduplication, normalization, correlation, and titling within the UI. All events sent to BigPanda must include a primary property. Payloads without a primary property will fail to process into the BigPanda pipeline. A secondary property is recommended, but not required.
By default, BigPanda treats certain fields as primary property: host, service, application, or device.
If a payload includes multiple of these fields, or if none can be sent, the primary_property attribute can define a field to function as the primary property.
Secondary property is always optional, but if check or sensor fields are included, they will be treated as the secondary property. Secondary property can also be manually set using the secondary_property field.
For example, you may have an event that is associated with both a host and an application, where the application is primary and the host is secondary. In these cases, use the primary_property and secondary_property attributes to define deduplication and correlation behavior.