Best Practices
Follow these guidelines when building custom integrations with the BigPanda Alerts API.
Event Properties
The attributes included in the JSON payload become tags in BigPanda.
You can add any number of custom JSON attributes with a string, integer, or array value to the payload. Common fields include host, host, service, application, device, check, sensor, cluster, node, data center, region, and description.
When configuring the data to send through the Alerts API, send attributes that:
Drive deduplication, correlation, and UI titling. Event attributes enable BigPanda to turn noisy events into high-quality alerts and actionable incidents.
Add context about the event, including where or when the alert triggered. Additional information helps your team understand the full situation while investigating an issue.
Enable automation and categorization. Tags such as
business elementorteamhelp you build environments, autoshare rules, analytics, and workflows.Set next steps for triage and remediation. Including links to runbook wikis or investigation tools helps your team take action quicker. If a tag value starts with
http, BigPanda automatically adds a link button.
Primary and Secondary Properties
Primary and secondary properties are key fields used for event deduplication, normalization, correlation, and titling within the UI. All events sent to BigPanda must include a primary property. Payloads without a primary property will fail to process into the BigPanda pipeline. A secondary property is recommended, but not required.
By default, BigPanda treats certain fields as primary property: host, service, application, or device.
If a payload includes multiple of these fields, or if none can be sent, the primary_property attribute can define a field to function as the primary property.
Secondary property is always optional, but if check or sensor fields are included, they will be treated as the secondary property. Secondary property can also be manually set using the secondary_property field.
For example, you may have an event that is associated with both a host and an application, where the application is primary and the host is secondary. In these cases, use the primary_property and secondary_property attributes to define deduplication and correlation behavior.
How Many Events to Send (Load Limitations)
Payload size
Alert payloads must be 6MB or smaller. Larger payloads will fail to process with BigPanda. We recommend reviewing your configurations to ensure that only actionable, useful information is being sent to BigPanda.
If an integration generates more than a few hundred events per hour, review the guidelines for when to send events and ensure that the integration is properly configured. Even with hundreds of thousands of monitoring metrics, status changes do not happen very often. If an integration exceeds this guideline, it may be sending duplicate or erroneous events to BigPanda.
Under maximum load, the API can support up to a few hundred events per minute.
When to Send Events
Send an event every time the status of an alert changes. For example, if a new Critical alert is opened or a Critical alert becomes a Warning. The status of an alert typically changes when a metric goes above or below a certain threshold.
Send an updated event when you need to change tag values for an alert. In some rare cases, you may want to send an event for an alert with the same status but a different value for one or more tags. For example, assume you have an alert with status of Warning and description of problem1. You can send a new Warning event with a description of problem2, and the alert status remains the same but the description is updated.
Send an OK resolution event when an alert is no longer active. BigPanda does not resolve alerts automatically. Alerts that have not been resolved remain open, and the corresponding incident also remains open and continues to appear in the incident feed.
Resolution events
Some monitoring tools don't trigger resolution events. In this case, you must manually resolve incidents to remove them from the incident feed.
What Tags to Send
The attributes included in the JSON payload become tags in BigPanda.
You can add any number of custom JSON attributes with a string, integer, or array value to the payload. Common fields include host, host, service, application, device, check, sensor, cluster, node, data center, region, and description.
When configuring the data to send through the Alerts API, send attributes that:
Drive deduplication, correlation, and UI titling. Event attributes enable BigPanda to turn noisy events into high-quality alerts and actionable incidents.
Add context about the event, including where or when the alert triggered. Additional information helps your team understand the full situation while investigating an issue.
Enable automation and categorization. Tags such as
business elementorteamhelp you build environments, autoshare rules, analytics, and workflows.Set next steps for triage and remediation. Including links to runbook wikis or investigation tools helps your team take action quicker. If a tag value starts with
http, BigPanda automatically adds a link button.
Nested objects
Nested objects are not supported when sending events to the Rest API
Incident_identifier
The incident_identifier is a unique ID used in BigPanda to recognize if two events are related to each other. By default BigPanda will use a combination of the primary and secondary properties to create a value for incident_identifier. Adding an incident_identifier into the payload will override the default value, and allows you to configure specific deduplication rules.
Read more about how incident_identifiers work in the Alert Identification documentation.