Supported Versions: | Type: |
Key Features
- Uses native webhook notifier in Prometheus Alertmanager to forward alerts to BigPanda
- Automatically resolves incidents in BigPanda when they resolve in Prometheus Alertmanager
- Extensible alert formatting using Prometheus Alertmanager notification templates
- Intelligently correlates alerts from Prometheus Alertmanager to help you understand and respond faster to production issues
How It Works
The integration works by configuring a webhook notifier in Prometheus Alertmanager which forwards any specified alert to the Prometheus integration where it is normalized and transmitted to BigPanda. BigPanda then processes and correlates the alert data from Prometheus to create and maintain up-to-date incidents in BigPanda.
Data Model
BigPanda normalizes alert data from Prometheus into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the correlation process. The following specifies the mapping of Prometheus alert properties to BigPanda tags
BigPanda Property | Prometheus Property |
---|---|
primary_property | |
secondary_property | |
status | Determined from severity AND/OR status |
description | annotations.description |
summary | annotations.summary |
startsAt | startsAt |
endsAt | endsAt |
generatorURL | generatorURL |
source_system | "prometheus" |
timestamp | timestamp OR auto-generated if not sent |
Primary & Secondary Key Logic
The primary & secondary for the BigPanda alert are dynamically selected based on properties present within the Prometheus alert. Both primary and secondary properties can also be overridden.
Primary Property Determination
- Set to value of
bp_primary_property
ifbp_primary_property
annotation exists - Set to
instance
if theinstance
label exists andbp_primary_property
annotation does not - Set to
job
if thejob
label exists and bothinstance
label andbp_primary_property
annotation do not - Set to
alertname
if no other matching annotation or label
Determining Secondary Property
- Set to value of
bp_secondary_property
ifbp_secondary_property
annotation exists - Set to
alertname
if primary property is not already set to alertname - Omitted if no other condition is satisified
Installing the Integration
Pre-requisites
Configure Prometheus Alertmanager
- Login to your Prometheus Alertmanager server
- Open your Alertmanager config YAML file for editing
- In the
receivers
config section create a new webhook configuration for BigPanda either under an existing receiver or as a new receiver. eg.
receivers:
- name: 'bigpanda'
webhook_configs:
- send_resolved: true
url: 'https://inbound.bigpanda.io/prometheus/alerts?app_key=BIGPANDA_APP_KEY'
http_config:
bearer_token: BIGPANDA_TOKEN
In this example you need to replace BIGPANDA_APP_KEY and BIGPANDA_TOKEN placeholders with the appropriate values from the integration page.
- To send all alerts to BigPanda configure the default route to send to the receiver which you added the webhook configuration to. eg.
route:
...
receiver: 'bigpanda'
Note: You can add the receiver to a different route than the default but it will only receiver alerts that are delivered to that route.
- Save the config file
- Restart your Alertmanager server so it loads the updated configuration
Additional configuration
Timestamps (strongly suggested)
By default subsequent message from an alert will be deduped by BigPanda as the Prometheus alert property startsAt
remains constant throughout the lifetime of the triggered alert (until it is resolved). In order to show the full stream of incoming alerts (avoid deduplication) it is necessary to add a custom timestamp to the alerts via an annotation template on the rule that generates the alerts.
- Login to your Prometheus server & open your Prometheus rules YAML file for editing
- Locate your rule
- Under the
annotations
for the rule add this timestamp template - Save the rules file & restart your Prometheus server so it loads the updated configuration
annotations:
timestamp: '{{ with query "time()" }}{{ . | first | value }}{{ end }}'
...
Severity Levels
BigPanda supports a certain set of severity levels for Prometheus alerts, it maps those severities to BigPanda statuses. BigPanda looks for the severity
label on an alert when determining the BigPanda status, if it is not present then the status will default to critical
for firing alerts. Resolved alerts always have an ok status.
The supported severity levels, and the mapping logic, follows.
- Severity
page
orcritical
➡ critical - Severity
warn
orwarning
➡ warning - Severity
acknowledged
orunk
➡ pass through as same - If there is no
severity
label and the status isfiring
then the status will be critical - If the Prometheus status is
resolved
the BigPanda status will always be OK
Primary & Secondary Property Overrides
BigPanda looks for a specific set of properties when trying to determine the primary and secondary properties for a Prometheus alert. You can override the keys for the primary and secondary properties by specifying one or both of the following annotations on your alert.
- Annotation
bp_primary_property
specifies which Prometheus alert property (either an annotation or label) to use as the primary property - Annotation
bp_secondary_property
specifies which Prometheus alert property (either an annotation or label) to use as the secondary property
Proxy Support
To send alerts from Alertmanager to BigPanda via a proxy you can add the proxy_url
setting on the BigPand receiver configuration:
receivers:
- name: 'bigpanda'
webhook_configs:
- send_resolved: true
url: 'https://inbound.bigpanda.io/prometheus/alerts?app_key=BIGPANDA_APP_KEY'
http_config:
bearer_token: BIGPANDA_TOKEN
proxy_url: 'http://myproxydomain.tld:port'
Uninstalling the Integration
Remove Integration from Prometheus
- Login to your Prometheus Alertmanager server
- Open your Alertmanager config YAML file for editing
- Remove the webhook configuration for BigPanda
- If you have a dedicated receiver for BigPanda also remove the entire receiver and remove the receiver from the route you were sending on
- Save the config file
- Restart your Alertmanager server so it loads the updated configuration
Manually Resolve All Open Alerts
Manually resolve any open Prometheus alerts as they will not auto-resolve with the integration removed.
Remove Integration from BigPanda
Delete the Integration in BigPanda to remove the Prometheus integration from your UI.
Updated 10 months ago
Recommended Reading
Integrating a Monitoring Service |