Key Features

• Uses native webhook notifier in Prometheus Alertmanager to forward alerts to BigPanda
• Automatically resolves incidents in BigPanda when they resolve in Prometheus Alertmanager
• Extensible alert formatting using Prometheus Alertmanager notification templates
• Intelligently correlates alerts from Prometheus Alertmanager to help you understand and respond faster to production issues

How It Works

The integration works by configuring a webhook notifier in Prometheus Alertmanager which forwards any specified alert to the Prometheus integration where it is normalized and transmitted to BigPanda. BigPanda then processes and correlates the alert data from Prometheus to create and maintain up-to-date incidents in BigPanda.

Data Model

BigPanda normalizes alert data from Prometheus into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the correlation process. The following specifies the mapping of Prometheus alert properties to BigPanda tags

Primary & Secondary Key Logic

The primary & secondary for the BigPanda alert are dynamically selected based on properties present within the Prometheus alert. Both primary and secondary properties can also be overridden.

Primary Property Determination

• Set to value of bp_primary_property if bp_primary_property annotation exists
• Set to instance if the instance label exists and bp_primary_property annotation does not
• Set to job if the job label exists and both instance label and bp_primary_property annotation do not
• Set to alertname if no other matching annotation or label

Determining Secondary Property

• Set to value of bp_secondary_property if bp_secondary_property annotation exists
• Set to alertname if primary property is not already set to alertname
• Omitted if no other condition is satisified

Installing the Integration

Pre-requisites

• Create a new Prometheus integration in the BigPanda console

Configure Prometheus Alertmanager

1. Login to your Prometheus Alertmanager server
2. Open your Alertmanager config YAML file for editing
3. In the receivers config section create a new webhook configuration for BigPanda either under an existing receiver or as a new receiver. eg.

receivers:
- name: 'bigpanda'
  webhook_configs:
  - send_resolved: true
    url: 'https://inbound.bigpanda.io/prometheus/alerts?app_key=BIGPANDA_APP_KEY'
    http_config:
      bearer_token: BIGPANDA_TOKEN

In this example you need to replace BIGPANDA_APP_KEY and BIGPANDA_TOKEN placeholders with the appropriate values from the integration page.

4. To send all alerts to BigPanda configure the default route to send to the receiver which you added the webhook configuration to. eg.

route:
  ...
  receiver: 'bigpanda'

Note: You can add the receiver to a different route than the default but it will only receiver alerts that are delivered to that route.

5. Save the config file
6. Restart your Alertmanager server so it loads the updated configuration

Additional configuration

Timestamps (strongly suggested)

By default subsequent message from an alert will be deduped by BigPanda as the Prometheus alert property startsAt remains constant throughout the lifetime of the triggered alert (until it is resolved). In order to show the full stream of incoming alerts (avoid deduplication) it is necessary to add a custom timestamp to the alerts via an annotation template on the rule that generates the alerts.

1. Login to your Prometheus server & open your Prometheus rules YAML file for editing
2. Locate your rule
3. Under the annotations for the rule add this timestamp template
4. Save the rules file & restart your Prometheus server so it loads the updated configuration

annotations:
  timestamp: '{{ with query "time()" }}{{ . | first | value }}{{ end }}'
  ...

Severity Levels

BigPanda supports a certain set of severity levels for Prometheus alerts, it maps those severities to BigPanda statuses. BigPanda looks for the severity label on an alert when determining the BigPanda status, if it is not present then the status will default to critical for firing alerts. Resolved alerts always have an ok status.

The supported severity levels, and the mapping logic, follows.

• Severity page or critical ➡ critical
• Severity warn or warning ➡ warning
• Severity acknowledged or unk ➡ pass through as same
• If there is no severity label and the status is firing then the status will be critical
• If the Prometheus status is resolved the BigPanda status will always be OK

Primary & Secondary Property Overrides

BigPanda looks for a specific set of properties when trying to determine the primary and secondary properties for a Prometheus alert. You can override the keys for the primary and secondary properties by specifying one or both of the following annotations on your alert.

• Annotation bp_primary_property specifies which Prometheus alert property (either an annotation or label) to use as the primary property
• Annotation bp_secondary_property specifies which Prometheus alert property (either an annotation or label) to use as the secondary property

Proxy Support

To send alerts from Alertmanager to BigPanda via a proxy you can add the proxy_url setting on the BigPand receiver configuration:

receivers:
- name: 'bigpanda'
  webhook_configs:
  - send_resolved: true
    url: 'https://inbound.bigpanda.io/prometheus/alerts?app_key=BIGPANDA_APP_KEY'
    http_config:
      bearer_token: BIGPANDA_TOKEN
      proxy_url: 'http://myproxydomain.tld:port'

Uninstalling the Integration

Remove Integration from Prometheus

1. Login to your Prometheus Alertmanager server
2. Open your Alertmanager config YAML file for editing
3. Remove the webhook configuration for BigPanda
4. If you have a dedicated receiver for BigPanda also remove the entire receiver and remove the receiver from the route you were sending on
5. Save the config file
6. Restart your Alertmanager server so it loads the updated configuration

Manually Resolve All Open Alerts

Manually resolve any open Prometheus alerts as they will not auto-resolve with the integration removed.

Remove Integration from BigPanda

Delete the Integration in BigPanda to remove the Prometheus integration from your UI.