Prometheus Installation Instructions

How to Integrate Prometheus with BigPanda

Create an App Key

First create an App Key. You'll need a separate App Key for each integrated system.

Configure Prometheus Alertmanager

1. Login to your Prometheus Alertmanager server

2. Open your Alertmanager config YAML file for editing

3. In the receivers config section create the following webhook configuration for BigPanda either under an existing receiver or as a new receiver:

receivers:
- name: 'bigpanda'
  webhook_configs:
  - send_resolved: true
    url: '$INTEGRATIONS_API_BASE_URL/prometheus/alerts?app_key=$YOUR_ID'
    http_config:
      bearer_token: $TOKEN

4. To send all alerts to BigPanda configure the default route to send to the receiver which you added the webhook configuration to. eg.

route:
  ...
  receiver: 'bigpanda'

Note: You can add the receiver to a different route than the default but it will only receiver alerts that are delivered to that route.

5. Save the config file

6. Restart your Alertmanager server so it loads the updated configuration

Configure Timestamps

Configuring timestamps is an optional but highly recommended step as without this configuration subsequent messages from a triggered alert may be deduped.

1. Login to your Prometheus server & open your Prometheus rules YAML file for editing

2. Locate your rule

3. Under the annotations for the rule add this timestamp template

annotations:
  timestamp: '{{ with query "time()" }}{{ . | first | value }}{{ end }}'
  ...

4. Save the rules file & restart your Prometheus server so it loads the updated configuration

Primary & Secondary Property Overrides (optional)

BigPanda looks for a specific set of properties when trying to determine the primary and secondary properties for a Prometheus alert. You can override the keys for the primary and secondary properties by specifying one or both of the following annotations on your alert.

  • Annotation bp_primary_property specifies which Prometheus alert property (either an annotation or label) to use as the primary property
  • Annotation bp_secondary_property specifies which Prometheus alert property (either an annotation or label) to use as the secondary property