Prometheus

The Prometheus integration works by configuring a webhook notifier in Prometheus Alertmanager. The webhook notifier forwards any specified alert to the Prometheus integration where it is normalized and transmitted to BigPanda. BigPanda then processes and correlates the alert data from Prometheus to create and maintain up-to-date incidents in BigPanda.

❗️

Caution When Enabling Prometheus

Prometheus can be a noisy monitoring system. For customers using the BigPanda consumption based pricing model, it can consume large amounts of credits very quickly.

Exercise caution when adding the BigPanda receiver to the default route. Instead, consider adding the receiver only to routes that you know contain valuable information rather than noisy routes.

👍

Open Integration Manager

The Open Integration Manager is available for use with Prometheus. For more information, see the Open Integration Manager documentation.

❗️

Payload Size

Alert payloads must be 6MB or smaller. Larger payloads will fail to process with BigPanda. We recommend reviewing your configurations to ensure that only actionable, useful information is being sent to BigPanda.

Key Features

• Uses native webhook notifier in Prometheus Alertmanager to forward alerts to BigPanda.
• Automatically resolves incidents in BigPanda when they resolve in Prometheus Alertmanager.
• Extensible alert formatting using Prometheus Alertmanager notification templates.
• Intelligently correlates alerts from Prometheus Alertmanager to help you understand and respond faster to production issues.

Data Model

BigPanda normalizes alert data from Prometheus into tags. You can use tag values to filter the incident feed and to define filter conditions for environments. The primary and secondary properties are also used during the deduplication and correlation processes. The following specifies the mapping of Prometheus alert properties to BigPanda tags:

Primary and Secondary Key Logic

The primary and secondary for BigPanda alerts are dynamically selected based on properties present within the Prometheus alert. Both primary and secondary properties can also be overridden.

Primary Property Determination

• Set to value of bp_primary_property if field exists
• Set to instance if no bp_primary_property field
• Set to job if the no instance or bp_primary_property field
• Set to alertname if no other matching field or label

Determining Secondary Property

• Set to value of bp_secondary_property if field exists
• Set to alertname if primary property is not already set to alertname
• Omitted if no other condition is satisified

Install the Integration

Create an App Key

Create an app key in BigPanda.

👍

Integration Specific

You'll need a separate app key for each integrated system.

Configure Prometheus Alertmanager

1. Login to your Prometheus Alertmanager server.

2. Open your Alertmanager config YAML file for editing.

3. In the receivers config section create the following webhook configuration for BigPanda either under an existing receiver or as a new receiver:

   receivers:
   - name: 'bigpanda'
     webhook_configs:
     - send_resolved: true
       url: 'https://integrations.bigpanda.io/prometheus/alerts?<Your App Key>'
       http_config:
         bearer_token: <Your Org Bearer Token>
   

4. To send all alerts to BigPanda configure the default route to send to the receiver which you added the webhook configuration to. For example:

   route:
     ...
     receiver: 'bigpanda'
   

   Note: You can add the receiver to a different route than the default but it will only receiver alerts that are delivered to that route.

5. Save the config file.

6. Restart your Alertmanager server so it loads the updated configuration.

Configure Timestamps

Configuring timestamps is an optional, but highly recommended, step as without this configuration subsequent messages from a triggered alert may be deduped.

1. Login to your Prometheus server and open your Prometheus rules YAML file for editing.

2. Locate your rule.

3. Under the annotations for the rule add this timestamp template:

   annotations:
     timestamp: '{{ with query "time()" }}{{ . | first | value }}{{ end }}'
     ...
   

4. Save the rules file & restart your Prometheus server so it loads the updated configuration.

Primary & Secondary Property Overrides (optional)

BigPanda looks for a specific set of properties when trying to determine the primary and secondary properties for a Prometheus alert. You can override the keys for the primary and secondary properties by specifying one or both of the following annotations on your alert.

• Annotation bp_primary_property specifies which Prometheus alert property (either an annotation or label) to use as the primary property.
• Annotation bp_secondary_property specifies which Prometheus alert property (either an annotation or label) to use as the secondary property.

Uninstall the Integration

Deleting an integration requires that you remove the integration in both the integrated system and BigPanda. We recommend that you first uninstall the integration on the integrated system to prevent traffic from being sent and rejected by BigPanda, since the app key will not exist once you delete the integration in BigPanda.

❗️

Caution During Replacement

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Deactivate Inbound Integration

If you want to stop sending data to BigPanda but don’t want to delete your integration, you can temporarily deactivate it.

To deactivate an inbound integration:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list. This will open integration details on the right side of the window.
2. At the top of the integration details, click the Active/Inactive toggle next to the application name to change the status of the integration.

In the integrations list, inactive integrations will be marked with a gray bar.

❗️

Alert Resolution for Inactive Integrations

Any active alerts belonging to an inactive integration must be manually resolved or they will stay in the system until the auto-resolve window is reached.

Stop Sending Data to BigPanda

1. Sign in to your Prometheus Alertmanager server.
2. Open your Alertmanager config YAML file for editing.
3. Remove the webhook configuration for BigPanda.
4. If you have a dedicated receiver for BigPanda also remove the entire receiver and remove the receiver from the route you were sending on.
5. Save the config file.
6. Restart your Alertmanager server so it loads the updated configuration.

Delete the Integration in BigPanda

Take the following steps to delete the integration from BigPanda:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
2. In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will be removed immediately.

👍

️Automatic Alert Resolution for Deleted Integrations

All active alerts from the integration will be resolved after deletion.

❗️

Data Removal

This procedure does not remove any data from the integrated system.