Prometheus
Send monitoring events from Prometheus to BigPanda.
Supported Versions | Type | Authentication Type |
---|---|---|
Alertmanager <= v0.24.0 | Webhook | Org Bearer Token |
The Prometheus integration works by configuring a webhook notifier in Prometheus Alertmanager. The webhook notifier forwards any specified alert to the Prometheus integration where it is normalized and transmitted to BigPanda. BigPanda then processes and correlates the alert data from Prometheus to create and maintain up-to-date incidents in BigPanda.
Caution When Enabling Prometheus
Prometheus can be a noisy monitoring system. For customers using the BigPanda consumption based pricing model, it can consume large amounts of credits very quickly.
Exercise caution when adding the BigPanda receiver to the default route. Instead, consider adding the receiver only to routes that you know contain valuable information rather than noisy routes.
The Open Integration Manager is available for use with Prometheus. For more information, see the Open Integration Manager documentation.
Key Features
- Uses native webhook notifier in Prometheus Alertmanager to forward alerts to BigPanda.
- Automatically resolves incidents in BigPanda when they resolve in Prometheus Alertmanager.
- Extensible alert formatting using Prometheus Alertmanager notification templates.
- Intelligently correlates alerts from Prometheus Alertmanager to help you understand and respond faster to production issues.
Data Model
BigPanda normalizes alert data from Prometheus into tags. You can use tag values to filter the incident feed and to define filter conditions for environments. The primary and secondary properties are also used during the deduplication and correlation processes. The following specifies the mapping of Prometheus alert properties to BigPanda tags:
BigPanda Property | Prometheus Property |
---|---|
primary_property | See Primary & Secondary Key Logic |
secondary_property | See Primary & Secondary Key Logic |
status | Determined from severity AND/OR status |
description | annotations.description |
summary | annotations.summary |
startsAt | startsAt |
endsAt | endsAt |
generatorURL | generatorURL |
source_system | "prometheus" |
timestamp | timestamp OR auto-generated if not sent |
Primary and Secondary Key Logic
The primary and secondary for BigPanda alerts are dynamically selected based on properties present within the Prometheus alert. Both primary and secondary properties can also be overridden.
Primary Property Determination
- Set to value of
bp_primary_property
ifbp_primary_property
annotation exists - Set to
instance
if theinstance
label exists andbp_primary_property
annotation does not - Set to
job
if thejob
label exists and bothinstance
label andbp_primary_property
annotation do not - Set to
alertname
if no other matching annotation or label
Determining Secondary Property
- Set to value of
bp_secondary_property
ifbp_secondary_property
annotation exists - Set to
alertname
if primary property is not already set to alertname - Omitted if no other condition is satisified
Install the Integration
Administrators can install the integration by following the on-screen instructions in BigPanda.
Uninstall the Integration
Deleting an integration requires changes to both the integrated system and BigPanda. You must uninstall the integration on the integrated system and then delete the integration from BigPanda.
When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.
Remove the Integration from Prometheus
- Sign in to your Prometheus Alertmanager server.
- Open your Alertmanager config YAML file for editing.
- Remove the webhook configuration for BigPanda.
- If you have a dedicated receiver for BigPanda also remove the entire receiver and remove the receiver from the route you were sending on.
- Save the config file.
- Restart your Alertmanager server so it loads the updated configuration.
Manually Resolve All Open Alerts
Manually resolve any open Prometheus alerts as they will not auto-resolve with the integration removed.
Delete the Integration from BigPanda
- In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
- In the integration details on the right, click Delete Integration. A support message opens, pre-populated with a request to delete the selected integration.
- Press Enter to send the request.
- The BigPanda support team will remove the integration from the UI.
οΈ
This procedure does not remove any data from BigPanda or the integrated system. As needed, remove data from each system before deleting the integration.
Updated 6 months ago