Grafana

To use this integration, you can configure a webhook notification channel in Grafana. This channel forwards alerts to the Grafana integration where they are normalized and transmitted to BigPanda. BigPanda then processes and correlates the alert data from Grafana to create and maintain up-to-date incidents in BigPanda.

❗️

Grafana 8+ Alerts

Grafana introduced an update to the format of alert notifications in v8+. This version now uses Prometheus as its Alert Manager rather than its legacy solution. If using version 8 or later, select the Grafana (version 8+). If using earlier versions (5.0 - 7.9), select Grafana.

👍

Open Integration Manager

The Open Integration Manager is available for use with Grafana v8+. For more information, see the Open Integration Manager documentation.

❗️

Payload Size

Alert payloads must be 6MB or smaller. Larger payloads will fail to process with BigPanda. We recommend reviewing your configurations to ensure that only actionable, useful information is being sent to BigPanda.

Key Features

• Uses native webhook notification channel in Grafana to forward alerts to BigPanda
• Automatically resolves incidents in BigPanda when they resolve in Grafana
• Extensible alert configuration
• Intelligently correlates alerts from Grafana to help you understand and respond faster to production issues

Data Model

BigPanda normalizes alert data from Grafana into tags. You can use tag values to filter the incident feed and to define filter conditions for Environments. The primary and secondary properties are also used during the deduplication and correlation processes. The following specifies the mapping of Grafana alert properties to BigPanda tags.

Install the Integration

Grafana v8.0

❗️

Grafana 8+ Alerts

Grafana introduced an update to the format of alert notifications in v8+. This version now uses Prometheus as its Alert Manager rather than its legacy solution. If using version 8 or later, select the Grafana (version 8+). If using earlier versions (5.0 - 7.9), select Grafana.

Create an App Key

Create an app key in BigPanda.

👍

Integration Specific

You'll need a separate app key for each integrated system.

Setup Contact Point in Grafana

1. Login to your Grafana instance.

2. In the left sidebar, click the alerting bell icon, then open the Contact points tab.

3. Click the New contact point button.

4. Fill in the Channel Settings as follows:

   • Name: Useful contact point name, such as BigPanda
   • Contact point type: Webhook
   • URL: https://integrations.bigpanda.io/prometheus/alerts?access_token=<Your Org Bearer Token>&app_key=<Your App Key>

5. Fill in the Optional Webhook Settings:

   • Http Method: POST
   • HTTP Basic Authentication - Username: leave blank
   • HTTP Basic Authentication - Password: leave blank

6. Click Test to ensure the configuration is correct. If successful, a green toast message should appear saying Test Notification Sent and you should receive an alert in BigPanda.

7. Click Save contact point.

Assign Contact Point to Alerts

Notification policies are used to route alerts to different contact points. Repeat the following process for every alert you wish to send to BigPanda:

1. In the left sidebar, click the alerting bell icon, then open the Notification policies tab.
2. Either modify the root policy or add new policy for specific routing.
3. Save the policy.

Advanced Configuration

Sending Custom Properties with Alerts

You can use the BigPanda Grafana integration to send custom attributes along with your alerts. To do this, add details for each alert you wish to send custom attributes for.

1. Within the Alert rules, locate and open the rule with the alert you wish to edit.
2. Go to Add details for your alert section.
3. Click Add Info to add additional tags. Select Add new to add new tag names as needed.
4. Click Save and exit.

Earlier Grafana Versions

❗️

Grafana 8+ Alerts

Grafana introduced an update to the format of alert notifications in v8+. This version now uses Prometheus as its Alert Manager rather than its legacy solution. If using version 8 or later, select the Grafana (version 8+). If using earlier versions (5.0 - 7.9), select Grafana.

Create an App Key

Create an app key in BigPanda.

👍

Integration Specific

You'll need a separate app key for each integrated system.

Setup Notification Channel in Grafana

1. Login to your Grafana instance

2. In the left sidebar click the alerting bell icon then open the Notification channels tab

3. Click the New channel button

4. Fill in the Channel Settings as follows:

   • Name: Useful channel name, such as BigPanda
   • Type: webhook
   • Send on all alerts: Check the box if you’d like to send all alerts to BigPanda. If not then you will need to manually add the BigPanda notification channel to every alert
   • Include image: Check the box if you’d like a link to an image of the metric to be available from the incident in BigPanda

5. Fill in the Webhook Settings as follows:

   • URL: https://integrations.bigpanda.io/grafana/alerts?access_token=<Your Org Bearer Token>&app_key=<Your App Key>
   • Http Method: POST
   • Username: leave blank
   • Password: leave blank

6. Click Send Test to ensure the configuration is correct. If successful a green toast message should appear saying Test Notification Sent and you should receive an alert in BigPanda

7. Click Save to save the notification channel

Assign Notification Channel to Alerts

Note: You can skip this step if you chose to send all alerts to the notification channel in the previous step.

Repeat the following process for every alert you wish to send to BigPanda:

1. Under the configuration for the alert click on Notifications
2. Click the + icon in the Send Top section of the form
3. Select the BigPanda notification channel created in the previous step
4. Save the dashboard

Advanced Configuration

Sending Custom Properties with Alerts

You can use the BigPanda Grafana integration to send custom attributes along with your alerts. You do this by adding a configuration block to the notification message on each alert you want to send custom attributes for.

1. Locate and open for editing the panel with the alert you wish to edit

2. Open the Alert > Notifications section

3. In the Message field add the [bigpanda_config] config block to the end, customizing as needed.

   Your normal alert message text, no need to change me...
   
   [bigpanda_config]
   my_custom_prop_1 = value_1
   my_custom_prop_2 = value_2
   

   The configuration section is INI format and supports comments and arrays according to the INI format. You can add as many custom properties as you wish.

   Note: All text after the [bigpanda_config] section will be treated as part of the configuration. All text before will be processed normally as part of the alert description.

4. Save the dashboard

Delete the Integration

Deleting an integration requires that you remove the integration in both the integrated system and BigPanda. We recommend that you first uninstall the integration on the integrated system to prevent traffic from being sent and rejected by BigPanda, since the app key will not exist once you delete the integration in BigPanda.

❗️

Caution During Replacement

When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.

Deactivate Inbound Integration

If you want to stop sending data to BigPanda but don’t want to delete your integration, you can temporarily deactivate it.

To deactivate an inbound integration:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list. This will open integration details on the right side of the window.
2. At the top of the integration details, click the Active/Inactive toggle next to the application name to change the status of the integration.

In the integrations list, inactive integrations will be marked with a gray bar.

Stop Sending Data to BigPanda

1. Login to your Grafana instance.
2. In the left sidebar click the alerting bell icon then open the Notification channels tab.
3. Locate the BigPanda notification channel and click the red X button to delete it.
4. Remove any [bigpanda_config] blocks you configured in your alert messages.

Delete the Integration in BigPanda

Take the following steps to delete the integration from BigPanda:

1. In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
2. In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will be removed immediately.

👍

️Automatic Alert Resolution for Deleted Integrations

All active alerts from the integration will be resolved after deletion.

❗️

Data Removal

This procedure does not remove any data from the integrated system.