PagerDuty
Automatically create PagerDuty incidents from incidents in BigPanda.
Supported Versions | Type | Authentication Type |
|---|---|---|
SaaS Deployments | API | Org Bearer Token |
Improve your team's response times and enhance visibility across systems by creating incident workflows between BigPanda and PagerDuty.
Create one integration per PagerDuty service
BigPanda integrations are configured at the service level in PagerDuty and will only create incidents for that service. If you want to assign BigPanda incidents to more than one service, create a separate integration in BigPanda for each PagerDuty service you want to use.
Key Features
Automatic incident creation. Share BigPanda incidents to PagerDuty to automatically create PagerDuty incidents with rich data from correlated alerts.
Bi-directional resolution. Resolve incidents in either system to streamline your workflow.
Comment syncing. Make a comment in BigPanda or a note in PagerDuty to automatically update the connected incident, improving communication and consistency.
Accessing BigPanda incident data in PagerDuty
When you share an incident from BigPanda, the title of the new incident in PagerDuty will include the host name for the first alert, the number of alerts in the incident, and the incident check (ex, [BigPanda] Incident wnl-prod-syn555 and 4 more / Synthetic Failure).
Access the data from BigPanda by either clicking Show Details under the PagerDuty incident, or by scrolling down to the Alerts tab within a specific PagerDuty incident. This will show you the Custom Details table, which contains a summary of the correlated alerts and all incident tags, as well as direct links to the BigPanda incident, the BigPanda incident timeline, and a preview of the BigPanda incident that can be viewed without a BigPanda account.
The Custom Details alert field within a PagerDuty incident
The description row tells you how many alerts are correlated to this incident as well as the status of those alerts. It also gives you a summary of key data for each alert:
status: The current status of the incident in BigPanda (critical, warning, unknown, acknowledged, resolved).primaryProperty: The primary property tag for the alert (if no property has been defined, the value will behost).secondaryProperty: The secondary property tag for the alert (if no property has been defined, the value will becheck).description: The alert's description tag.startedOn: The Unix time when the initial alert started, formatted as date and time.sourceSystem: The integrated monitoring system that sent the alert to BigPanda.
Adding custom alert tags to PagerDuty incidents
Custom alert tags are not included by default, but can be added to PagerDuty integrations via custom headers. Alert tags sent through custom headers will not appear with other alert data in the Description row, but will instead create a separate row in the Custom Details table.
See our advanced PagerDuty configuration documentation for instructions.
Install the PagerDuty integration
Create an app key
Create an app key in BigPanda.
Integration specific
You'll need a separate app key for each integrated system.
App key configuration in BigPanda
Create an API Key in PagerDuty
In PagerDuty, go to Integrations > API Access Keys.
Click Create a New API Key and give it a short description before generating the key.
Copy the key and keep it somewhere you can find it again if needed, as you will only be able to access it once. You will need this key for both the PagerDuty Webhook Configuration and BigPanda Webhook Configuration steps.
Create a Webhook in PagerDuty
From the top menu go to Integrations > Generic Webhook (v3) or from within the service go to Integrations > Manage Webhooks
Click +New Webhook.
Use the following information to set up the webhook:
Webhook URL: Enter the URL
https://integrations.bigpanda.io/latest/pagerduty/webhook?token=<Your Org Bearer Token>Scope Type: The type is
Service.Scope: Select the service that this webhook should trigger on.
Description: Appropriate description of the webhook.
Webhook Status: Click to enable.
Events Subscription: Select the events you want to enable. Note that BigPanda only supports incident events (
incident.xyz), service events will be ignored.
Click Add Custom Header.
Set the name of the custom header to
x-bp-config-pdtoken, then copy your PagerDuty API key into the value field.Click Add Webhook.
Create an Integration in PagerDuty
Go to Services > Services Directory.
Select the service you wish to add the BigPanda integration to.
In the service details page, from the Integrations tab, click + Add an Integration.
Select the Events API v2 tile from the Integrations selection screen. (Note, do not use the BigPanda or API v1 options that may appear in the search bar)
Click Add Integration to save. You will be redirected to the Integrations page for your service.
Click the cog wheel to open the integration settings.
Copy the Integration Key. You’ll need it in the BigPanda Webhook Configuration step.
Configure BigPanda Webhook
Paste the integration key from your PagerDuty integration as the value for the
x-bp-config-pdkeyheader.Add a new custom header and use the header key
x-bp-config-pdtoken. For the value, enter the API key you created in PagerDuty.Click Configure Webhook.
Test the integration
You are now ready to share incidents to PagerDuty!
After you share an incident to PagerDuty, the integration will generate a comment on the incident that includes the PagerDuty ticket link. It will also generate a separate comment that details the service where the ticket was created, who it was assigned to, and what escalation policy was applied.
If these comments do not appear, review the installation steps and make sure you included the PagerDuty API key in the webhook configuration for both PagerDuty and BigPanda.
Create multiple PagerDuty integrations
If you want to share BigPanda incidents to other PagerDuty services, create a new PagerDuty integration in BigPanda, then follow the same process to create a new webhook and integration in PagerDuty. You do not need to create a new API key for each integration.
(Optional) Send custom alert tags to PagerDuty
If you want to send custom alert tags to PagerDuty in addition to the default incident data, you can do so through custom headers in your webhook configuration.
Alert tags sent through custom headers will not appear with other alert data in the Description row, but will instead create a separate row in the Custom Details table. If an incident contains multiple alerts that share this tag, the values will all appear in the same row, separated by commas.
To configure these custom headers:
Go to the Integrations tab in BigPanda and either Create a New Integration or select the desired PagerDuty integration and click Configuration to access the setup page.
In the webhook configuration step, add a new custom header to the list. Enter the following into the key and value fields:
Header key: Enter the alert tag name using the format
x-bp-field-<tag-name>Header value: Enter your desired name for the new Custom Details table row in PagerDuty. (Note that you do not need to configure this in PagerDuty first. The integration will create this row automatically.)
Repeat step #2 for every alert tag you wish to add.
Click Configure Webhook to save your header.
If a shared incident includes an alert with this tag, it will now appear as a row in the Custom Details table with the name you entered in the Header value. (Note that this row will only appear if the incident has an alert with this tag. If the incident does not have any alerts with this tag, the row will not be generated for that incident.)
Uninstall the Integration
Deleting an integration requires that you remove the integration in both the integrated system and BigPanda. We recommend that you first uninstall the integration on the integrated system to prevent traffic from being sent and rejected by BigPanda, since the app_key will not exist once you delete the integration in BigPanda.
Caution during replacement
When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.
Stop Sending Data from BigPanda
In BigPanda, disable any settings that send data to the integrated system. This includes modifying or removing AutoShare rules or Environments that are tied to the integration.
In PagerDuty, navigate to Configuration > Services.
Select the service with the BigPanda Integration configured.
From the Integrations tab click the settings gear icon and select Delete.
If you have configured an API Key, remove the key from PagerDuty.
In PagerDuty, navigate to Configuration > API Access.
Locate the API Key created for the BigPanda integration and select Delete, confirming when prompted.
Delete the integration in BigPanda
Take the following steps to delete the integration from BigPanda:
In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will be removed immediately.
️Data Removal
This procedure does not remove any data from BigPanda or the integrated system. As needed, remove data from each system before deleting the integration.