Changes
BigPanda’s standard integrations collect and normalize change data from change feeds such as CI/CD pipelines, Change Management tools, auditing systems, and orchestration tools.
BigPanda includes several standard integrations ready to connect your change feeds, or create custom integrations using the Changes REST API.
Connecting each of your change feeds to BigPanda gives your Ops teams deeper insights into the system changes that may be triggering system events and outages. Change integrations in particular dramatically speed up MTTR and dramatically reduce the number of bridge calls by proactively encouraging collaboration between Ops and Dev teams.
Select the desired tool from the list below to reference detailed information about the standard Topology integration. See the Standard Integrations to see the complete list of available integrations that are built-in to BigPanda. To build custom change integrations, use the Changes RCC API.
Change Data Manager (CDM)
BigPanda’s Root Cause Changes (RCC) correlates change data with active incidents to dramatically speed up the root cause analysis process for change-related issues.
The Change Data Manager (CDM) enables you to create customizable change integrations by configuring a generic integration rather than creating custom code. Quickly integrate more of your CI/CD and change management tools so you can correlate those changes with incidents in BigPanda.
Key Features
Build your own custom change integrations
Ingest change data from emails
Tag mapping enables systems to send native payloads, reducing necessary tool configuration
Configurable fields include:
Users can map payload fields to populate key change values such as status and start time, and also preview sample changes based on the configuration settings.
The Change Data Manager (CDM) normalizes payload data for the BigPanda enrichment engine, and is ideal for change management tools that do not support customizing REST API payloads.
Single Change or Array of Changes
CDM is able to read multiple changes from a single payload using Tag Mapping.
Change Data Manager
Install the Integration
Administrators can install a CDM integration by following the steps below
Initial configuration delay
After first creating a CDM-enabled integration in the BigPanda UI, ensure that the integration configuration has been saved and wait five minutes before sending changes via the integration. Changes sent immediately after saving an integration within the UI may not be successful.
Alternate Authentication
The Open Integration Manager (OIM) offers additional flexibility with authentication parameters. The following syntax can be used for the Token and App Key:
Auth Token:
query string: access_token=<token>
header: Authorization: Bearer <token>
header: x-auth-token: <token>
App Key:
query string: app_key=<app_key>
header: x-app-key: <app_key>
header: app_key: <app_key>
body: { "app_key": "<app_key>" }
This means that an event can be sent to BigPanda even if the integrated tool is not able to include any information in the API call's header, like in the example below:
https://integrations.bigpanda.io/oim/api/alerts?access_token=xxx&app_key=xxx
Integration Manager
You can customize how BigPanda processes payloads by adjusting the configuration defined in the Integration Manager.
Tag Mapping
Payload fields can be mapped to BigPanda change tags. Each matched field will be processed into BigPanda tags used for change correlation and root cause suggestions.
Additional tag mapping can be added to accommodate the full list of enrichment values to add to the event.
See the Open Integration Manager Advanced Configuration documentation for details on how to use the Advanced Tag Mapping Options.
Tag Mapping
Tag names must:
Start with a letter from a to z
Max 64 characters
Contain only lowercase letters (a-z), numbers (0-9), underscores ( _ ) and hyphens ( - )
Cannot include spaces
See Tag Naming Requirements for more information.
Note
Null values
CDM automatically removes null values, empty strings, empty objects, and empty arrays from payloads received.
Payload Processing Options
The Open Integration Manager (OIM) offers flexibility in payload structure and content, giving you tools to adjust how the event data should appear in BigPanda.
Force Lowercasing
The Lowercase created tag name toggle lets you automatically convert all tag names to lowercase. This keeps tags consistent throughout BigPanda, regardless of the capitalization from ingested tags.
Multiple alerts per payload
The Multiple alerts per payload toggle enables you to select a field that is an array of JSON or XML alert objects. Toggling this feature on will present you with a dropdown list of available fields. Select the field name of the object array. The fields listed will be populated by the last 10 previous payloads received as well as the sample payload. Add a test payload with the desired array field to have a new tag array appear in the dropdown.
If a tag contains nested JSON or XML objects (one tag contains multiple pieces of relevant information), you can choose which properties to include depending on the configuration.
Exclude Tags
The Ignore any tag not mapped above toggle controls whether only the desired fields are included in payloads. When toggled on, only tags that match the mapped fields will be included in the alert.
You can choose to instead select specific tags to leave out in the Exclude the following tags field. All tags that are not ignored will be included in the alert, even if they do not map to an existing BigPanda tag.
Either/Or
If Ignore any tag not mapped above is toggled on, all non-mapped tags will be excluded, not just those defined in the Exclude the following tags field.
Add Multiple Source Tags
Multiple source tags can be used for a single BigPanda tag, with the first tag populated defining the value. When listing multiple source tags, the tags run in the order they appear in the editor. Drag and drop source tags in the tag manager fields to rearrange the run order.
For example:
If two source tags, host and device, are listed, the system will first check for the host field in the payload. If there is a value for host, this will set the BigPanda tag value. If host is empty, the system will then check for the device field.
 |
Multiple Source Tag Mapping
If all source tag fields are empty, the system will use a default value if defined.
Array Flattening
Default Change
As of October 2024, new OIM integrations will not flatten arrays by default. Existing integrations will not be affected by this change. If you would like to turn off array flattening for an integration created before October 2024, please reach out to support to update the array flattening setting.
When flattened, each element within the array is sent as a separate tag in the event payload.
For example, an array like "annotation":["description1","summary1"] would send two tags when flattened: "annotation_0":"description1" and "annotation_1":"summary1".
By default, arrays are not flattened. Select a tag within Tag Mapping and enable the Array Flattening toggle to flatten the incoming payload into separate tags.
Change Fields
Change Fields
We recommend keeping the SummaryStart Time
If no value is found in the mapped tag, the time the event was received will be applied
End TimeTicket link
Destination tag names
CDM integration configuration defines change fields based on Destination tags rather than Source tags.
Required fields
All changes must include an Identifier, Summary, and Start Time. Ensure all event payloads will include the fields you have mapped for these tags..
Status Mapping
BigPanda change data manager statuses are determined by specific incoming tag values. Change status determines several system events and is necessary for correlating changes with potential incidents. Read more about how change status affects change correlation in the Root Cause Changes documentation.
In the Status Mapping fields, list payload values to map to each BigPanda event status. These must be an exact match, meaning that if ‘stop’ is in the value mapping, a message with ‘stopped’ will not be matched. If there are multiple tags that can determine a change's status (status, statusExtended, etc.), tags will be evaluated against all mapping values in listed order until a match is found or all tags rules have run.
One status will be selected as Default. If a payload does not match any listed status values for any of the specified tags, this status will be applied.
Event Filtering
The Event Filtering section uses the same BigPanda Query Language as the rest of the BigPanda platform. Any events that meet the defined criteria will be dropped upon ingestion and never visible in BigPanda.
This feature should only be used as a garbage filter for events that can never be actionable and would only add clutter in BigPanda. Below are some examples:
Misconfiguration (necessary tags are missing or cannot be parsed)
Non-actionable (lowest severity or contains no signal)
Events from Dev/QA environments
Non-events (info, logs, etc.)
Keep in mind that the syntax used here is based on source tags included in the event payload, not destination tags. If two different types of events from your tool use different tag names, be sure to include both in your filter criteria. The Query Assist feature will provide guidance based on your sample event payloads and recently delivered events.
Arrayed and Nested Alerts
If you are using an array of alerts by selecting Multiple alerts per payload, you will need to use the @alerts.<tag> syntax. For example, your filter criteria could be something like: @alerts.error_code="sdwan-datapath-down".
If the source system payload contains nested JSON, use dot walking notation: element.code=INFO, element.0.code=INFO.
Edit the Integration Configuration
Administrators can change the configuration settings of integrations at any time within BigPanda.
To open the Integration Manager:
Navigate to the Integrations Tab
Select the integration from the list
Click Review Instructions
Select the Integration Manager tab
Upload an example payload or send events to preview the configuration
Make any desired changes
Test Before Finalizing
We recommend testing configuration changes using test events and the preview panel before finalizing changes to an integration configuration.
Uninstall the Integration
Deleting an integration requires that you remove the integration in both the integrated system and BigPanda. We recommend that you first uninstall the integration on the integrated system to prevent traffic from being sent and rejected by BigPanda, since the app key will not exist once you delete the integration in BigPanda.
Caution during replacement
When replacing an existing integration with a new tool or system, we recommend configuring the new integration first to ensure no data is lost.
Deactivate Inbound Integration
If you want to stop sending data to BigPanda but don’t want to delete your integration, you can temporarily deactivate it.
To deactivate an inbound integration:
In BigPanda, navigate to the Integrations tab and select the desired integration from the list. This will open integration details on the right side of the window.
At the top of the integration details, click the Active/Inactive toggle next to the application name to change the status of the integration.
In the integrations list, inactive integrations will be marked with a gray bar.
Stop Sending Data to BigPanda
Within the integrated system, disable any settings that send data to BigPanda.
Each system requires specific changes to disable the integration with BigPanda. For example, you must delete the topic in CloudWatch, and you must disable the alert channel in New Relic. To determine the changes for your integrated system, reference the relevant documentation or contact BigPanda support.
Stop Sending Data from BigPanda
In BigPanda, disable any settings that send data to the integrated system. This includes modifying or removing AutoShare rules or Environments that are tied to the integration.
Delete the Integration in BigPanda
Take the following steps to delete the integration from BigPanda:
In BigPanda, navigate to the Integrations tab and select the desired integration from the list.
In the integration details on the right of the page, click the trash icon, then confirm you want to delete the integration. The integration will be removed immediately.
Data Removal
This procedure does not remove any data from the integrated system.
Best Practices with Change Integrations
When configuring change integrations, make sure to include as much relevant information into the payload as possible. In addition to giving your teams vital information, change data is key to quality-related change suggestions. The more details available, the better BigPanda’s RCC algorithms can work.
Ensure full change description and details are available through either the description field or custom tags
RCC uses free text when looking for matches, the data doesn’t need to be in any particular format as long as it is included in the payload for BigPanda.
Each integration is configured to pull specific information from the change management tool into BigPanda. If useful details are not currently included, consider using a custom integration or Changes REST API
Change rate limitations
To maintain optimal system health, the BigPanda rate limitation for Root Cause Changes is capped at max. 400 changes per minute and 50K changes per week.
Sending Custom Tags
Each integration includes optional fields that can be configured in the integration’s direction. When configured, each of these optional fields will appear as a new column in the change table.
For many integrations, the start timestamp is optional, which means that changes can be sent to BigPanda without a start time marked. However, these changes will not appear in the UI. Only changes sent to BigPanda with a start timestamp will appear in the UI. Most integrations will insert the current timestamp for changes missing a timestamp, but this may vary depending on the integration configuration.
Status field
The status field is taken from the payload of a change and is NOT updated according to the start/end timestamps
Column rate limitation
To optimize change table performance, max. 30 columns are displayed in the table at a time. Click on any one of the changes in the table to see a pop-up with the full list of tags associated with the change.
Integrate a Change Tool with REST API
Send potential change data through the Changes API to identify potential root cause for BigPanda incidents.
Easily push change records to BigPanda using our REST API for changes. View change records in the context of the incident to visualize and correlate between incidents and changes.
Learn more about supported data fields and best practices in the Root Cause Changes API documentation.
Create an App Key
Create an app key in BigPanda.
Integration specific
You'll need a separate app key for each integrated system.
App Key Configuration in BigPanda
Make sure you have an API Key
If you don’t have an API Key, create one in the BigPanda app.
Use this key whenever it says <your api key>.
Note: Don’t forget to copy the key before you leave as the key will be encrypted.
Make a REST Call From Your Change System
Configure the integrated system to POST to the changes API endpoint:
https://api.bigpanda.io/data/changesUse the following HTTP headers:
Authorization: Bearer <Your Org Bearer Token> x-bp-app-key: <Your App Key> Content-Type: application/json
The JSON payload can contain all or a subset of the following fields:
identifier: A unique identifier of this Change. Usually this will be the ticket key coming from the change system.status: Status of the change. One of [Planned,In Progress,Done,Canceled].summary: Short summary/title of the changestart: (Optional) Actual start time of the change in Epoch Unix Time (or any start time the user specified in the integration).end: (Optional) Actual end time of the change in Epoch unix time (or any end time the user specified in the integration)ticket_url: (Optional) Url of the record in the change system.tags: (Optional) JSON object that keeps all the tags for the record. Shown in the UI in the Related Changes tabmetadata: (Optional) JSON object for keeping any extra data that is not shown in the UI. Useful for machine learning correlation
Below is an example of a full payload:
{
"identifier": "CNG-123",
"status": "Done",
"summary": "Update to Contrail configuration for CAP to reduce network processing overhead",
"start": 19797234,
"end": 18797234,
"ticket_url": "https://jira.link.com/ticket?change=chng123",
"tags": {
"sys_id": "system_123",
"type": "risky"
},
"metadata": {
"opened_by": "userId123"
}
}Create a Test Change
To validate that everything is configured correctly, send a test change by running the following command in Shell:
curl -XPOST -H "Content-Type: application/json" \ -H "Authorization: Bearer <Your Org Bearer Token>" \ -H "x-bp-app-key: <Your App Key>" \ https://api.bigpanda.io/data/changes \ -d '{ "identifier": "CNG-166146", "tags": { "sys_id": "system_123", "type": "risky"}, "ticket_url": "https://jira.link.com/ticket?change=chng123", "summary": "Update to Contrail configuration for CAP to reduce network processing overhead", "end": 18797234, "start": 19797234, "status": "Done" }'