Identifying the root cause of an outage or a poorly performing application is one of the biggest challenges that IT organizations face today, and the fast-changing nature of modern dev only makes that more difficult.
BigPanda’s Root Cause Changes (RCC) feature simplifies this process by collecting change data right into the incidents dashboard, and then leveraging BigPanda’s machine learning algorithms to identify changes that may have led to incidents.
Use the Changes section and suggested matches to search and mark suspect changes and collaborate with other users to investigate which change caused the incident.
Marking changes as the suspected or matched root cause change of an incident is a vital tool in identifying historically problematic changes and in training the BigPanda OBML to recognize patterns between incidents and changes in your system.
Root Cause Changes is an optional feature and may not be turned on for all organizations. If you would like to begin using the root cause changes feature, contact us at [email protected]
Roles with the following permissions can interact with the Related Changes section of the incident details:
Mark changes as Suspect/Match and edit changes marked by other users
To learn more about how BigPanda's permissions work, see the RBAC - Role Based Access Control documentation.
Change data related to an incident is displayed in the Changes tab of the Incident Details pane. BigPanda uses Open Box Machine Learning (OBML) algorithms to correlate and suggest changes that may have caused an incident. If BigPanda has found a change to be highly correlated with an incident, it will appear at the top of the change table and appear on the Overview tab as a Potential Root Cause Change.
The Changes tab lists the details of changes that occurred shortly before or during the incident.
To navigate to the Changes tab:
- Select an incident in the incident feed.
- In the Incident Details pane on the right-hand, click on the Changes tab.
BigPanda digests change records from change management integrations and correlates them by start time with incoming incidents in real-time. By default, the change table chronologically displays the changes made in the 4 hours before the incident.
The change table contains columns that provide background information on the change:
- Status - The status of the change can be Planned, In Progress, Done. Cancelled changes persist in BigPanda but are not displayed in the change table.
- Key - The key is the original ID from the integrated change. Click on the hyperlinked change ID in the Key column of the table to view changes in the external change feed.
- Summary - A short description of the change.
- Start Time/End Time - The timestamps marking the duration of the change.
- Root Cause - Changes that may be a suspected or matched root cause of the incident can be marked manually or through BigPanda OBML.
Administrators have the option to add custom tags to the change integration to display additional columns to the change table, ie: type, assignee, CI, etc. To learn more about customizing the change table, see the Managing Root Cause Changes documentation.
You are able to change the size and order of the columns within the change table. Hover over the space between column names to bring up the dividing line or the 6 dots.
- To resize a column, click and drag the dividing line to the desired column width
- To move a column, click and drag the 6 dots icon to the desired column placement
Use the Show potential RCC only toggle to limit the change table to only show changes BigPanda has found highly correlated with the incident
Click on any one of the changes in the table to see a pop-up with the full list of tags and other data associated with the change.
If the BigPanda OBML change correlation algorithm marked the change as a suspected match, it will include a note about why the algorithm suspects the change. Hover over the information icon beside the note to get more details about why BigPanda’s OBML suspects that change.
BigPanda will only mark changes as Suspect (and not Match) to give users the final say on whether or not the change is the root cause of the incident.
You can search the change table for changes that meet specific criteria and fall within a selected time frame. Use the time frame selection tool or the search bar at the top of the change table to find specific types of changes.
By default, the Change Table displays changes that were active in the 1 hour before the incident started. You are able to select a different time frame from a set of options, or select a custom date range.
- To change the change table’s time frame, click the current time frame. From the dropdown, select the desired time window
- To filter by specific dates, select Custom Dates Range and enter the relevant dates and times in the dialogue box
The table displays changes that are active during a specified time frame. Changes are considered active if they:
- Start within the specified time frame
- End within the specified time frame
- Start before and remain active after the specified time frame
Changes can be searched using BigPanda Query Language (BPQL) to find specific tag values. Use free text, tag values, boolean, or regex queries to narrow the list of changes to only those that meet that requirement.
To learn more about using BigPanda Query Language to search for specific tag values within changes, see our BigPanda Query Language (BPQL) guide.
The Root Cause Change (RCC) status column lists whether a change has been identified as a suspected or matched cause of the incident.
Click on the RCC status for a change to see a popup that contains info about the user that set the status, the most recent activity, and its date/time. Any comment the user added when setting the status will be included in the popup.
If the RCC status was set by BigPanda’s OBML algorithms, the blue BigPanda icon will appear in the user field. Hover over the information icon to get more details about why BigPanda suspects the change is the potential root cause of the incident.
Mark changes as suspected or matched root causes to record the connection between the change and incident for your team, analytics, and BigPanda’s algorithm.
Marking changes as Suspect or Match is a vital tool in training the BigPanda OBML to recognize patterns between incidents and changes in your system
All changes can be marked with 1 of 3 statuses related to an incident. Select the status that best describes the change’s relationship to the cause of the incident:
- None - The change is likely not the cause of the incident. This is the default RCC status.
- Suspect - The change may have been related to the cause of the incident. If BigPanda’s RCC algorithms believe there is a strong connection between a change and incident, it will automatically mark the change as Suspect.
- Match - The change is likely the cause or related to the cause of the incident. BigPanda will never automatically mark matches - changes can only be marked as a match by a human teammate.
To set a change RCC status:
- In the Change table, click the status dropdown for the change
- Select the desired status
- Click on the status to enter a comment to add details or reasoning to the RCC status.
Comments can be a great foundation for collaboration, post-outage review, and machine-learning training.
When a different status is selected for a change, a record of the activity is created, with information about who set the change status, when, and any comment associated with it.
In addition, activities related to RCC (ie: type and time of correlation, comments, latest interaction, etc.) are listed chronologically in the activity feed.
BigPanda's OBML (Open Box Machine Learning) algorithms automatically detect connections between changes made to the system and incidents.
As new incidents are created or new alerts join an existing incident, BigPanda's OBML calculates their match potential with each past change.
If a change with high match potential is found, BigPanda marks the change as Suspect and adds a comment to the info popup explaining why the change was marked. Suspected changes will appear on the Overview tab as well as at the top of the Changes table. Filter the table to show only suspected and matched changes by clicking the Show potential RCC only toggle.
Suggested changes can rapidly speed up the root cause investigation process by identifying potential problems right at incident detection.
BigPanda will only mark changes as Suspect (not Match) to give users the final say on whether the change is the root cause of the incident
Updated 5 months ago