Manage Alert Correlation
Correlation patterns can be customized to fit the needs of your organization.
Correlation patterns set rules to define relationships between system elements, which are then used to dynamically cluster alerts into incidents.
Correlation patterns define the relationships between alerts by using the following parameters:
- Source System
- Tags
- Time Window
- Filter (optional)
For example, you can create a pattern to correlate AppDynamics alerts together with matching applications, starting within 30 minutes of one another, that have a specific value in the production cluster.
UI Incident Title
If multiple correlation patterns match an incident, the pattern with the longest time window is the one that appears as the incident title.
Default Patterns
By default, during onboarding with BigPanda, several basic correlation patterns are configured for your organization.
Learn more about the default correlation patterns and how they cluster alerts into incidents in the Alert Correlation Logic documentation.
You can customize correlation patterns to tailor alert correlation to the specifics of your infrastructure. Correlation patterns are managed from the Correlation Patterns settings page.
Audit Logs
When multiple users are working in the BigPanda platform, it’s vital to see historic information on system configuration changes. Audit logs are available for alert correlation in BigPanda, allowing admins to keep track of configuration changes.
Users with the Audit Logs permission can view historical configuration changes and actions made related to alert correlation. See the Use the Audit Log documentation for more information.
Relevant Permissions
Roles with the following permissions can access Correlation Patterns in BigPanda Settings:
| Role Name | Description |
|---|---|
| Alert Correlation | View, edit, and create new Correlation Patterns in BigPanda Settings and API. |
Permission access levels can be adjusted by selecting either View or Full Access. To learn more about how BigPanda's permissions work, see the Roles Management guide.
Create New Correlation Patterns
You can create new correlation patterns at any time. Once activated the correlation patterns will automatically begin correlating new alerts.
Incoming Alerts Only
When you create or edit a correlation pattern or activate a previously disabled pattern, only new incidents will be correlated according to the pattern. Existing incidents and alerts will not be impacted. When you disable or delete a pattern, new alerts are no longer correlated according to it. However, existing incidents stay correlated according to the pattern logic for the remaining life cycle of the incident
To create a new correlation pattern:
- Navigate to Settings > Alert Correlation
- Click New Pattern
- Define the conditions that indicate that alerts might be related
- When satisfied with the pattern settings, click Create Pattern
Correlation Pattern Editor
| Field | Description |
|---|---|
| Source Systems | One or more integrated monitoring systems for which this pattern applies. |
| Enable cross source correlation | Option to correlate alerts from different source systems into the same incident. This option applies only if you select more than one source system for the pattern. Select the check box to correlate alerts from different source systems into the same incident, when applicable. Clear the check box to correlate only alerts from the same source into the same incident. The pattern still applies to every alert from every selected source, but will create different incidents for each source. |
| Correlation tags | Tag names to correlate alerts with matching values. For example, enter cluster and check to correlate all alerts that come from the same cluster and have the same check.You can enter up to five tags. The tags can be entered in any order. |
| Time Window | Maximum duration between the start time of correlated alerts in minutes. You can select a time window from 1 minute up to 4320 minutes (3 days). |
| Query Filter | (Optional) Query that further refines which alerts are correlated. Queries use BigPanda Query Language (BPQL) to filter alerts by alert tag data. For example, you can specify a tag of datacenter and then enter a query of check=ping to correlate only ping alerts by datacenter. When using the query filter, the Query Assist feature is available to help you build a query. See Query Assist for more information. |
| Note | (Optional) Short description of the pattern. E.g., a note that explains why the pattern is important and how it works. |
| Create As Inactive | Option to save the pattern definition without affecting your BigPanda instance. Select the check box to create the correlation pattern but not begin correlating alerts according to the pattern. Clear the checkbox to begin correlating new alerts according to the pattern immediately after it is created. |
Recent Tags Only
Dropdowns only include alert tags that have been updated or included in an alert in the last 90 days. To use an older tag, simply type the tag name.
Correlation Time Window
The correlation time window applies to the first event for a new alert. Alerts are correlated into the same incident only if their first event falls within the same time window (that is, they started around the same time).
If more than one pattern matches an incident, the incident title is based on the pattern with the largest time window.
The Preview Pane
The Preview pane appears to the right of the Correlation Pattern editor and displays the effects this correlation pattern would have had. The Preview pane uses real historical data in BigPanda without affecting any live data.
The preview displays detailed information for up to 50 incidents that match the correlation pattern within a selected time frame.
The Preview pane allows you to evaluate the correlation results for:
- Effectiveness—review the compression rate to see the percentage of alerts that are correlated into incidents. If a pattern is not as effective as it used to be, you may need to optimize the pattern to account for infrastructure changes.
- Accuracy—review how actual alerts would have been correlated into incidents according to this pattern. Confirm that alerts in each incident are related to the same problem.
To generate a preview, click Preview Correlation.
Correlation Pattern Preview Pane
As you make adjustments to the correlation pattern settings, click Preview Correlation to update the Preview pane to reflect the changes.
To the right, adjust the date range to see the impact the correlation pattern would have had on older alerts, or in a broader time window. The default time window is the past 48 hours.
Limited Time Windows
The correlation preview pulls data from your most recent events. Older dates may not change preview results as those events are not available. The available search window will depend on the volume of events in your system. If you do not see the results you expect, select a more recent date window.
Preview Results
The correlation preview will only display incidents that would have been formed by correlation logic. This means all incidents will have at least 2 alerts, and won’t include incidents that may be created by manual merges. When evaluating correlation effectiveness, keep in mind that preview results will differ dramatically from unified search results for similar incidents.
Alert Values
The preview looks at the original alert values attached to an incident at the time of its creation. Since incidents in BigPanda are updated in real time, it's possible that those tag values have changed as the incident evolved. This would appear in the preview as correlation between two alerts with values that don't match the pattern definitions.
You can use these incidents to see places where initial correlation might have tied alerts together that wound up not being related over the lifetime of the incident.
Manage Alert Correlation
You can edit, duplicate, temporarily deactivate, or permanently delete each alert correlation pattern you created. You can filter the list of alert correlation patterns by entering a search term in the field above the list. Or, by using predefined filtering by correlation tag, source, and status.
Search and Filter Correlation Patterns
By default, the correlation patterns list is sorted by most recently created or edited pattern. You are able to search and filter the pattern list to make managing your patterns easier.
Enter a term into the search bar to look for correlation patterns with specific names and properties. For example, enter Nagios to see all of the correlation patterns that have Nagios included as a source system.
The filter dropdowns beneath the search allow you to further hone the correlation patterns list. Filter by Correlation Tag, Source, or Status by selecting options from the dropdown lists. To remove your filter settings, click Clear.
Analytics Insights
In the right pane, you can view data related to the selected correlation pattern. Information about Compression Rate, Correlated Alerts, and Formed Incidents appears.
Deeper analytics into the effectiveness of your correlation patterns are available in the Correlation Patterns Insights dashboard. Click View Full Dashboard or navigate to Unified Analytics > Correlation Pattern Insights to open up the detailed reports. See Correlation Patterns Insights for more information.
Manage Correlation Patterns
To manage alert correlation patterns:
- Navigate to Settings > Alert Correlation. A list of existing alert correlation patterns appears.
- Select the alert correlation pattern you wish to edit, activate/deactivate, or delete.
- Use any of the following options to modify the alert correlation pattern:
| Option | Description |
|---|---|
| Edit | Edit a correlation pattern at any time to adjust which alerts will be clustered together. a. Click the Pencil icon or click Edit Pattern Details in the details pane. b. In the Correlation Pattern editor, adjust the properties to fit your needs. c. Click Update pattern to apply the changes. If a pattern is saved as Active, new alerts will be correlated by the pattern. Open incidents will be correlated by the original pattern that formed the initial incident. |
| Duplicate | Duplicate a pattern to use a previously created pattern as a template for a new pattern. a. Click the Copy icon. b. Adjust the pattern as necessary to fit the new properties. The system will not allow duplicate patterns and will trigger an error message if you try to save the pattern while it is identical to any other pattern in the system. c. Click Duplicate Pattern when you are happy with the changes. |
| Active or Deactivate | Inactivating a correlation pattern stops BigPanda from using that correlation pattern to cluster alerts together, but still preserves the pattern if you would like to turn it on again. Inactivating a correlation pattern is common when you are researching which adjustments need to be made, or when you are unsure if a correlation pattern is adding meaningful correlation. Using the toggle button, select Active or Deactivate. |
| Delete | Delete a correlation pattern when you are sure that it no longer applies to your infrastructure or process. A deleted correlation pattern cannot be recovered. a. Click the Trash icon. b. Click Delete to confirm, or Cancel to return to the previous page. New incidents will not be correlated by the deleted pattern, but existing incidents will continue to use the correlation pattern that they were created under. |
Correlation Patterns Run Order
Limited Availability
This feature is enabled for all organizations created after March 14th, 2024. For organizations that are interested in enabling this functionality, contact your BigPanda account team.
By default, correlation patterns are run in order of time window, from longest time window to shortest.
While this order is usually BigPanda’s recommendation, there are scenarios where patterns with shorter time windows should be prioritized. For example, a data center outage is a high-priority incident, but will likely have a shorter time window setting to prevent over-correlating.
If your organization encounters scenarios like these, you can manually order your correlation patterns to better reflect your system relationships.
Edit Pattern Run Order
To access the run order for correlation patterns, click the bidirectional arrow icon to the right of the correlation pattern search bar.
Access run order on the Alert Correlation screen
This will open a list of all your correlation patterns, both active and inactive. This list will also show you the time window for each correlation pattern, as well as the source system(s) and any filters.
Edit the pattern run order
To manually set a new run order, simply drag and drop correlation patterns to arrange them in your desired order. This action will switch your run order to manual mode. Once you save the new order, all new alerts will use the new pattern order.
New Pattern Placement in Manual Ordering Mode
If you create a new correlation pattern while patterns are manually ordered, that pattern will automatically be added to the bottom of the run order. To change when this pattern runs, you’ll need to edit the run order and drag and drop the new pattern to the desired placement.
To run correlation patterns in order of time window instead, simply click Order by time window in the upper right corner of the window. The run order for all correlation patterns will then be sorted from longest time window to shortest. Any new patterns created will be added in order by run time.
Display Discrepancy in Incident Console V1
If your organization is not using the V2 incident console you may see discrepancies between the manual correlation pattern order and the order in which correlated alerts are listed in the incident details pane. This is a known UI issue that displays alerts based on time window, even when the alerts themselves are correlated in a manual order. It does not affect the function of the manual run order.
This issue exists only in the V1 incident feed. If you are interested in enabling the V2 incident console for your organization, contact your BigPanda account team.
Correlation Pattern Run Order can also be managed with the Correlation Patterns API.
Correlation Pattern Suggestions
In addition to the default correlation patterns. BigPanda will also generate correlation pattern suggestions based on your organization’s historical data. As soon as you integrate your first monitoring tool, BigPanda begins to automatically review patterns in incoming data and identify potential patterns that may have been missed or overlooked. These generated patterns will be suggested in the correlation patterns list.
Onboarding Feature
Correlation pattern suggestions will only be available during the first three months of the onboarding process.
Suggested Correlation Pattern Flag
Suggested Correlation Pattern Flag
Suggested patterns are highlighted with the purple Suggested label and by default are not active. You are able to edit, activate, duplicate, and delete a suggested correlation pattern like any other correlation pattern, but they will always be marked as system recommended.
To activate a suggested pattern:
- Navigate to Settings > Alert Correlation.
- From the list of correlation patterns, select the suggested pattern that should be activated.
- To activate the pattern, click the Active toggle icon in the pattern details pane. The toggle will turn green.
New incoming alerts will be clustered by the correlation pattern.
New Suggestions
Suggested patterns are not calculated regularly after initial onboarding. To see a new batch of suggested correlation patterns, reach out to your account team.
Next Steps
Learn more about Alert Correlation Logic
Dig into Managing Incident Enrichment
Learn more about Navigating the Settings Menu
Updated about 2 months ago