Manage Alert Enrichment

Alert tags are used to add contextual data to your alerts based on predefined rules for improved alert management and reduced MTTR.

BigPanda ingests raw event data from integrated monitoring systems and normalizes them into key-value pairs called tags. Based on these existing tags, you can create new alert enrichment tags that add metadata to incoming events in your organization’s system.

For example, you can add operational data that enables you and your team to categorize, prioritize and remediate an incident (e.g., add a tag named “owner,” “priority,” or “category”). Or, add topological information that provides context to the physical and logical elements of the alert (e.g., add a tag named “cluster,” "data center,” or “city”).

Relevant Permissions

Roles with the following permissions can access the Alert Enrichment Tags page in the BigPanda Settings:

Role NameDescription
Custom_Tags_ReadRead-only - view existing alert enrichment tags.
Custom_Tags_Full_AccessFull access - preview and create new and inactive Extraction and Composition tags.

To learn more about how BigPanda's permissions work, see the Roles Management guide.

Alert Enrichment Tag Types

You can create the following alert enrichment tags and rule types:

  • Extraction: extract values from an existing tag to create new custom tags.
  • Composition: combine multiple values of existing tags to create one new custom tag.
  • Multi Type: create a tag composed of several function types.
  • Mapping: (API Only) added automatically to the list of tag rules when a map includes a result tag value with the same name as a tag.

Extraction

Using the extraction process, you can extract important metadata from existing tag values and create new tags. For instance, a hostname is generally comprised of key pieces of information, such as service, node, cluster, datacenter, and domain. Each of these data points can be extracted into their own tag.

Tag Extraction ProcessTag Extraction Process

Tag Extraction Process

For example, if you wish to correlate events according to their cluster you can extract the cluster data point from an existing source tag by using regular extraction expressions (regex). You can define the new tag with the name “cluster.”

Composition

Using the composition process, you can create new tags by combining the value of any number of existing tag values and/or additonal information. For example, if you wish to add a runbook URL for alert remediation you can combine the values of your base wiki URL, cluster, and check to create one new tag value.

Tag Composition ProcessTag Composition Process

Tag Composition Process

Multi Type

You can define a tag as one particular function type; extraction, composition, or fixed-value. In addition, you can define a tag as a multi-type tag which allows you to create a tag composed of several function types. For example, you can create a tag including fixed-value items, extraction items, and composition items. Each of these enrichment items runs for the tag and returns values.

Multi-Type Tag EditorMulti-Type Tag Editor

Multi-Type Tag Editor

Create Alert Enrichment Tags

Each tag is an aggregation of its rules. When opening a tag, a list of all its condition rules appears in the right pane.

  1. Navigate to Settings > Alert Enrichment
  2. Click New Tag.
  3. Define the following parameters:
ParameterDescription
Tag NameName the tag (e.g., cluster, city, or runbook).
Tag names must meet the following requirements:

1. Maximum length of 64 characters.
2. Start with a letter from a to z.
3. Contain only lowercase letters (a-z), numbers (0-9), and some special characters, including underscores ( _ ) and hyphens ( - ) and cannot contain spaces.
Function TypeSelect a function type.
Composition: defines alert tags by combining values of existing tags.
Extraction: defines alert tags by extracting values from an existing tag.
Multi Type: uses values from multiple function types.
Add description(Optional) Add a description for the tag.

An automatic enrichment box opens.

Automatic Enrichment EditorAutomatic Enrichment Editor

Automatic Enrichment Editor

  1. Define the following parameters for each different function type:

Composition Tag

ParameterDescription
ConditionDefine a condition (query filter) to filter which alerts should contain the tag (e.g., host=ny).
Source SystemSelect a specific integrated monitoring system for which the tag applies. Or, select All Systems to apply this tag to events from all source systems.
Composition TemplateProvide the expression to create and combine data for building the tag value from other existing tag values.

Use any tag value as a variable, in the format ${<tag_name>}. For example: <https://mywiki.com/${host}/${check}>

Special formatting is required for tag values that contain encoded characters for URLs. If the tag value contains encoded values for the characters %, +, or a space, use the format ${exact(<tagname>)} to protect the values from being re-encoded.
Add a Note(Optional) Add a short description about the tag (e.g., explain why the tag is important).

Extraction Tag

ParameterDescription
ConditionDefine a condition (query filter) to filter which alerts should contain the tag (e.g., host=ny).
Source SystemSelect a specific integrated monitoring system for which the tag applies. Or, select All Systems to apply this tag to alerts from all source systems.
Source TagSelect the original tag from which the new tag is extracted.
Extraction RegexProvide the Regular expression to extract the new tag value from the source tag value.

- Use a caret (^) to indicate starts with and a dollar sign ($) to indicate ends with.
- Use parentheses to surround the capture group for the extraction. BigPanda uses the contents of the first capture group to create the value of the alert enrichment tag.
- Use a non-capturing group to ignore part of the contents.
If a tag value doesn't match the pattern, the alert is not enriched with the alert enrichment tag.
Add a Note(Optional) Add a short description about the tag (e.g., explain why the tag is important).

📘

Reserved Words

When creating BPQL conditions for tags, some words in BigPanda are Reserved and cannot be used.

description can be used in BPQL to create extraction enrichment tags.

  1. To add Automatic Enrichment items to apply to the tag, select Add Item. Each new item is a rule that creates the alert enrichment tag.

👍

Add a Multi Type Tag

You can create Multi Type tags with both Composite and Extraction function types by clicking Add Item. Define the following parameters for each enrichment item on a Multi Type tag:

  1. Select Preview. A preview of the tag appears in the right pane. If necessary, adjust the tag definitions.
  2. Select Preview All.
  3. Select Create Tag.
    In the right pane, you can view all of the alert’s details including the new tag.

Preview Tag Results

With the complexity of modern ITOps, even small adjustments to enrichment logic can make big changes for managing incidents. To help you find the right enrichment patterns, the Alert Enrichment engine offers a Preview capability to help rapidly develop, test, and iterate enrichment logic.

Previewing an Enrichment TagPreviewing an Enrichment Tag

Previewing an Enrichment Tag

The Enrichment Preview fetches sample historical alerts from your BigPanda instance that match the Condition for any one of the enrichment items for that tag. Using these historical alerts as a “working set,” the preview will show how the current enrichment logic would apply to these tags.

To generate a Preview for the enrichment logic of a specific item, click on the Preview button next to the desired enrichment item. In the Preview pane to the right, each of the alerts from the working set will populate in a table showing:

  • The calculated value for the enrichment tag currently being edited, highlighted in purple
  • The source tag(s) with any extracted values highlighted in yellow
  • Additional alert data to give context on the alert type

️ Missing Enrichment

If the leftmost column is blank for any sample alerts, this means the enrichment logic would not apply for similar alerts in production. Reasons for a tag to not be enriched include:

  • Dependent tags not existing in the source alert
  • Extraction regex failing to match the source tag
  • Source values not found in a mapping enrichment

If values are failing to populate the left column, consider changing the item’s definition to be more generic, or creating more items to cover these alert types.

Literal Pipes in Tag Values

Pipes: | are used in BigPanda as a delimiter for array values.
If the value should have a literal pipe, wrap the entire cell in three quotes:
"""this is a | literal pipe"""
If the value should have both a literal pipe and quotation marks, then the cell should be wrapped in three quotes, and the quoted text needs to be wrapped in four quotes:
"""this is a | literal pipe with """"quoted"""" text"""

Tag Execution Order

Each tag is treated as a complete set, with all rules being run before moving on to the next tag. By default, alert enrichment tags run in the order of their creation. You are able to rearrange the tags through a simple drag and drop.

To change the order of execution for alert enrichment tags:

  1. Navigate to Settings > Alert Enrichment.
  2. Select the Execution Order icon.
  3. Click and hold the left edge of the tags and drag them into the desired order. To send a tag to the beginning or end of the execution order, click the Three Dots Icon, and select Top of the List, or Bottom of the List.
Modifying the Alert Tags Execution OrderModifying the Alert Tags Execution Order

Modifying the Alert Tags Execution Order

  1. Click Save to finalize the changes.

Enrichment Rules Execution Order

When an alert enrichment tag has multiple enrichment rules, BigPanda uses the first matching rule in the enrichment tag. After finding the first match, the system does not run the remaining rules and moves to the next tag.

By default enrichment rules run in the order of their creation. You are able to rearrange the rules through a simple drag and drop.

To change the order of execution for an alert tag:

  1. Navigate to Settings > Alert Enrichment.
  2. Select the alert tag to modify.
  3. Click the Pencil icon or Edit Alert Tag button to open the tag editor.
  4. Click and hold the left edge of the rules and drag them into the desired order.
Modifying Tag Execution OrderModifying Tag Execution Order

Modifying Tag Execution Order

  1. Click Update Tag to finalize the changes.

Mapping Enrichment

Mapping enrichment allows you to enrich your alerts with additional information about your organization by importing data from external data sources, such as a CMDB or team spreadsheet.

For example, you can upload a data mapping table including a list of application names, their associated owners, and runbook URLs. If any monitoring tool generates an event with a matching application name, the event is enriched with data about the associated owner and runbook URL.

BigPanda Mapping ProcessBigPanda Mapping Process

BigPanda Mapping Process

To add or change mapping enrichment data in BigPanda, you can upload a CSV file through the Alert Enrichment API.

Maps are interconnected and by default run in the defined order according to creation time.

Mapping rules

An alert enrichment mapping tag is an aggregation of its rules. Each of these rules appear as enrichment items within the mapping tag.

By default, tags and their enrichment items are ordered by their creation time. You can reorder the list of enrichment items by dragging and dropping them in the tag editor.

You are not able to edit the rules of mapping enrichment items within the UI. To edit mapping rules, reupload the enrichment map through the BigPanda API.

📘

When viewing mapping enrichment, only result tags are listed in the alert enrichment tag list. Query or key tags are not listed.

Tag Dependencies

In certain cases of enrichment, you need to ensure that a specific tag or map runs before another one because the tags and maps are dependent on each other. A dependency is any reference of a tag or map that runs after another in the execution order. For example, if you created an extraction tag named “Service” using regex from the source tag named "Host," the tag “Service” is dependent on the source tag "Host." Similarly, a composition tag is dependent on the original existing tags that you have combined together.

Tags within BigPanda are executed in the order they appear in the BigPanda UI. The tag should be below the source tag in the enrichment order for any dependencies to run correctly.

You can choose to create dependencies between tags and maps, but they are not enforced. However, in certain cases when creating dependencies the results may be different in each run. For instance, if the tag “Host” is also received in the original payload then the “Service” tag uses this original value and not the “Host” tag created by the enrichment process.

🚧

Mapping item reordering

Any enrichment order changes will need to be adjusted when updating mapping enrichment.

To change the rules or content of a mapping enrichment tag, the entire map must be reuploaded. If tags or enrichment items were reordered within the UI, they will lose the new order upon reupload as the tags will revert to the created first rules.

Manage Alert Enrichment Tags

You can edit, temporarily deactivate, or permanently delete each tag you created. You can filter the list of tags by entering a search term in the field above the list. Or, by using predefined filtering by status, type, and source.

To manage alert enrichment tags:

  1. Navigate to Settings > Alert Enrichment. A list of existing tags appears.
  2. Select the tag you wish to edit, activate/deactivate, or delete.
  3. Use any of the following options to modify the tag:
ParameterDescription
Edita. Select the Edit icon.
b. Modify the tag's definitions according to your needs.
c. Select Update Tag to apply the modifications.
Active or DeactivateUsing the toggle button, select Active or Deactivate. BigPanda adds an active tag to new alerts that match the criteria immediately after the tag definition is created. The new alert enrichment tag is not added to existing alerts in the system.
When deactivating a tag that includes a map, the mapping rule is still preserved.
Deletea. Select the Delete icon.
b. Select Delete to confirm the deletion.

📘

Manage Alert Enrichment Tags

Modifications made to alert enrichment tags affect only new alerts, not existing alerts. When you disable or delete an alert enrichment tag, the tag is no longer added to new alerts. However, existing alerts that contain the tag are not affected, and the tag value is still available in the UI and in searches.

🚧

Custom Tags and Correlation Patterns

Correlation patterns are based off tag names. When editing custom tag names, correlation patterns with the tag must be updated to reflect the change.

Next Steps

Start Managing Alert Correlation

Learn more about Navigating the Settings Menu

Dig into Alert Enrichment