Enrichment

Enrichment is the process of adding contextual information to alerts so that incidents can be intelligently correlated and easy to understand. BigPanda enriches alerts with custom tags based on the alert data, your operations processes, and known patterns and relationships in your infrastructure. Enriching alerts can help reduce alert noise and make it faster to detect and resolve problems.

Benefits of Custom Tags

Custom tags can enable more powerful correlation patterns to help detect issues more effectively. They can also help users understand and resolve incidents faster, with more information in incidents, Environments, unified searches, and analytics. For example, custom tags can add the following types of information:

  • Operational—helps IT Operations teams categorize, prioritize, route, and remediate an incident.
    For example, add the alert owner, wiki page URL, priority, or category.

  • Topological—provides context on the physical and logical relationships between the alerting object and the rest of the infrastructure.
    For example, add the alert cluster, tower, data center, or zone.

How It Works

BigPanda ingests the raw alert data from an integrated monitoring system and normalizes it into standard key-value pairs, called tags. Then, additional tags are created by using any of the following techniques:

  • Extraction—breaking down the value of one existing tag.

  • Composition—combining together any number of existing tag values and/or additional information.

  • Mapping—looking up values in a data mapping table.

The alert is enriched with any applicable custom tags, and the values of all the standard and custom tags are used to correlate the alert into an incident with other, highly related alerts. All the alert tags are available in the UI to help users search for, understand, and resolve incidents.

Extraction

With extraction, BigPanda creates new tags from the values of existing tags by looking for defined patterns in the source data. For example, if the host value follows a standard naming convention, you may be able to extract several key pieces of information from it.

If necessary, BigPanda can use different extraction patterns to create the same custom tag on different alerts. For example, suppose the host value follows different naming conventions in different data centers, but both include the cluster. BigPanda can extract the cluster for alerts from either data center by evaluating the value against both patterns.

For more information, see Example: Defining Extraction Tags.

Composition

With composition, BigPanda creates new tags by combining together any number of existing tag values and additional information. For example:

  • Build a runbook URL from the base URL of your wiki, the cluster, and the check.

  • Add a category, such as network or load, to alerts with certain check names.

  • Combine the values of several tags to create a new tag value.

Mapping

With mapping, BigPanda creates new tags by looking up values in a data mapping table. This technique allows you to leverage other sources of information about your infrastructure, such as a CMDB, registry, or team spreadsheet. For example, suppose certain applications have an owner and a standard operations runbook. You can upload a table with these application names and the associated owners and runbook URLs. Then, if any monitoring tool generates an alert with a matching application name, the alert is enriched with the associated owner and runbook URL.

For more information, see Quick Start: Defining A Mapping Enrichment.

Tags

BigPanda ingests raw alert data from an integrated monitoring system and normalizes it into standard key-value pairs, called tags. These tags become the data model for the alert. In BigPanda, tags enable alert correlation, provide incident information in the UI, and help you configure Environments, perform unified searches, and collect analytics.

Alert Tags

Tags are key-value pairs, which means they have two parts: the tag name and the tag value. For example, an alert may contain the tag host: acmegmon001, where host is the tag name and acmegmon001 is the tag value. Examples of how you can use tags in BigPanda include:

Tag Sources

Tags can be created directly from data sent by the monitoring system or through enrichment of the data that BigPanda receives.

  • Standard tags—standard data in the source system is normalized to standard tags for built-in integrations in BigPanda. To learn more, you can reference the documentation on standard tags for an integration.

  • Custom tags (sent with alert data)—custom data from the source system is sent to BigPanda. For example, a custom variable created in Nagios, or a custom attribute included in the JSON payload of an Alerts API request.

  • Custom tags (created through enrichment)—additional tags are created based on the alert data, your operations processes, and known patterns in your infrastructure. To learn more, see Enrichment.

Tag Naming Requirements

Tag names must meet these requirements:

  • Start with a letter (a - z).

  • Contain only lowercase letters (a - z), numbers (0 - 9), underscores (_), and hyphens (-). They cannot contain spaces or other special characters.

  • Maximum length of 64 characters.

Tag names that are created without the approved formatting will be unsearchable via BigPanda's Query Language and the Unified Search.

Defining Custom Tags

You can define custom tags that help users understand incidents more quickly and enrich BigPanda functionality. For example, based on naming conventions in your company's monitoring data, you can create custom tags to correlate alerts by cluster or to add runbook URLs for alert remediation.

Prerequisites

Creating New Custom Tags

When creating a new custom tag, you define which alerts the tag applies to and how the tag value is derived. Custom tags are created only for alerts that match the tag definition criteria.

  1. In the top right, click the Settings icon (), and then click Custom Tags. A list of the existing custom tags appears.
  2. Click New Tag.
  3. Click the tab for how you want to create the tag value:

    • Extraction—from the value of an existing tag. For example, extract the cluster name from the host value. For more information, see Example: Defining Extraction Tags.

    • Composition—based on existing tag values and additional information. For example, build a runbook URL from the base URL of your wiki, the host, and check name.

  4. Define the custom tag by filling in the following fields.

Field
Description

Custom Tag Name

Name for the tag in the BigPanda. For example, cluster or wiki_url. The name must follow the tag naming requirements.

Source System

Integrated monitoring systems for which to create this custom tag. You can include:

  • All alerts from a source type (such as Nagios or New Relic).

  • Alerts from a specific instance of a source type (for example, Nagios-US-EAST1).
    Alerts from all systems.

Extraction Only

Field
Description

Source Tag

Original tag from which the custom tag value is extracted. For example,host. See below for more.

Extraction Regex

Regular expression that defines the pattern for extracting the custom tag value from the source tag value.

  • Use a caret (^) to indicate starts with and a dollar sign ($) to indicate ends with.

  • Use parentheses to surround the capture group for the extraction. BigPanda uses contents of the first capture group to create the value of the custom tag.

  • Use a non-capturing group to ignore part of the contents.

If a tag value doesn't match the pattern, the alert is not enriched with the custom tag.

Source Tag

To ensure that the tag is created correctly, the source tag value must come directly from the monitoring system and not from another custom tag. For example, instead of extracting a custom tag from another extracted tag, you can extract the custom tag from the original value.

Source Tag

To learn more about a tag, you can view an alert in the BigPanda UI, reference the documentation on standard tags, or review the other custom tags defined for your organization.

Composition Only

Field
Description

Composition Template

Format for building the tag value from existing tag values and additional information. You can use any tag value as a variable, in the format ${<tag_name>}. For example: https://mywiki.com/${host}/${check}

Optional Fields

Field
Description

Note

Short description of the tag. Consider explaining why the tag is important and how it's created.

Create As inactive

Option to save the tag definition without affecting your BigPanda instance. If you do not select the check box, BigPanda adds the custom tag to new alerts that match the criteria immediately after the tag definition is created.

Query Filter

Query that further refines which alerts should contain this tag. For example, create this tag only for alerts from a specific data center or domain.

Click Add a query filter and enter a query in BigPanda Query Language (BPQL). As you type, the field displays suggested tags and relevant source system names.

  1. Check the preview in the right pane to make sure the custom tag will generate accurate tag values, and adjust the definition if necessary.

The preview shows the calculated tag value for a random set of matching alerts. To see a different set of matching alerts or to update the preview after changing the definition, click Refresh.

  1. Click Create Tag.

When Tags Are Created

If you did not select the Create as inactive check box, the tag will be created for any new alerts that match the tag criteria. Existing alerts are not affected by the tag.

Managing Existing Tags

You can edit, duplicate, temporarily disable, or permanently delete custom tags.

  1. In the top right, click your name, then click Settings.
  2. In the left pane, click Custom Tags. The existing custom tags are sorted with the most recently created tag listed first.
  3. Locate the custom tag you want to change.

Searching For Tags

You can filter the list of tags by entering a search term in the field above the list. For example, enter Nagios to see all of the custom tags that have Nagios as the source system.

  1. Use any of the following options to manage the tag:

    • To edit the tag, click the Edit icon (), and then modify the definition for how the tag is created. Click Update Tag to apply the changes.

    • To duplicate the tag, click the Duplicate icon (), and then modify the definition as necessary. Click Duplicate Tag to save the definition as a new custom tag.
      For example, to create similar tags for two different data sources, you can create the first tag definition. Then, duplicate and modify the copy to select a different source system and source tag.

    • To temporarily disable the tag, click the tag, and then click the Active toggle at the top right of the tag details pane. Click Deactivate to confirm the change.
      You can re-enable the tag by clicking the Inactive toggle.

    • To permanently delete the tag, click the Delete icon (), and then click Delete to confirm the deletion.

Changes Affect Only New Alerts

Changes to custom tag definitions affect only new alerts, not existing alerts. When you disable or delete a custom tag, the tag is no longer added to new alerts. However, existing alerts that contain the tag are not affected, and the tag value is still available in the UI and in searches.

Post-Requisites

Example: Defining Extraction Tags

You can define an extraction tag to enrich alerts with custom tags by breaking down the value of an existing tag. These examples demonstrate how to extract the cluster, data center, and other topological information based on the host naming conventions in a hypothetical infrastructure.

Prerequisites

Extraction Tags Overview

Define an extraction tag to extract data from a specific tag by using a regular expression.

  • The regular expression defines the pattern for extracting the custom tag value from the source tag value.

  • Use a caret (^) to indicate what the pattern starts with and a dollar sign ($) to indicate what it ends with.

  • Use parentheses to surround the capture group for the extraction. BigPanda uses contents of the first capture group to create the value of the custom tag.

  • Use a non-capturing group (?:) to ignore part of the contents.

  • The original tag value must match the regex pattern and the query filter, or the alert will not be enriched with the custom tag.

Example 1: Extracting Custom Tags From The Host Name

For this example, assume all hosts monitored by Nagios in your New York, San Francisco, and Sydney data centers follow this naming convention: service-node.cluster.datacenter.domain. For example, these host names follow the convention:

  • billing-1.db.nydc.acme.com

  • sales-3.app.sfdc.acme.com

  • mysql-5.db.sydc.acme.au

Also, assume that your London and Atlanta data centers follow a slightly different naming convention (see example 2), and you do not want to apply the same extraction pattern to alerts from those data centers.

Procedure

  1. In the top right, click the Settings icon (), and then click Custom Tags.
  2. Click New Tag
  3. Click the Extraction tab.
    4.To specify the original tag from which the custom tag is extracted, complete the following fields:

    • Source System—Nagios

    • Source Tag—host

  4. To define the name and value of the extracted tag, enter the values as listed in one row of the following table.

Custom Tag Name
Extraction Regex

service

^([a-zA-Z]*)-[0-9]*\..*\..*\..*\.[a-zA-Z]*$

node

^[a-zA-Z]*-([0-9]*)\..*\..*\..*\.[a-zA-Z]*$

cluster

^[a-zA-Z]*-[0-9]*\.(.*)\..*\..*\.[a-zA-Z]*$

datacenter

^[a-zA-Z]*-[0-9]*\..*\.(.*)\..*\.[a-zA-Z]*$

domain

^[a-zA-Z]*-[0-9]*\..*\..*\.(.*\.[a-zA-Z]*)$

These regex statements are based on the hypothetical sample environment. You can adapt the values as necessary to meet your infrastructure conventions and the needs of your teams.

  1. To limit the alerts for which this extraction pattern is applied, click Add a query filter and enter the following BPQL query: host NOT IN [*.londc.*, *.atldc.*]
  1. Check the preview in the right pane to make sure the custom tag will generate accurate tag values, and adjust the definition if necessary.
    The preview shows the calculated tag value for a random set of matching alerts. To see a different set of matching alerts or to update the preview after changing the definition, click Refresh.

  2. Click Create Tag.

  3. Repeat Steps 2 – 9 for each row in the table.

Results

The next time Nagios generates an alert with a host name that matches these conventions, the alert will be enriched with additional topological information. You can use the new tags in correlation rules and to search, sort, and investigate issues in the BigPanda UI. The following table shows how the original example values are enriched with custom tags.

Original Alert Tag
Enriched Alert Tags

host: sales-3.app.sfdc.acme.com

host: sales-3.app.sfdc.acme.com

service: sales

node: 3

cluster: app

datacenter: sfdc

domain: acme.com

host: mysql-5.db.sydc.acme.au

host: mysql-5.db.sydc.acme.au

service: mysql

node: 5

cluster: db

datacenter: sydc

domain: acme.au

host: billing-1.db.nydc.acme.com
`

host: billing-1.db.nydc.acme.com

service: billing

node: 1

cluster: db

datacenter: nydc

domain: acme.com

Example 2: Extracting Custom Tags from the Host Name

For this example, assume all hosts monitored by Nagios in your London and Atlanta data centers follow this naming convention: service-datacenter.cluster-node. For example, these host names follow the convention:

  • billing-londc.prod-1
  • accounts-atldc.test-5
  • accounts-atldc.prod-1

These data centers follow a slightly different naming convention from your New York, San Francisco, and Sydney data centers (see Example 1), and you do not want to apply the same extraction pattern to alerts from those data centers.

Procedure

  1. In the top right, click the Settings icon (), and then click Custom Tags.
  2. Click New Tag
  3. Click the Extraction tab.
    4.To specify the original tag from which the custom tag is extracted, complete the following fields:

    • Source System—Nagios

    • Source Tag—host

  4. To define the name and value of the extracted tag, enter the values as listed in one row of the following table.

Custom Tag Name
Extraction Regex

service

^([a-zA-Z]*)-.*\..*-[0-9]*$

datacenter

^[a-zA-Z]*-(.*)\..*-[0-9]*$

cluster

^[a-zA-Z]*-.*\.(.*)-[0-9]*$

node

^[a-zA-Z]*-.*\..*-([0-9]*)$

These regex statements are based on the hypothetical sample environment. You can adapt the values as necessary to meet your infrastructure conventions and the needs of your teams.

  1. To limit the alerts for which this extraction pattern is applied, click Add a query filter and enter the following BPQL query: host NOT IN [*.nydc.*, *.sfdc.*, *.sydc*]
  1. Check the preview in the right pane to make sure the custom tag will generate accurate tag values, and adjust the definition if necessary.
    The preview shows the calculated tag value for a random set of matching alerts. To see a different set of matching alerts or to update the preview after changing the definition, click Refresh.

  2. Click Create Tag.

  3. Repeat Steps 2 – 9 for each row in the table.

Results

The next time Nagios generates an alert with a host name that matches these conventions, the alert will be enriched with additional topological information. You can use the new tags in correlation rules and to search, sort, and investigate issues in the BigPanda UI. The following table shows how the original example values are enriched with custom tags.

host: billing-londc.prod-1

host: billing-londc.prod-1

service: billing

datacenter: 1

cluster: db

node: nydc

host: accounts-atldc.test-5

host: accounts-atldc.test-5

service: accounts

datacenter: atldc

cluster: test

node: 5

host: accounts-atldc.prod-1

host: accounts-atldc.prod-1

service: accounts

datacenter: atldc

cluster: prod

node: 1

Post-Requisites

Enrichment

Enrichment is the process of adding contextual information to alerts so that incidents can be intelligently correlated and easy to understand. BigPanda enriches alerts with custom tags based on the alert data, your operations processes, and known patterns and relationships in your infrastructure. Enriching alerts can help reduce alert noise and make it faster to detect and resolve problems.