Alert Enrichment V2.1

Use the Alert Enrichment API to define alert tags that help users understand incidents more quickly and enrich BigPanda functionality, including correlation patterns and maintenance plans.

Read more about how alerts are contextualized and the potential correlation and automation possibilities of alert enrichment in the Alert Intelligence documentation.

Mapping enrichment UI

Mapping Enrichments can be managed right in the BigPanda UI. Read more in the Enrichment Maps documentation.

Update existing processes

If you have previously configured API management of Alert Enrichment using the V1 or V2.0 API, you can update those processes to this new API by following the steps in the Migrating to Alert Enrichment V2.1 documentation.

No additional steps are necessary to configure a new API call using the V2.1 API.

Alert Enrichment 2.1

Authentication

All BigPanda APIs require Bearer Token Authorization in the call headers.

This API uses the User API Key type of Authorization token.

BigPanda recommends adding Authentication headers only in the secure tool you use to make API calls.

Alert Tag Object

BigPanda normalizes alert data from integrated monitoring systems into standard key-value pairs, called tags. Alert tags are the fundamental data model for your alerts and drive alert deduplication, correlation, and enrichment.

In addition to the core properties such as name, active status, and description, all alert tags also include at least one enrichment item. Enrichment items set rules and conditions to populate values for the tag based on event-payload fields.

There are 3 types of enrichment items:

Composition
Extraction
Mapping

Enrichment items are tied to their tag. If all enrichment items are deleted from a tag, the tag will also be removed from the system. Deleting a tag will remove all of the tag’s enrichment items as well.

Delete mapping enrichment items

To delete a tag with mapping enrichment items, you must first delete the associated mapping enrichment schema.

Mapping enrichment schemas apply to multiple tags, and deleting a schema will remove the mapping enrichment from all associated tags. If any associated tags should remain in BigPanda, we recommend first creating the new schema, then deleting the old one.

Run order is an important part of successful alert enrichment, allowing you to create tags that leverage the values of other tags. Alert tags and the enrichment items within each tag can be reordered so that tag values are populated in the needed order.

Enrichment item order is based on the order it appears in the API call. When merging new or changing enrichment items, new enrichment items will be added to the top of the order.

Parameters

The Alert Tag object schema includes the following attributes:

Attribute	Description	Type
name	Tag name in BigPanda.	String
created_by	The unique ID associated with the user who created the tag.	String
created_at	The Unix epoch time when the tag was created (in seconds).	Timestamp
updated_by	The unique ID associated with the user who last updated the tag.	String
updated_at	The Unix epoch time when the tag was last updated (in seconds).	Timestamp
description	A brief summary of the tag	String
active	Whether the tag should be applied to incoming alerts.	Boolean
enrichments	List of automatic enrichment items configured for this tag. Items are listed in their run order. See the Enrichment Item Object table for details on enrichment object fields.	Array of Objects

Sample Return Object

{
      "name": "tag_name",
      "created_by": "565ee8e0dab117512da3c411",
      "created_at": 1464080054,
      "updated_by": "58d77efa5eafe459163f4211",
      "updated_at": 1641796851,
      "description": "",
      "active": true,
      "enrichments": [
        {
          "id": "191217f6-481d-464e-9f12-3ed846a72b11",
          "type": "composition",
          "active": true,
          "when": {
            "IN": [
              "source_system",
              [
                {
                  "type": "regex",
                  "value": "*"
                }
              ]
            ]
          },
          "version": 1,
          "config": {
            "destination": "source",
            "value": "compttest"
          },
          "created_by": "58d77efa5eafe459163f4211",
          "updated_by": "58d77efa5eafe459163f4211",
          "note": null
        },
        {
          "id": "74092648-393a-4896-b5c4-798ab0f00a11",
          "type": "extraction",
          "active": true,
          "when": {
            "AND": [
              {
                "IN": [
                  "source_system",
                  [
                    "api.test"
                  ]
                ]
              }
            ]
          },
          "version": 1,
          "config": {
            "source": "source",
            "destination": "tag_name",
            "regex": "Erroz: [^\\[]+ \\[[^\\]]+\\][\\. ]?\\[([^\\]^\\.]+)\\..*\\]",
            "template": "$1"
          },
          "created_by": "565ee8e0dab117512da3c411",
          "updated_by": "58d77efa5eafe459163f4211",
          "note": null
        }
      ]
    }

Composition Enrichment Items

Composition enrichment items are created by defining a value template.

Composition Template: A simple formula to populate tag value using strings and other tag values.

Use any tag value as a variable, in the format ${<tag_name>}. For example:

mywiki.com/${host}/${check}

Special formatting is required for tag values that contain encoded characters for URLs. If the tag value contains encoded values for the characters %, +, or a space, use the format ${exact(<tagname>)} to protect the values from being re-encoded.

Extraction Enrichment Items

Extraction enrichment items are created by defining a regular expression (regex formula).

Extraction Regex: The Regular expression to extract the new tag value from the source tag value.

Use a caret ^ to indicate starts with and a dollar sign $ to indicate ends with.
Use parentheses to surround the capture group for the extraction. BigPanda uses the contents of the first capture group to create the value of the alert enrichment tag.
Use a non-capturing group to ignore part of the contents.

If a tag value doesn't match the pattern, the alert is not enriched with the alert enrichment item.

Enrichment Item Parameters (Composition & Extraction)

For composition and extraction enrichment items, a unique object is created for each item.

Enrichment item order is based on the order it appears in the UI or API call. New rules will be added to the top of the existing enrichment order for that tag.

The enrichment item object schema includes the following attributes:

Attribute	Description	Type
id	System-generated unique identifier for the automatic enrichment item.	String
type	Automatic Enrichment rule type. One of: composition, extraction	String
active	Whether the rule should be applied to incoming alerts.	Boolean
when	A BPQL object query to filter incoming alerts. Only matched alerts will receive a value based on the enrichment item.	Object
version	Internal version number of the current enrichment item. This number is incremented automatically each time the item is updated.	Integer
config	Rules for building or extracting tag values from event data. Configuration attributes are unique to the enrichment type. See the Config Attributes tables for more information.	Object
created_by	The unique ID associated with the user who created the enrichment item.	String
updated_by	The unique ID associated with the user who last updated the enrichment item.	String
note	A description of the enrichment item purpose or functionality.	String

Config Object Attributes

Composition Config Attributes

Attribute

Description

Type

source

Filter incoming alerts by event source. Only alerts from the listed source system will receive a value based on the enrichment item.

If left empty, source will default to all sources.

String

destination

Which tag the enrichment item will apply to.

String

value

Template for building a value based on existing tags and string values.

String

Extraction Config Attributes

Attribute	Description	Type
source	Filter incoming alerts by event source. Only alerts from the listed source system will receive a value based on the enrichment item. If left empty, source will default to all sources.	String
destination	Which tag the enrichment item will apply to	String
regex	The pattern for extracting tag value from alert data	String
template	The structure of the value - system field. This value should not be modified.	String

Mapping Enrichment Items

Mapping enrichments are managed through their map object. See the Best Practices for Mapping Enrichments for details on defining mapping enrichments.

Example Objects

Extraction

{
"type": "extraction",
"active": true,
"when": {
   "IN": [
     "source_system",
      [
       {
     "type": "regex",
     "value": "*"
       }
      ]
    ]
  },
"config": {
"source": "host",
"destination": "tag_name",
"regex": "(.*)",
"template": null
},
"note": null
}

Composition

{
"type": "composition",
"active": true,
"when": {
   "IN": [
     "source_system",
     [
     {
     "type": "regex",
     "value": "*"
    }
    ]
  ]
},
"config": {
"destination": "tag_name",
"value": "val-${host6666}"
},
"note": "wiki.com"
}

Enrichment Retrieval Result

{
    "id": "d8215c7d-04ad-41fd-8293-5aca3002b111",
    "type": "composition",
    "active": true,
    "when": true,
    "version": 1,
    "config": {
        "destination": "dest",
        "value": "TEST - *"
    },
    "created_by": "58d77efa5eafe459163f4111",
    "updated_by": "58d77efa5eafe459163f4111",
    "note": null
}

Composition Config Attributes

Attribute

Description

Type

source

Filter incoming alerts by event source. Only alerts from the listed source system will receive a value based on the enrichment item.

If left empty, source will default to all sources.

String

destination

Which tag the enrichment item will apply to.

String

value

Template for building a value based on existing tags and string values.

String

Extraction Config Attributes

Attribute	Description	Type
source	Filter incoming alerts by event source. Only alerts from the listed source system will receive a value based on the enrichment item. If left empty, source will default to all sources.	String
destination	Which tag the enrichment item will apply to	String
regex	The pattern for extracting tag value from alert data	String
template	The structure of the value - system field. This value should not be modified.	String

Example Objects

Extraction

{
"type": "extraction",
"active": true,
"when": {
   "IN": [
     "source_system",
      [
       {
     "type": "regex",
     "value": "*"
       }
      ]
    ]
  },
"config": {
"source": "host",
"destination": "tag_name",
"regex": "(.*)",
"template": null
},
"note": null
}

Composition

{
"type": "composition",
"active": true,
"when": {
   "IN": [
     "source_system",
     [
     {
     "type": "regex",
     "value": "*"
    }
    ]
  ]
},
"config": {
"destination": "tag_name",
"value": "val-${host6666}"
},
"note": "wiki.com"
}

Enrichment Retrieval Result

{
    "id": "d8215c7d-04ad-41fd-8293-5aca3002b111",
    "type": "composition",
    "active": true,
    "when": true,
    "version": 1,
    "config": {
        "destination": "dest",
        "value": "TEST - *"
    },
    "created_by": "58d77efa5eafe459163f4111",
    "updated_by": "58d77efa5eafe459163f4111",
    "note": null
}

Mapping Enrichment Items

Mapping enrichment allows users to upload a CSV to map tag values to an enrichment table. Incoming alert data will be compared to the values in query_tag columns. If a match is found, the related tag will be updated with a value from the result_tag column.

Mapping enrichment items are added automatically to existing tags upon the upload of a mapping enrichment table with matching result_tag column names. If a map contains a result_tag column name that does not match an existing tag, a new tag is automatically created for the enrichment item.

result_tag column

If a result_tag column should match to a BigPanda tag with a name different from the column title, use the tag_name attribute. This will map the column to the BigPanda tag. No new tag will be created for the result_tag column title.

When the values of a mapping table should be updated, a new CSV file can be uploaded. All future incoming alerts will be mapped to the updated values.

Mapping enrichment items are created using two separate elements:

Mapping Enrichment Schema: The configuration of the Mapping Table to be uploaded. Defines properties such as: active status, source tag names (query_tag), BigPanda tag names (result_tag), and override settings. Individual tag values do not need to be configured, as values will be extracted from the uploaded table.
Mapping Enrichment Table: The dynamic table of values to populate into BigPanda based on the predefined query_tag and result_tag rules. This defines the actual data that will be used when enriching alerts.

Enrichment Map Parameters

Attribute	Description	Type
total_count	Total number of results. Returned with a GET call only.	Integer
items	The mapping data, in a JSON body	Array of strings

Mapping enrichment tables

Mapping Enrichment Tables are initially uploaded as CSV files. When managing individual map rows or retrieving mapping enrichment tables, the results will be returned as a JSON object.

Sample Object

{
  "items":
    [
      {
        "application": "sale24s",
        "region": "us-east",
        "team": "east-ops"
      },
      {
        "application": "billing23",
        "region": "us-west",
        "team": "west-ops"
      }
  ]
}

Parameters

The Mapping Enrichment object schema includes the following attributes:

Attribute	Description	Type
id	System-generated unique identifier for the enrichment map schema.	String
type	Enrichment function type used to create enrichment items. Must be “mapping” for mapping enrichment endpoints.	String
active	Whether the mapping rules should be applied to incoming alerts.	Boolean
When	BPQL object query to filter incoming alerts. Only matched alerts will receive a value based on the automatic enrichment item.	String
Version	Internal version number of the current schema. This number is incremented automatically each time the schema is updated.	Integer
config	Rules for mapping tag values to event data. See the Mapping Enrichment Configuration table for details.	Object
created_by	The unique ID associated with the user who created the mapping enrichment rule.	String
updated_by	The unique ID associated with the user who last updated the mapping enrichment rule.	String
note (optional)	A description of the mapping rule purpose or functionality.	String

Mapping Enrichment Configuration Parameters

The mapping enrichment configuration schema sets mapping rules for the related mapping tables and includes the following attributes:

Attribute	Description	Type
name	A system name for the mapping table. This name will appear in the UI.	String
fields	Settings for each field included in the mapping table. Attributes: title - column name on the mapping table type - whether query_tag or result_tag tag_name - (Optional) the name of the tag in BigPanda override_existing - (Optional) if a value has been populated for this tag by another map or enrichment rule, whether this value should replace that value optional - (optional) whether or not the tag must be applied to all incoming alerts. Default is false.	Array of Objects
id_generation_function	Method for id creation in system. System field.	String
case_insensitive	Whether the field names for this configuration take character case into consideration.	Boolean

override_existing

If override_existing is set to false, mapping values will not be added to any alert where the field is already populated, even if the field is populated with a “null.”

Best Practices for Mapping Enrichment

Mapping enrichment allows you to upload a CSV to map tag values to an enrichment table. Incoming alert data will be compared to the values in query_tag columns. If a match is found, the related tag will be updated with a value from the result_tag column.

You can define a mapping enrichment with the Alert Enrichment API using these resources:

Enrichments object—defines the schema for your enrichment table. The schema determines how the information in each column of the table will be used in the enrichment process—as a query field, to match a value in the incoming alert, or as a result field, to enrich the alert.
Map endpoint—allows you to upload and maintain a mapping table that follows a given enrichments schema.

https://files.readme.io/e37c858-mappingProcess.png

Step 1: Define The Mapping Structure

For a mapping enrichment, the Enrichments object defines how the information in each column of the table will be used in the enrichment process:

Query field—used to match a value in the incoming alert.

Result field—used to enrich the alert.

To define the mapping structure for a new enrichment:

Send a POST request to /enrichments. Specify the type as mapping. For each column in your data table, specify the name and whether to use the values as a query_tag or a result_tag. The request creates a new Enrichments object for this mapping enrichment schema.
Token value
Be sure to replace ${token} with the corresponding value for your organization.
Copy the id value from the response body. You will use it to identify this schema when you upload and update the mapping data.

Step 2: Upload The Mapping Data

After you create an Enrichments object that defines its structure, you can upload the table of the mapping data. For example, suppose that the Billing and Sales applications both have owners and remediation procedures. You can upload a table with three rows—one row for the column names and one row for each application.

To upload the mapping data for an enrichment:

Send a POST request to /enrichments/${id}/map. Replace ${id} with the value you copied from the response body in Step 1 and ${token} with the corresponding value for your organization.
Mapping data
The mapping data must be sent as comma-separated values (CSV) in the body of the request or in a CSV file. Ensure the data follows the structure you defined in Step 1 and has the same column names.
Enriched tag for analytics
If your organization would like to accurately visualize enrichment rates in Unified Analytics, you must include 2 enrichment flag columns in the CSV:
- An enriched or enrichment column with true/false rows, where true means alerts that match on that row should be considered enriched.
- A <map name> column with enriched or enrichment in each row (whichever was the name of the previous column), to add the enriched quality to any alert matched on the map.
The request initiates an asynchronous job to upload the table and creates a Job object to track its progress.
The payload must end with an empty new line. Make sure there are no spaces before the closing quotation marks on the last line.
Copy the trigger_uri value from the response body. You will use it to check the status of the upload.
Send a GET request to the ${trigger_uri} to verify that the upload has completed successfully. Replace ${trigger_uri} with the value you copied from the response body and ${token} with the corresponding value for your organization.
(Recommended) Send test alerts to ensure that the mapping enrichment works as expected. For example, you can run the following command to send matching test alerts via the Alert API. For more information, see POST alerts.
Resolve any test alerts, as necessary.

Step 3. Maintain the Mapping Data

As your team and infrastructure changes, you may need to update the values for your mapping enrichment. Depending on what changes you want to make, you can either completely replace the existing table, change only specific rows in the table, or create a new enrichment to change the structure.

To completely replace the existing table but keep the same structure, repeat Step 2. Upload the Mapping Data.

To change only specific rows in the existing table:

Send a PATCH request to /enrichments/${id}/map. Replace ${id} with the value you copied from the response body in Step 1 and ${token} with the corresponding value for your organization. For each row, specify the operation to perform and the data required to perform the operation. The request initiates an asynchronous job to upload the table and creates a Job object to track its progress.
Copy the trigger_uri value from the response body. You will use it to check the status of the update.
Send a GET request to the ${trigger_uri} to verify that the update has completed successfully. Replace ${trigger_uri} with the value you copied from the response body and ${token} with the corresponding value for your organization. For more information, see Check Status of Upload Job.
(Recommended) Send test alerts to ensure that the mapping enrichment works as expected.
Resolve any test alerts, as necessary.

To change the structure of your mapping data:

Deactivate the existing enrichment by sending a POST request to /enrichments/${id}. Replace ${id} with the value you copied from the response body in Step 1 and ${token} with the corresponding value for your organization. Set active to false. For more information, see Update Enrichment.
Create a new mapping enrichment, starting from Step 1: Define The Mapping Structure.

Sample Objects

{
"type":"mapping",
"active": true,
"when": {},
"config":{
"name": "mapping_rule",
"fields":[
{
"title":"application",
"type":"query_tag"
},
{
"title":"yes",
"type":"result_tag",
"override_existing":false
},
{
"title":"very_cool",
"type":"result_tag",
"tag_name":"very_cool",
"override_existing":false
}
],
"case_insensitive": false
},
"note": ""
}

Job Status Parameters

Attribute	Description	Type
status	Current status of the job, includes `pending`, `started`, `done`, and `failed`.	String
job_id	System-generated unique identifier for the asynchronous enrichment change job.	String
start_time	Unix epoch time in seconds when the job started.	Integer
end_time	Unix epoch time in seconds when the job completed.	Integer

Sample Object

{
  "status": "done",
  "job_id": "c750243a-39dd-4666-bf0c-f8715e19eb96_1662894455_indexcsv",
  "start_time": 1662894455,
  "end_time": 1662894462
}

Tag Order Parameters

Attribute	Description	Type
created_by	The user who created the tag order.	String
alert_tags_order	Alert tags listed in the order they should be run. By default tags are ordered by creation time.	Array of Strings

Sample Object

{
     "alert_tags_order": [
          "host",
          "application",
          "check"
     ]
}

Enrichment Item Order Parameters

Attribute	Description	Type
created_by	The user who created the enrichment item order.	String
enrichment_order	Enrichment items listed in the order they should be run. By default items are ordered by the order they appear in the call.	Array of Strings

Sample Object

{
     "enrichment_order": [
          "b6a7e384-ad82-4db4-b83b-9c87fabafdc9",
          "b6a7e384-ad82-4db4-b83b-9c87fa8a1118"
     ]
}

In this section:

Alert Enrichment V2.1

Alert Enrichment 2.1

Alert Tag Object

Parameters

Sample Return Object

Composition Enrichment Items

Extraction Enrichment Items

Enrichment Item Parameters (Composition & Extraction)

Config Object Attributes

Composition Config Attributes

Extraction Config Attributes

Mapping Enrichment Items

Example Objects

Composition Config Attributes

Extraction Config Attributes

Example Objects

Mapping Enrichment Items

Enrichment Map Parameters

Sample Object

Parameters

Mapping Enrichment Configuration Parameters

Best Practices for Mapping Enrichment

Step 1: Define The Mapping Structure

Step 2: Upload The Mapping Data

Step 3. Maintain the Mapping Data

Sample Objects

Job Status Parameters

Sample Object

Tag Order Parameters

Sample Object

Enrichment Item Order Parameters

Sample Object

Search results