Skip to main content

Best Practices for Mapping Enrichment

Mapping enrichment allows you to upload a CSV to map tag values to an enrichment table. Incoming alert data will be compared to the values in query_tag columns. If a match is found, the related tag will be updated with a value from the result_tag column.

Mapping enrichment items are added automatically to existing tags upon the upload of a mapping enrichment table with matching result_tag column names. If a map contains a result_tag column name that does not match an existing tag, a new tag is automatically created for the enrichment item.

You can define a mapping enrichment with the Alert Enrichment API using these resources:

  • Enrichments object—defines the schema for your enrichment table. The schema determines how the information in each column of the table will be used in the enrichment process—as a query field, to match a value in the incoming alert, or as a result field, to enrich the alert.

  • Map endpoint—allows you to upload and maintain a mapping table that follows a given enrichments schema.

https://files.readme.io/e37c858-mappingProcess.png

Step 1: Define The Mapping Structure

For a mapping enrichment, the Enrichments object defines how the information in each column of the table will be used in the enrichment process:

Query field—used to match a value in the incoming alert.

Result field—used to enrich the alert.

To define the mapping structure for a new enrichment:

  1. Send a POST request to /enrichments. Specify the type as mapping. For each column in your data table, specify the name and whether to use the values as a query_tag or a result_tag. The request creates a new Enrichments object for this mapping enrichment schema.

    Token value

    Be sure to replace ${token} with the corresponding value for your organization.

  2. Copy the id value from the response body. You will use it to identify this schema when you upload and update the mapping data.

Step 2: Upload The Mapping Data

After you create an Enrichments object that defines its structure, you can upload the table of the mapping data. For example, suppose that the Billing and Sales applications both have owners and remediation procedures. You can upload a table with three rows—one row for the column names and one row for each application.

To upload the mapping data for an enrichment:

  1. Send a POST request to /enrichments/${id}/map. Replace ${id} with the value you copied from the response body in Step 1 and ${token} with the corresponding value for your organization.

    Mapping data

    The mapping data must be sent as comma-separated values (CSV) in the body of the request or in a CSV file. Ensure the data follows the structure you defined in Step 1 and has the same column names.

    Enriched tag for analytics

    If your organization would like to accurately visualize enrichment rates in Unified Analytics, you must include 2 enrichment flag columns in the CSV:

    • An enriched or enrichment column with true/false rows, where true means alerts that match on that row should be considered enriched.

    • A <map name> column with enriched or enrichment in each row (whichever was the name of the previous column), to add the enriched quality to any alert matched on the map.

    The request initiates an asynchronous job to upload the table and creates a Job object to track its progress.

  2. The payload must end with an empty new line. Make sure there are no spaces before the closing quotation marks on the last line.

  3. Copy the trigger_uri value from the response body. You will use it to check the status of the upload.

  4. Send a GET request to the ${trigger_uri} to verify that the upload has completed successfully. Replace ${trigger_uri} with the value you copied from the response body and ${token} with the corresponding value for your organization.

  5. (Recommended) Send test alerts to ensure that the mapping enrichment works as expected. For example, you can run the following command to send matching test alerts via the Alert API.Open Integration Manager (OIM) Alerts

  6. Resolve any test alerts, as necessary.

Step 3. Maintain the Mapping Data

As your team and infrastructure changes, you may need to update the values for your mapping enrichment. Depending on what changes you want to make, you can either completely replace the existing table, change only specific rows in the table, or create a new enrichment to change the structure.

To completely replace the existing table but keep the same structure, repeat Step 2. Upload the Mapping Data.

To change only specific rows in the existing table:

  1. Send a PATCH request to /enrichments/${id}/map. Replace ${id} with the value you copied from the response body in Step 1 and ${token} with the corresponding value for your organization. For each row, specify the operation to perform and the data required to perform the operation. The request initiates an asynchronous job to upload the table and creates a Job object to track its progress.

  2. Copy the trigger_uri value from the response body. You will use it to check the status of the update.

  3. Send a GET request to the ${trigger_uri} to verify that the update has completed successfully. Replace ${trigger_uri} with the value you copied from the response body and ${token} with the corresponding value for your organization.

  4. (Recommended) Send test alerts to ensure that the mapping enrichment works as expected.

  5. Resolve any test alerts, as necessary.

To change the structure of your mapping data:

  1. Deactivate the existing enrichment by sending a POST request to /enrichments/${id}. Replace ${id} with the value you copied from the response body in Step 1 and ${token} with the corresponding value for your organization. Set active to false.

  2. Create a new mapping enrichment, starting from Step 1: Define The Mapping Structure.

Composition Enrichment Items

Composition enrichment items are created by defining a value template.

Composition Template: A simple formula to populate tag value using strings and other tag values.

Use any tag value as a variable, in the format ${<tag_name>}. For example:

mywiki.com/${host}/${check}

Special formatting is required for tag values that contain encoded characters for URLs. If the tag value contains encoded values for the characters %, +, or a space, use the format ${exact(<tagname>)} to protect the values from being re-encoded.

Extraction Enrichment Items

Extraction enrichment items are created by defining a regular expression (regex formula).

Extraction Regex: The Regular expression to extract the new tag value from the source tag value.

  • Use a caret ^ to indicate starts with and a dollar sign $ to indicate ends with.

  • Use parentheses to surround the capture group for the extraction. BigPanda uses the contents of the first capture group to create the value of the alert enrichment tag.

  • Use a non-capturing group to ignore part of the contents.

If a tag value doesn't match the pattern, the alert is not enriched with the alert enrichment item.

JSON Format

The Alert Enrichment API accepts only standard double-quoted JSON formatting. Python JSON uses single quotes and will not be accepted by the API.

If using Python, please sanitize your JSON to adhere to standard double-quoted JSON formatting.

Example JSON formatted payload:

curl --request POST \
  --url https://api.bigpanda.io/resources/v2.1/alert-enrichments \
  --header "Authorization: Bearer ${your_api_key}" \
  --header 'Content-Type: application/json' \
  --data '{
	"field": "states",
	"secondary_field": true,
	"object": {
		"subobject": [
			{
				"field": "string",
				"secondary_field": "also string"
			]
	},
	"another_field": "url.com"
}'
In this section: